Method and system for eliminating two-dimensional features from image

Abstract

A method and system for eliminating two-dimensional features from a planar surface in a two-dimensional image. Three digital images of a subject, such as a tray of a case, are captured from a single camera at a fixed position. An image (IA) is captured under ambient lighting, an image (I1) has ambient lighting plus a first additional light source, and an image (I2) has ambient lighting plus a second additional light source. An output image Q is then calculated by Q = (I1-IA) / (I2-IA). And the environment image is subtracted, so that environment diffuse reflection and mirror reflection are removed. The division eliminates all color-induced variations in the output image. The uniquely retained change is a change due to the angle between the normal direction of each surface point and the direction from the light source to that point. The output image Q without all colors and two-dimensional features is well suited for computing the capture of objects in the image by the robot.

Description

Technical Field

[0001] This disclosure relates to the field of image analysis, and more specifically, to a method and system for removing 2D features from a planar surface and highlighting 3D features in a 2D image, wherein three images are captured by a fixed camera using different combinations of ambient light and supplementary light sources, and image arithmetic is used to calculate an output image without 2D features, including shadow variations caused by color, graphics, markings, and tape. Background Technology

[0002] Using camera images as input to machine control systems is well-known, with applications ranging from car collision avoidance to industrial robot motion programming. In a common robotic application, a pallet of boxes is provided, and an industrial robot picks up one box at a time from the pallet and places each box at a destination location (such as on a conveyor), from which the boxes are removed for further processing. In such depalletizing operations, it is known to use one or more cameras to provide images of the box pallet and analyze these images to identify the corners, edges, and sides of the boxes. A depalletizing algorithm is then used to select specific boxes for the next robot gripping operation, and this process is repeated until the pallet is empty.

[0003] Depending on the nature of the container on the pallet, it may be difficult to accurately identify the shape and size of the container using known image processing techniques, including both two-dimensional (2D) and three-dimensional (3D) methods. 3D methods typically use point cloud data, such as that from optical methods (e.g., stereo imaging, structured light, or time-of-flight), to approximate the three-dimensional shape of the observed object's surface. However, point cloud data can be noisy, often has sparse data point spacing, and may include areas of "dropout" or missing surface points. These and other problems with point cloud data make it difficult to accurately determine the shape of the observed object using 3D image analysis.

[0004] Analyzing 2D images to identify objects can also present problems. Many boxes include text, graphics, color variations, tape, and other features on their surfaces, making edge and corner detection using 2D camera images unreliable. If a box's lines or color features are incorrectly identified as box edges, this can cause the robot to attempt to grasp in the wrong location, leading to grasping failure or the box falling.

[0005] In view of the above, there is a need for an improved image analysis technique that eliminates shadow variations caused by color and other 2D features from planar surfaces to improve image-based object recognition. Summary of the Invention

[0006] Based on the teachings of this disclosure, a technique is provided for removing 2D features from a planar surface in a 2D image. This technique involves capturing three digital images of a subject (such as a stack of palletized boxes). All images are captured by a single camera at a fixed location. One image (I A The images (I1 and I2) were taken under ambient lighting conditions. One image (I1) has ambient lighting plus a first additional light source at a first location, and the other image (I2) has ambient lighting plus a second additional light source at a second location. Then, the equation Q = (I1 - I2) is used to... A ) / (I2-I A The output image Q is calculated. Subtracting the environment image removes ambient diffuse and specular reflections. Division eliminates all color-related variations in the output image. The only remaining variation is due to the angle between the normal direction of each surface point and the direction from the light source to that point, and the output image Q has none of the color-based shading variations and reflections (such as markings, graphics, and tape) while preserving all 3D features (such as gaps between boxes). The output image Q is particularly well-suited for computational robotic grasping of objects in an image.

[0007] Additional features of the currently disclosed apparatus and method will become apparent from the following description and appended claims, taken in conjunction with the accompanying drawings. Attached Figure Description

[0008] Figure 1 It is an image of a collection of boxes on a pallet, illustrating how the graphic patterns and colors on the boxes can make it difficult to distinguish one box from another in 2D image analysis;

[0009] Figure 2 The image shows a collection of boxes on a pallet, illustrating the tape at the seams of the boxes and how the resulting reflections can make it difficult to distinguish one box from another in 2D image analysis.

[0010] Figure 3 This is a diagram illustrating the diffuse and specular reflection behavior from a surface;

[0011] Figure 4 This is an illustration of a system for acquiring and processing an image of a subject according to an embodiment of the present disclosure to produce an output image, which has no shading variations due to 2D features such as color, graphics, and tape.

[0012] Figure 5 This is a flowchart of a method for obtaining a subject image and processing the image to produce an output image according to an embodiment of the present disclosure, wherein the output image does not have shadow variations due to 2D features (such as color, graphics, and tape);

[0013] Figure 6A yes Figure 1 The image of the box set, and Figure 6B Is using Figure 4 The system and Figure 5 The method removes the corresponding output image of the planar features;

[0014] Figure 7A yes Figure 2 The image of the box set, and Figure 7B Is using Figure 4 The system and Figure 5 The method removes the corresponding output image of the planar features;

[0015] Figure 8A It is an image of a flat-packaged collection, and Figure 8B Is using Figure 4 The system and Figure 5 The method removes the corresponding output image of the planar features. Detailed Implementation

[0016] The following discussion of embodiments of the present disclosure relating to methods and systems for removing 2D features from planar surfaces and highlighting 3D features in 2D images is merely exemplary in nature and is in no way intended to limit the disclosed apparatus and techniques or their application or use.

[0017] It is well known that camera images and / or sensor data are used as input to a wide variety of machine control systems. One known application is box depalletizing, where a large number of boxes are on a pallet, and the robot's task is to pick up one box at a time and move each box to a second position. Both two-dimensional (2D) image and data processing techniques and three-dimensional (3D) image and data processing techniques exist, and they are used to identify individual boxes in a stack as input to the box depalletizing operation. However, both these existing 2D and 3D techniques encounter difficulties in accurately analyzing images or data.

[0018] In 3D techniques using point cloud data, the points in the point cloud may be sparse, and some regions of the point cloud may experience "loss" due to missing data points. The resolution of 3D data can also be problematic; coarse resolution can cause errors in box shape recognition, while fine resolution can be slow and computationally intensive. In 2D image analysis techniques, text, images, color patterns, and other 2D features on the box surface can cause inaccurate box shape recognition, as shown in the figure below and discussed below.

[0019] Figure 1Image 100, a collection of boxes on a tray, illustrates how graphic patterns and colors on the boxes can make it difficult to distinguish one box from another in 2D image analysis. The subject in image 100 is a tray with two layers of boxes stacked—an upper layer 110 containing smaller boxes and a lower layer 120 containing larger boxes. The boxes in the upper layer 110 include numerous graphic designs, images, and text on their surfaces, which can make it difficult to identify box edges via 2D image analysis. In particular, dark bars 130 surrounded by brighter areas could be mistaken for box edges, as could straight transitions from a light to a dark background, as indicated at 140 and 150. Several other graphic features on the boxes in the upper layer 110 that could easily be mistaken for box edges in 2D image analysis are readily apparent. Techniques to overcome box edge misidentification (such as using multiple cameras) can increase complexity and have proven to be only partially effective.

[0020] Figure 2 Image 200 shows a collection of boxes on a pallet, illustrating the tape at the seams of the boxes and how the resulting reflections can make it difficult to distinguish one box from another in 2D image analysis. The subject in image 200 is a pallet with two layers of boxes stacked—including the upper layer 210, which will be the focus of this discussion. The boxes in the upper layer 210 are made of ordinary brown cardboard, without... Figure 1 Any graphic image of the middle box. However, the box in the upper layer 210 has transparent tape applied across its surface. Tape strip 220 is placed primarily across the middle portion of the cardboard panel on the box, away from the edge. Tape strip 230 is placed across the folded edge in the middle of the top of the box. And tape strip 240 is placed along the top edge of the box to press down the folded edge. Even with the use of transparent tape, any of tape strips 220, 230, and 240 can be identified as a possible box edge using 2D image analysis because the tape causes a significant change in the "color" (shadow or pixel intensity) of the box, and glare in the image increases bright reflections and makes it difficult to determine what the tape is covering.

[0021] Figure 1 and Figure 2Both illustrations demonstrate how 2D features (such as graphics and tape) on a planar surface (the top of a box) can interfere with box edge recognition using 2D image analysis. This disclosure provides methods and systems for processing images to remove 2D features (intensity variations due to color, tape, etc.) and highlight 3D features (such as actual box edges), thereby enabling subsequent image analysis to perform box recognition more robustly and reliably. Throughout this disclosure, when the technique is described as “decolorizing” the output image, this does not simply mean converting colors to grayscale levels as a monochrome camera would do. Rather, “decolorizing” means completely removing all intensity variations from the output image—as if there were no colors, markings, or 2D features on the subject in the first place.

[0022] Figure 3 This is a diagram illustrating the diffuse and specular reflection behavior from surface 310. Only specular reflection (such as glare) is visible when the normal vector of the reflective surface bisectes the vector from the surface to the light source and the vector from the surface to the observer. In other words, if incident light from source 312 strikes surface 310 along vector 320, specular reflection will be visible only to observer 314 who is viewing surface 310 along vector 322—where vectors 320 and 322 are symmetrically opposite each other across the surface normal.

[0023] Diffuse reflection has the same brightness to the observer regardless of the angle from which he / she views surface 310. For example, if a light source is incident on surface 310 at a low angle (almost parallel to the surface) along vector 330, low-brightness or low-intensity diffuse reflection will be visible from any angle relative to surface 310, as indicated by the dashed arrow 332. If a light source is incident on surface 310 at a high angle (almost perpendicular to the surface) along vector 340, high-brightness or high-intensity diffuse reflection will be visible from any angle relative to surface 310, as indicated by the long dashed arrow 342.

[0024] Most objects exhibit a combination of specular and diffuse reflection, which can be a problem to be solved in 2D image analysis. However, this disclosure provides a technique for capturing multiple images of a subject under specific, different lighting conditions and mathematically combining the images in a manner that eliminates unwanted reflective properties and highlights desired reflective properties. This technique is discussed in detail below.

[0025] Figure 4This is an illustration of a system 400 for acquiring and processing an image of a subject according to an embodiment of the present disclosure to produce an output image, which has no intensity variations due to 2D features such as color, graphics, and tape. The subject 410 is one or more objects that will be represented in an image or sensor data. Continuing with the examples used throughout the present disclosure, the subject 410 may be a collection of boxes on a tray. However, the subject 410 may be a single object (such as a box) or any other plurality of objects. The techniques of the present disclosure will work well in any embodiment of the subject 410, and particularly well when the subject 410 has a relatively flat top surface 412 or any other relatively flat surface in which 2D features such as color, graphics, and tape that need to be removed from the output image are present.

[0026] Figure 4 One implementation of the disclosed technology is described in some detail. Figure 4 The example implementation depicts a scene in which the subject 410 has a horizontal top surface 412, for which an image without intensity variations due to 2D features is required. Figure 4 The placement of the light source and camera is suitable for top-surface imaging scenarios. However, Figure 4 The scenarios described are merely examples to illustrate the disclosed techniques. Many other imaging scenarios are possible using the same disclosed techniques—such as imaging the vertical side surfaces of a subject, imaging surfaces at angles relative to the vertical and horizontal directions, etc. Some scenarios involve imaging more than one surface, more than two light sources, and combining the output images into a composite. These scenarios are discussed further below. Similarly, Figure 4 A specific, non-restrictive example is illustrated.

[0027] The first light source 420 and the second light source 422 are fixed at different positions, pointing downwards at an oblique angle towards the subject 410. The light sources 420 / 422 should not point vertically downwards onto the horizontal top surface 412, but should have an aiming angle between vertical and horizontal. In one embodiment, the light sources 420 / 422 have an elevation angle of 25° to 45° above the horizontal; higher or lower elevation angles are also suitable. In a preferred embodiment, both light sources 420 / 422 have the same elevation angle above the horizontal.

[0028] Light sources 420 / 422 must be located at different positions because the intention is to illuminate subject 410 differently in different images. In one embodiment, when viewed from directly above subject 410, light sources 420 / 422 have positions and aiming vectors approximately 90° apart in a top view. However, other relative position angles are also suitable—including light sources 420 / 422 being 180° apart in a top view (directly opposite each other). In one embodiment, light sources 420 / 422 are light-emitting diode (LED) light sources, but other types of light sources are also suitable.

[0029] In addition to light sources 420 / 422, the workspace where the subject 410 is located will typically have sources of ambient light. Ambient light can be a combination of, for example, artificial lighting (such as fluorescent lights in a warehouse) and natural sunlight. The presence and uncontrollability of ambient light are generally unavoidable, and therefore, currently disclosed techniques have been developed to acknowledge this fact and compensate for it, as discussed below.

[0030] A two-dimensional (2D) sensor 430 is fixed at another location (preferably different from the location of the light sources 420 / 422) and is configured to capture sensor data or images of the subject 410. In one embodiment, the sensor 430 is a digital camera that captures black-and-white (grayscale) images of the subject 410 with a resolution in the range of 1 million to 5 million pixels. Higher or lower resolutions may also be used. Color images may be used; however, grayscale images are suitable because the image arithmetic techniques discussed below operate on pixel intensity values. The data provided by the 2D sensor 430 will hereafter be referred to as an image; however, it should be understood that other types of 2D sensors and data may also be used. The top surface 412, which will be compensated in the image to remove 2D features including color effects, must be illuminated by both light sources 420 and 422, and the surface 412 must, of course, be within the field of view of the sensor 430.

[0031] Computer 440 receives a set of three images from sensor 430 for use in the disclosed image analysis techniques. Computer 440 can communicate wirelessly with sensor 430, or via a hardwired connection. Computer 440 can also control light sources 420 / 422. After subject 410 is in place, computer 440 can automatically control the acquisition of the three images—including capturing an image only in ambient light (hereinafter referred to as I...). A The image will be captured under ambient light conditions with light source 420 on and light source 422 off (hereinafter referred to as I1) and under ambient light conditions with light source 422 on and light source 422 off (hereinafter referred to as I2).

[0032] The following is a discussion of how to use image arithmetic to combine three images in a specific way to remove 2D features and enhance 3D features in a subject 410. Lambert's law of diffuse reflection can be written as:

[0033] I D =L· NCI L (1)

[0034] Among them I D I represents the intensity of diffuse reflected light. L Let L be the intensity of the incident light (assumed to be a point light source at infinity), L be the unit vector from the surface to the light source, N be the unit normal vector of the surface, and C be the reflectivity of the surface (related to color). It should be understood that Lambert's law of diffuse reflection assumes that all light rays originate from a distant point light source, such that they are all parallel to vector L and have a constant intensity I within the camera's field of view. L In fact, it has been found that the illumination can be made sufficiently close to satisfy these conditions, so that equation (1) can be usefully used in the image processing methods described below.

[0035] Equation (1) can be rewritten to help illustrate the image processing concept of this disclosure. The dot product of vector L and vector N is defined as L·N = ||L|||N||cosθ, where θ is the angle between the surface normal vector and the vector from the surface to the light source. Since L and N are unit vectors, both ||L|| and ||N|| are equal to 1, therefore L·N = cosθ. Substituting L·N into equation (1), we get I D =CI L cosθ. This form of the equation is known as Lambert's law of cosines.

[0036] Using Lambert's cosine law, the contribution of all ambient light sources to diffuse reflection in an image can be defined as:

[0037]

[0038] Among them I DA I represents the intensity of diffuse reflected light from all ambient light sources (number M). A,i Let θ be the intensity of the incident light from a specific ambient light source i, and θ be the intensity of the incident light. A,i Let be the angle between the surface normal vector and the vector from the surface to the specific ambient light source i.

[0039] Referring again to equation (1) rewritten as Lambert's cosine law, the contribution of the supplementary light source (light source 420 / 422) to the diffuse reflection light in the image can be defined as:

[0040] I D1 =CI L1 cosθ1 (3)

[0041] I D2 =CI L2 cosθ2 (4)

[0042] In equation (3), I D1 I represents the intensity of diffuse reflected light from supplementary light source 1 (light source 420). L1 Let θ1 be the intensity of the incident light from supplementary light source 1 (light source 420), and let θ1 be the angle between the surface normal vector and the vector from the surface to supplementary light source 1 (light source 420). Equation (4) is similarly defined for supplementary light source 2 (light source 422).

[0043] use Figure 4 The system 400, when capturing an image with one of the supplementary light sources (light source 420 or 422) lit, produces an image that includes contributions from diffuse light from both the supplementary light source and the ambient lighting. That is,

[0044] I D1,A = I D1 +I DA (5)

[0045] I D2,A = I D2 +I DA (6)

[0046] In equation (5), I D1,A Let I be the intensity of diffuse light from both supplementary light source 1 (light source 420) and ambient lighting; its intensity (temporally neglecting specular reflection) is image I1 captured using system 400. Similarly, for supplementary light source 2 (light source 422), equation (6) is defined, where I... D2,A Image I2 is represented by the image captured using system 400.

[0047] Because ambient light is inherently uncontrollable and can contain sources of unwanted variation, it is desirable to first subtract the ambient light contribution from each element in the supplementally illuminated image. Rewriting equations (5) and (6) by rearranging the terms yields the following:

[0048] I D1 = I D1,A -I DA (7)

[0049] I D2 = I D2,A -I DA (8)

[0050] Another objective of the currently disclosed technology is to completely eliminate intensity variations caused by color in the output image, in order to avoid... Figure 1 and Figure 2 The problem described herein is that the edges of the container are difficult to identify due to the color and other 2D features on the top surface of the container tray. The output image Q can be defined as follows:

[0051]

[0052] Then, substituting equations (3) and (4) into equation (9), we get:

[0053]

[0054] This simplifies to:

[0055]

[0056] The color component C does not appear in equation (11) because it has been canceled out from the numerator and denominator. Therefore, the image division technique in equations (9) to (11) eliminates color (that is, pixel intensity variations associated with different colors) from the output image Q.

[0057] Assuming light intensity value I L1 and I L2 If the constant value is within the image field of view, then equation (11) can be further simplified to:

[0058]

[0059] Where K is defined as I L1 / I L2 K is a constant value. In a preferred embodiment, light sources 420 and 422 are adjusted so that the value of K is close to 1.

[0060] Equation (12) clearly shows that the brightness variation between points in the output image Q is caused only by the variation of the normal vector at each point on the surface with the relative angles (θ1 and θ2) between the two light sources. On a flat surface, the normal vector is constant, so all pixels on a given flat surface have the same constant value. Pixels on different flat surfaces have different constant values ​​depending on the direction of the normal vectors of these planes.

[0061] It should be understood that all of the above equations are approximations, because light rays from real physical light sources do not all originate from the same far point, and therefore are not all parallel and do not all have the same intensity. However, in practice, these approximations have been found to be good enough to make plane-clearing methods useful.

[0062] Then, substituting equations (7) and (8) into equation (9), we obtain:

[0063]

[0064] As discussed above, neglecting specular reflection, I D1,A For image I1 (ambient lighting plus light source 420) captured using system 400, I D2,A Image I2 (ambient lighting plus light source 422) captured using system 400, and I DA The image was captured using system 400. A (Ambient lighting only). Therefore, the three images of the subject 410 captured by sensor 430 can be combined to produce an output image Q, which has no color or other 2D features:

[0065]

[0066] The above discussion describes how diffuse reflection components are handled in the calculation. Ambient specular reflections are also removed by image subtraction in equation (13). Specular reflections added by the supplementary light source are not removed, which is good, as those reflections help reveal the top edges and other 3D features of the subject 410. In the output image Q, the only remaining variations on the top surface 412 are due to changes in the angle between the surface normal at each point and the direction of the light source, resulting from variations in the 3D shape of surface 412 such as wrinkles, fold seams, and box edges.

[0067] Some of the equations (1) through (14) include image arithmetic—specifically, image subtraction and image division. Each of these operations is a pixel-by-pixel calculation using the pixel intensity values ​​of each corresponding pixel pair in the two images. The pixel intensity value for each pixel is an integer value within a predefined range (such as 0 to 255 or 0 to 4095). Other ranges of pixel intensity values ​​may be used.

[0068] For example, consider an example where each image is 1000 pixels by 1000 pixels (1000 rows × 1000 columns = 1 million pixels). If the intensity of pixel 1 (row 1, column 1) in the first image is 247 and the intensity of the corresponding pixel 1 in the second image is 133, then when the second image is subtracted from the first image, the intensity of pixel 1 in the resulting image is 114 (247 - 133). This calculation is performed for each pixel in each pair of images being subtracted. The same pixel-by-pixel calculation concept is applied to image partitioning, where the intensity value of a pixel in one image is divided by the intensity value of the corresponding pixel in the other image.

[0069] Image I AEach of I1 and I2 was taken using an equal exposure time. The exposure time was chosen to be low enough to ensure that the intensity values ​​of pixels with bright colors in any image were not clipped (that is, the maximum intensity value of all pixels in all images should be less than the maximum value, such as 255 or 4095).

[0070] Similarly, if the same processing is applied to the red, green, and blue channels respectively (Equation (14)), the same technique can be applied to color images to remove 2D features. However, using color will not provide any particular benefit, since the remaining 3D features in the image do not have distinct colors.

[0071] Using the techniques discussed above, system 400 can capture images of subject 410, which can be processed quickly to produce output images with colors and other 2D features removed. This system is particularly well-suited for subsequent processing (such as segmentation) to identify real box edges and corners and thereby determine the box shape and size necessary for robotic box depalletizing operations.

[0072] Figure 5 This is a flowchart 500 of a method for obtaining a subject image and processing the image to produce an output image according to an embodiment of the present disclosure, wherein the output image does not exhibit intensity variations due to 2D features such as color, patterns, and tape. At block 502, as provided... Figure 4 The workspace shown has a system 400 including light sources 420 / 422 and a 2D sensor 430. At least sensor 430 communicates with computer 440. Figure 5 Flowchart 500 illustrates a general image analysis method, and thus, Figure 4 The specific orientations illustrated in the diagram (horizontal top surface, light source, and downward aiming angle of the sensor / camera) should be considered merely non-limiting examples.

[0073] At frame 504, sensor 430 captures three images of subject 410—including the first image I. A (Under ambient light only), second image I1 (under ambient light plus light source 420), and third image I2 (under ambient light plus light source 422). These images can be acquired in a fully manual manner (where light source 420 / 422 and sensor 430 are manually controlled by switches / buttons), or in a fully automated manner (where the operation of light source 420 / 422 and sensor 430 is controlled by computer 440), or any combination thereof. Image I A I1 and I2 are provided to computer 440 for processing.

[0074] At box 506, image subtraction is performed indicatively in the numerator and denominator of equations (7) and (8) and equation (14) above. That is, the first intermediate image (representing diffuse reflection due to the first supplementary light source) is calculated as I1-I A And the second intermediate image (representing diffuse reflection due to the second supplementary light source) is calculated as I2-I A Subtracting the ambient image from images I1 and I2 eliminates the ambient diffuse and specular reflection components.

[0075] At box 508, image division is performed as indicated in equation (14) above. That is, one of the intermediate images calculated at box 506 is divided by the other intermediate image. Dividing the first supplementary lighting image by the second supplementary lighting image (after ambient subtraction) eliminates color and other 2D surface features from the output image Q. The division can be performed with either the first or the second supplementary lighting image in the numerator and the other in the denominator, since which of the light sources 420 / 422 corresponds to image I1 and which corresponds to image I2 is, after all, an arbitrarily defined question. This will be discussed again below.

[0076] At box 510, the output image Q is used for further processing or analysis. For example, the output image Q is particularly well-suited for “segmentation” calculations on box trays, where color and 2D features are removed from image Q while individual boxes are identified by their edges, as these edges are clearly visible in the output image Q.

[0077] It should be noted that, Figure 5 The disclosed method is described with very careful steps. In its practical implementation, the planar clearing technique can be performed in essentially two steps: capturing the image and then calculating the output image Q using equation (14). Intermediate images are not required at all when calculating the output image Q. Alternatively, the output image Q can be calculated directly by applying equation (14) (two subtractions followed by a division) to each pixel in a single step.

[0078] The techniques discussed above have been developed and demonstrated in numerous experiments, and have shown to be highly effective for removing color and 2D features from planar surfaces for a wide variety of subjects (particularly the top surface of a tray containing boxes of different styles and arrangements). Using the disclosed techniques, “removing color” from the output image does not simply mean converting colors to grayscale levels as would be done with a black-and-white camera. Rather, “removing color” means completely eliminating all intensity variations caused by color and 2D features from the output image—as if there were no color, markings, or 2D features on the subject in the first place. Three examples are shown and discussed below.

[0079] Figure 6A yes Figure 1 The image of the box set, and Figure 6B Is using Figure 4 The system and Figure 5 The method removes the corresponding output image of the planar features. Figure 6A This is what was discussed above. Figure 1 The image 100 is reproduced, wherein the upper layer 110 of the housing has a number of graphic design features on its surface, including dark stripes 130 and light / dark transitions 140 and 150.

[0080] Figure 6B Image 600 is created using the planar cleanup technique disclosed above. Specifically, image 600 is obtained from three input images I using equation (14). A The output image Q created by I1 and I2 contains the same upper layer 110 of the box. It is immediately apparent in image 600 that none of the graphic features (130, 140, and 150) of the box from image 100 are visible. Therefore, the problem of misidentifying box edges at 2D features 130, 140, and 150 has been resolved. In image 600, only 3D box features are visible on the top surface of the upper layer 110 of the box. Most of these 3D features are true box edges (610, 620, and 630), which are enhanced in visibility due to the image arithmetic techniques of this disclosure. Some small indentations in one of the boxes are also visible, as indicated at 640. Image 600 illustrates how the currently disclosed planar removal technique effectively eliminates 2D features and enhances 3D features in a subject. Clearly, for further analytical applications such as box segmentation, image 600 is far superior to image 100.

[0081] Figure 7A yes Figure 2 The image of the box set, and Figure 7B Is using Figure 4 The system and Figure 5 The method removes the corresponding output image of the planar features. Figure 7A This is what was discussed above. Figure 2 The reproduction of image 200, wherein the upper layer 210 of the box has a number of 2D features (brightness and reflection) associated with the transparent tape applied to its surface.

[0082] Figure 7B Image 700 is created using the planar cleanup technique disclosed above. Specifically, image 700 is obtained from three input images I using equation (14). AThe output image Q, created by I1 and I2, contains the same upper layer 210 of the box. In image 700, the disappearance of the transparent tape is immediately noticeable—both the light / dark effect and the specular reflection are gone. Figure 7A The location of the middle tape strip 220 is in Figure 7B None of them (just plain cardboard) are visible. Figure 7A The 230mm tape covers the seam of the folded panel. Figure 7B Only the folded joint 730 itself (which has a 3D groove shape) is visible. Figure 7A Where tape strips 240 are applied along the edges of the box, these box edges are Figure 7B It is visible at 740. The edges of the other boxes are... Figure 7B The differences are also clearly visible, for example, as indicated at 750, 760, and 770. Image 700 again illustrates how currently disclosed planar removal techniques effectively eliminate 2D features and enhance 3D features in a subject. Clearly, for further analytical applications (such as box segmentation), image 700 is far superior to image 200.

[0083] Figure 8A It is a flat-packaged collection of images 800, and Figure 8B Is using Figure 4 The system and Figure 5 The method clears the corresponding output image 850 of the planar features. Figure 8A and Figure 8B This provides another clear and compelling example of the effectiveness of the disclosed planar removal technique—where light / white and dark / black packaging appear identical in output image 850, and labels, markings, and other 2D features are absent in output image 850. Similarly, image 850 outperforms image 800 for further processing (such as packaging segmentation) (to identify the edges of these flattened packages).

[0084] The planar removal technique disclosed herein has been used to study many other arrangements of palletized stacking and packaging of cases, and they have produced equally compelling results in eliminating all 2D features (such as, but not limited to, print, graphics, labels, coloring, strapping, and tape) from the target surface of the case / packaging, while enhancing 3D features (such as actual case / packaging edges).

[0085] As mentioned earlier, the division operation in equation (14) can be performed if the numerator is either the first supplementary lighting image or the second supplementary lighting image and the denominator is the other. Reversing the numerator and denominator in equation (14) (or reversing the light sources associated with images I1 and I2) will have the effect of changing the intensity levels in the pixels of the output image Q, but in either case, the output image Q will still have no color effects and other 2D features, and only the 3D features will be noticeable on the top surface 412 of the subject 410. Figure 7B The output image Q is shown based on a specific demonstration workspace setup, with one supplementary light source associated with image I1 and another supplementary light source associated with image I2. If the light sources or image divisions were reversed, the effect would be that the near / left end of the cabinet (indicated by arrow 780) would be very bright rather than very dark, and the near / right side of the cabinet (indicated by arrow 790) would be very dark rather than very bright. Aside from the highly visible cabinet edges and fold seams, the top surface of the stacked cabinets would still appear substantially uniform.

[0086] exist Figure 6A / Figure 6B , Figure 7A / Figure 7B and Figure 8A / Figure 8B The striking results (cleared planar features and enhanced 3D features) are partly a result of the thoughtful arrangement of complementary light sources. For example, in Figure 7A / Figure 7B In this arrangement, the supplementary light sources will preferably be positioned such that the aiming vector of one of the light sources (e.g., light source 420) has a horizontal component perpendicular to one edge of the stacked housings, and the aiming vector of the other light source (e.g., light source 422) has a horizontal component perpendicular to the adjacent edge of the stacked housings (90° clockwise or counterclockwise when viewed from above). This arrangement provides the most striking bright reflections and dark shadows at the edges of the housings, as in... Figure 7B Visible in the image. Other supplementary light source arrangements can be better suited to other types of subjects—for example, in cases where the box has a shape or stacking arrangement that causes the edges of the box to be at an angle relative to the tray itself, the light source can be aligned with the aiming vector perpendicular to those diagonal edges.

[0087] As mentioned before, it has been provided Figure 4The system and the before-and-after images in Figures 6 to 8 clearly illustrate and demonstrate the effectiveness of the planar removal technique applied to a horizontal top surface. However, it should be emphasized that the planar removal technique is applicable to any relatively flat surface, not just a horizontal top surface. That is, the disclosed technique can be applied equally effectively to remove color and 2D features from vertical surfaces (e.g., the sides of a box) and inclined surfaces (the faces of oddly shaped or positioned objects that are neither horizontal nor vertical).

[0088] Consider, for example Figure 7A and Figure 7B The assembly of boxes shown is illustrated. The disclosed planar removal technique can be applied to produce images of the near / left end (indicated by arrow 780) of the boxes, but not the top surface, devoid of color and 2D features. To produce a planar-removed image of the near / left end of the box, it will only be necessary to provide two supplementary light sources, both illuminating the near / left end, capturing three images (I... A I1 and I2), and use equation (14) to calculate the output image Q. The same concept applies to other vertical and inclined surfaces.

[0089] continue Figure 7A and Figure 7B The example readily suggests how an output image of the entire cabinet stack could be generated, where all surfaces have been cleaned to remove intensity variations due to color and 2D features. This could be achieved by capturing an ambient light image (IA) by providing multiple supplementary light sources at different locations around the cabinet stack. A This is accomplished by capturing pairs of supplementary lighting images (I1 and I2) for each planar surface (e.g., top, near / left, and near / right). An output image Q can then be calculated for each of these three planar surfaces (top, near / left, and near / right) using the corresponding images, and a composite output image can then be generated using the planar-cleaned image Q for each of these three planar surfaces. It is easy to imagine how such fully planar-cleaned composite images would be most effectively used for 3D box segmentation of an entire box stack.

[0090] All the above discussion aims to eliminate color and 2D features from images of planar surfaces. Using more advanced lighting, image capture, and computational techniques, the same concept can be applied to eliminate color and 2D features from non-planar surfaces. While completely eliminating color from all surface orientation angles on arbitrarily shaped objects may be impractical, in some cases, the disclosed techniques can produce similar effects by reducing shadow variations caused by surface color without affecting shadow variations caused by 3D shape. Therefore, it can be concluded that the disclosed image processing techniques can be used to reduce shadow variations in output images of objects of any shape, and these techniques work particularly well for images of flat planar surfaces, where intensity variations in the image caused by color and 2D features can be completely eliminated.

[0091] The disclosed planar erasure technique describes a general imaging method. While the examples provided and discussed above illustrate the application of this method in box segmentation in industrial environments, many other applications are contemplated. For example, the disclosed technique can be applied to other computer vision and non-computer vision applications, including but not limited to:

[0092] • Visual surface inspection using computer vision

[0093] Imaging devices used to enhance the appearance of surfaces for human inspection.

[0094] • The artistic effect of photographic imaging

[0095] Special effects in film production

[0096] • The use of cameras and light sources employing radiation of invisible wavelengths (such as infrared or ultraviolet) to make the flash source invisible to human observers. • Outdoor imaging for camouflage removal or reduction.

[0097] -Corporate intelligence gathering: Imaging prototype cars and trucks painted with camouflage paint and removing or reducing the camouflage to provide clearer visibility of their shapes.

[0098] Military surveillance: Imaging camouflaged soldiers and vehicles outdoors using fixed or aerial drones equipped with infrared cameras and light sources, producing images that remove or reduce camouflage to enhance visibility.

[0099] Throughout the foregoing discussion, various image arithmetic operations have been described and implied. It should be understood that these calculations can be performed on a computer (such as...). Figure 4The software applications and modules of the computer 440 are executed. The computer 440, including one or more processors and memory along with input / output ports, receives and processes the input images from the sensor 430, provides the output image obtained from equation (14), and can also control the sensor 430 and the light sources 420 / 422. The computer 440 can also be a robot controller—that is, the controller of a robot performing a box depalletizing operation. In such an implementation, the sensor 430 provides its images / data directly to the robot controller performing image arithmetic, and the attached computer is not necessary.

[0100] As outlined above, the disclosed implementation of removing planar features from 2D images offers significant advantages over existing image processing methods and provides an output image of a subject that has essentially no 2D features (including color) on its top surface, wherein the output image is suitable for direct human viewing and / or for further computer processing, such as box segmentation applications.

[0101] While numerous exemplary aspects and implementations of methods and systems for removing 2D features from planar surfaces and highlighting 3D features in 2D images have been discussed above, those skilled in the art will recognize modifications, arrangements, additions, and sub-combinations thereof. Therefore, it is intended that the appended claims and the claims introduced herein be construed as including all such modifications, arrangements, additions, and sub-combinations within their true spirit and scope.

Claims

1. A method for removing two-dimensional features from an image, the method comprising: Provide a workspace with a single two-dimensional sensor in a fixed position and orientation, and a first supplementary light source and a second supplementary light source fixed at different positions from each other; The two-dimensional sensor provides a first input image of the subject under ambient lighting, a second input image of the subject under ambient lighting plus the first supplementary light source, and a third input image of the subject under ambient lighting plus the second supplementary light source; as well as The computer with a processor and memory calculates an output image by subtracting the first input image from the second input image to generate a first difference, subtracting the first input image from the third input image to generate a second difference, and dividing the first difference by the second difference to eliminate the two-dimensional features.

2. The method according to claim 1, wherein, The two-dimensional sensor is a two-dimensional camera.

3. The method according to claim 1, wherein, The subject has a flat surface, and the two-dimensional features are eliminated from the flat surface in the output image.

4. The method according to claim 3, wherein, The two-dimensional sensor is pointed vertically or at an angle toward the flat plane.

5. The method according to claim 3, wherein, The supplementary light source is directed at the flat plane at an angle.

6. The method according to claim 1, wherein, Subtracting the first input image from the second input image and subtracting the first input image from the third input image includes correspondingly subtracting pixel intensity values ​​pixel by pixel, and wherein dividing the first difference by the second difference includes correspondingly dividing by the pixel intensity values ​​pixel by pixel.

7. The method according to claim 6, wherein, Calculating the output image includes calculating a first intermediate image by subtracting a portion or all of the first input image from a corresponding portion or all of the second input image, calculating a second intermediate image by subtracting a portion or all of the first input image from a corresponding portion or all of the third input image, and then calculating the output image by dividing the first intermediate image by the second intermediate image.

8. A method for removing two-dimensional features from an image, the method comprising: A two-dimensional sensor provides a first input image of the subject under ambient lighting, a second input image of the subject under ambient lighting plus a first supplementary light source, and a third input image of the subject under ambient lighting plus a second supplementary light source; as well as The computer with a processor and memory calculates an output image by subtracting the first input image from the second input image to generate a first difference, subtracting the first input image from the third input image to generate a second difference, and dividing the first difference by the second difference to eliminate the two-dimensional features.

9. A system for removing two-dimensional features from an image of a subject, the system comprising: A two-dimensional sensor is positioned and in a fixed orientation, aiming at the subject. The first supplementary light source and the second supplementary light source are positioned at different fixed locations when aiming at the subject. as well as A computer, having a processor and memory, communicates with and is configured to communicate with the two-dimensional sensor. The two-dimensional sensor receives a first input image of the subject under ambient lighting, a second input image of the subject under ambient lighting plus the first supplementary light source, and a third input image of the subject under ambient lighting plus the second supplementary light source. as well as The calculation involves subtracting the first input image from the second input image to generate a first difference, subtracting the first input image from the third input image to generate a second difference, and dividing the first difference by the second difference to generate an output image that eliminates the two-dimensional features.

Images