Systems and methods for posture detection and measurement - Patents.com
The system uses a multi-camera setup with polarization capabilities and deep learning to improve pose estimation for objects with complex optical properties, addressing accuracy issues in robotic manipulation and assembly.
Patent Information
- Application Number
- JP2022546126
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-03-29
- Filing Date
- 2021-01-29
- Publication Date
- 2024-11-28
- Estimated Expiration
- 2041-01-29
AI Technical Summary
Existing technologies face challenges in accurately determining the pose of objects, particularly those with complex optical properties like transparency and symmetry, using conventional imaging methods, which complicates tasks such as robotic manipulation and assembly.
A system and method utilizing a multi-camera setup with polarization capabilities and deep learning techniques to estimate the pose of objects by analyzing polarization information and surface normals, refining the pose estimation through multi-view geometry and edge detection.
Enhances the accuracy of pose estimation for objects with complex optical properties, enabling precise robotic manipulation and assembly by accounting for polarization effects and object symmetry.
Smart Images

Figure 0007591577000027 
Figure 0007591577000028 
Figure 0007591577000029
Abstract
Description
[Technical field]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to and the benefit of U.S. Provisional Patent Application No. 62 / 967,487, filed in the United States Patent and Trademark Office on January 29, 2020, U.S. Provisional Patent Application No. 62 / 968,038, filed in the United States Patent and Trademark Office on January 30, 2020, and U.S. Provisional Patent Application No. 63 / 001,445, filed in the United States Patent and Trademark Office on March 29, 2020, the disclosures of which are incorporated herein by reference in their entireties.
[0002] Aspects of embodiments of the present disclosure relate to the field of sensors, and in particular to systems and methods for performing pose measurements, such as position and orientation, of objects within a scene. [Background technology]
[0003] Many areas of automation, such as robotics, use sensors to determine the physical relationships of real-world objects. For example, robotic systems often use sensing systems to measure the location of various physical objects, for example, to grasp an object that may be reached at various orientations, reorient the object to a desired position, and connect the object to another object. The position and orientation of an object relative to a reference coordinate system is sometimes called a "pose," and in a three-dimensional coordinate system generally includes six degrees of freedom: rotation about three axes and translation along three axes. Summary of the Invention
[0004] Aspects of embodiments of the present disclosure relate to the field of sensors, and in particular to systems and methods for performing pose measurements of objects in a scene.
[0005] According to one embodiment of the present disclosure, a method for estimating a pose of an object may include receiving a plurality of images of the object captured from a plurality of viewpoints relative to the object, initializing a current pose of the object based on calculating an initial estimated pose of the object from at least one of the plurality of images, predicting a plurality of 2D keypoints associated with the object from each of the plurality of images, and calculating an updated pose that minimizes a cost function based on a plurality of differences between the 2D keypoints and a plurality of 3D keypoints associated with a 3D model of the object positioned according to the current pose and projected onto each of the viewpoints.
[0006] Each of the multiple differences may correspond to a different one of the viewpoints, and each of the differences may be calculated, for a viewpoint of the viewpoints, based on a difference between a plurality of 2D keypoints associated with an image of the plurality of images corresponding to the viewpoint and a projection of 3D keypoints of the 3D model positioned in the current pose onto the viewpoint.
[0007] The cost function is
number
[0008] The cost function is
number
[0009] Predicting the multiple 2D keypoints may be performed using a deep learning keypoint detector.
[0010] The image may include polarization information, and the deep learning keypoint detector may be configured to calculate predicted 2D keypoints based on the polarization information of the image.
[0011] The deep learning keypoint detector can be trained based on the synthesized training data, which includes the synthesized polarization information and the synthesized keypoint labels.
[0012] The images may include polarization information, and the method may further include refining the updated pose by registering the 3D model with the polarization information.
[0013] Refining the updated pose may include detecting one or more edges of the object in the image to generate one or more measured edge images, detecting correspondences between rendered edge images of the 3D model positioned at the current pose of the one or more measured edges, and iteratively refining the updated pose using a multi-view perspective n-point based on the detected correspondences.
[0014] Edges may be detected by calculating a surface normal based on the polarization information and detecting edges of the object at discontinuities in the surface normal.
[0015] The edges may be detected by computing multiple instance segmentation maps from the image and detecting object edges at boundaries of object instances in the instance segmentation maps.
[0016] Refining the updated pose may include calculating a measured surface normal of the object based on the polarization information, placing a 3D model of the object based on the updated pose, and iteratively updating the pose of the 3D model to align the surface normal of the 3D model to the measured surface normal.
[0017] According to one embodiment of the present disclosure, a computer-implemented method for processing an image includes receiving, by a computing system, a first image and a second image; invoking, by the computing system, a neural network to detect a first instance of an object in the first image and a second instance of the object in the second image; determining a first region of the first instance of the object in the first image and a second region of the second instance of the object in the second image; determining a correspondence between the first instance of the object in the first region and the second instance of the object in the second region; and generating an output based on determining the correspondence.
[0018] The method may further include generating a first segmentation mask based on determining a first region of the first instance of the object, and generating a second segmentation mask based on determining a second region of the second instance of the object.
[0019] The method may further include, in response to determining a correspondence between the first instance of the object in the first region and the second instance of the object in the second region, confining a second processing task to the first region and the second region.
[0020] The second processing task may be to determine a correspondence between one or more pixels in the first region and one or more pixels in the second region.
[0021] The second processing task may be to determine a correspondence between one or more keypoints in the first region and one or more keypoints in the second region.
[0022] The output may be a measure of the disparity between the first image and the second image.
[0023] The output may be a three-dimensional relative pose between the first instance of the object and the second instance of the object.
[0024] The first image may be a first polarized image, and the first image may be captured by a polarized camera.
[0025] The first image may provide a first viewpoint of the object and the second image may provide a second viewpoint of the object that is different from the first viewpoint.
[0026] Determining correspondence between a first instance of an object in a first region and a second instance of an object in a second region may include identifying an epipolar line associated with the first instance of the object and constraining a search of the second image along the epipolar line.
[0027] According to one embodiment of the present disclosure, a method for computing a pose estimate of one or more objects in a scene includes capturing multiple images of one or more objects in the scene from multiple viewpoints, performing instance segmentation on the images to compute multiple instance segmentation maps of one or more detected objects in the scene, computing one or more depth estimates of the one or more objects by matching the instance segmentation among the instance segmentation maps, and estimating the pose of the one or more objects according to a multi-view joint optimization of multiple keypoints for each of the one or more detected objects based on the multiple images across the multiple viewpoints. The instance segmentation may be performed using the method of instance segmentation described in any of the paragraphs above.
[0028] Estimating the pose may include, for each of one or more detected objects in the scene, providing to the method according to any one of claims 1 to 12 a region of the image that corresponds to the same one of the detected objects.
[0029] According to one embodiment of the present disclosure, a system includes a main camera, one or more support cameras spaced apart from the main camera, where the main camera and the one or more support cameras have overlapping fields of view, and a pose estimator including a processor and a memory, where the memory stores instructions that, when executed by the processor, cause the processor to control the main camera, the one or more support cameras, and to perform any of the methods described above.
[0030] The main camera and one or more support cameras may form a passive camera system.
[0031] The system may further include an active projector configured to project structured light onto an area within the overlapping fields of view of the main camera and the support camera.
[0032] According to one embodiment of the present disclosure, a method for automatically calibrating a camera system as described above includes receiving a plurality of images of an object within overlapping fields of view of a main camera and one or more support cameras, the images being captured from a plurality of viewpoints relative to the object; initializing a current pose of the object based on calculating an initial estimated pose of the object from at least one of the plurality of images; predicting a plurality of 2D keypoints associated with the object from each of the plurality of images; and calculating an updated pose of the object and updated viewpoints of the main camera and one or more support cameras that minimize a cost function based on a plurality of differences between the 2D keypoints and a plurality of 3D keypoints associated with a 3D model of the object positioned according to the current pose and projected onto each of the viewpoints.
[0033] The cost function is
number
[0034] The accompanying drawings illustrate exemplary embodiments of the invention and, together with the description, serve to explain the principles of the invention.
[0035] [Figure 1A] FIG. 1 is a schematic diagram depicting a pose estimation system according to one embodiment of the present disclosure. [Figure 1B] A high-level description of how light interacts with transparent and non-transparent (e.g., diffuse and / or reflective) objects. [Figure 1C] FIG. 1 is a perspective view of a polarized camera module according to an embodiment of the present disclosure. [Figure 1D] 1 is a cross-sectional view of a portion of a polarization camera module according to one embodiment of the present disclosure. [Figure 1E] FIG. 1 is a perspective view of a stereo polarized camera system according to one embodiment of the present disclosure. [Figure 2A] 1 is a flowchart depicting a method for estimating pose according to one embodiment of the present disclosure. [Figure 2B] 1 depicts four views of a scene containing an arrangement of objects captured by a camera of a pose estimation system according to one embodiment of the present disclosure. [Figure 2C] 1 is a depiction of 2D keypoints detected in four different views of the same instance of an object from four different viewpoints captured by a camera of a pose estimation system, according to one embodiment of the present disclosure. [Figure 2D] A crop depicting a single object in the scene extracted from an image captured from one of the cameras, and the same crop with overlapping renderings of a 3D model of the object posed in the scene according to an estimated pose of the single object calculated by pose estimator 100 according to one embodiment of the present disclosure. [Figure 2E]The present invention depicts an image of a scene including multiple objects captured from one of the cameras, and the objects with overlapping renderings of 3D models of the objects posed in the scene according to the estimated pose of the objects detected by a pose estimator 100 according to an embodiment of the present disclosure, the pose being due to the pose estimator 100 according to an embodiment of the present disclosure. [Diagram 3] 1 is a flowchart depicting a method for estimating an object pose based on multiple views of the object, according to one embodiment of the present disclosure. [Figure 4] 1 is a flowchart of a method for edge-based refinement of pose according to one embodiment of the present disclosure. [Diagram 5] 1 is a flowchart illustrating a method for surface normal-based refinement of pose based on polarization data, according to one embodiment of the present disclosure. [Figure 6] FIG. 1 is a block diagram depicting a system for training statistical models to perform computer vision tasks based on images of various modalities, the training being performed using data generated by an embodiment of the present disclosure. [Figure 7A] This is an image, or intensity image, of a scene in which one real transparent ball is placed on top of a photographic printout depicting another scene containing two transparent balls (the "spoofs") and some background clutter. [Figure 7B] Figure 7A shows the intensity image overlaid with a segmentation mask computed by a comparative mask region-based convolutional neural network (Mask R-CNN) to identify instances of transparent balls. The real transparent ball is correctly identified as an instance, and the two spoofs are incorrectly identified as instances. [Figure 7C] 4 is a polarization angle image calculated from captured polarization raw frames of a scene, according to one embodiment of the present invention; [Figure 7D]7B depicts the intensity image of FIG. 7A overlaid with a segmentation mask computed using polarization data in accordance with one embodiment of the present invention, where the real transparent ball is correctly identified as an instance and the two spoofs are correctly rejected as instances. [Figure 8] 1 is a graph of transmitted versus reflected light energy over a range of angles of incidence onto a surface having a refractive index of approximately 1.5. [Figure 9] 1 is a flowchart depicting a pipeline for generating a synthetic wizard according to one embodiment of the present disclosure. [Figure 10] FIG. 1 is a schematic diagram of sampling a real material from multiple angles using a polarization camera system according to one embodiment of the present disclosure. [Figure 11] 1 is a flowchart depicting a method for generating a training data set according to one embodiment of the present disclosure. [Figure 12] FIG. 1 is a flow diagram of a process for object-level support according to one embodiment. [Figure 13] FIG. 2 is a block diagram of an architecture for step instance segmentation and mask generation according to one embodiment. [Figure 14] FIG. 12 is a more detailed flow diagram of the matching algorithm employed in step 1204 (FIG. 12) to identify object-level correspondences for particular object instances in the first segmentation mask, according to one embodiment. [Figure 15] 2C is an instance segmentation mask generated for image 211-L of FIG. 2B according to one embodiment. [Figure 16A-16B] 13 is an image depicting the results of the process for determining object-level correspondence described with respect to FIG. 12, according to one embodiment. [Figure 17] 1 is a schematic illustration of camera auto-calibration according to one embodiment of the present disclosure. [Figure 18] FIG. 1 is a schematic diagram illustrating that the accuracy and repeatability of a robot arm is a function of both the robot's pose and position. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0036] In the following detailed description, only certain exemplary embodiments of the present invention are shown and described by way of example. As one skilled in the art will recognize, the present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein.
[0037] Pose estimation generally refers to computer vision techniques for estimating or predicting the location and orientation of an object. Some forms of pose estimation refer to detecting the physical pose of a person, such as the position and orientation of the person's head, arms, legs, and joints. Pose estimation may also refer more generally to the position and orientation of various animate or inanimate physical objects in a scene. For example, a robot navigating autonomously may maintain information about the physical pose of surrounding objects to avoid collisions and predict the trajectories of other moving objects. As another example, in robotics used in manufacturing, pose estimation can be used by a robot to detect the location and orientation of a physical manufacturing part so that the robot arm can approach the part from the correct angle to properly grasp the part for assembly with other parts of the product (e.g., grasping the head of a screw to screw the screw into a hole, whereas grasping the tip of the screw would make it difficult to insert the screw into the hole).
[0038] Aspects of an embodiment of the present disclosure relate to systems and methods for automatic six degree of freedom (6DoF) estimation of a wide variety of objects in a scene. The six degrees of freedom in three-dimensional space include position coordinates (e.g., x, y, and z translation coordinates in a three-dimensional global coordinate system) and orientation coordinates (e.g., θ, φ, and ψ rotation coordinates in the three-dimensional coordinate system).
[0039] Some aspects of the embodiments of the present disclosure relate to specific techniques for performing 6DoF estimation to calculate the pose of a single object or multiple objects in a cluttered box of these objects. Many objects have different bidirectional reflectance distribution functions (BRDFs), making it difficult to achieve a single framework for 6DoF estimation using comparative methods such as depth capture. To address this, some aspects of the embodiments of the present disclosure relate to calculating pose estimates or measurements using simulation, physics-based learning, and multi-view geometry to estimate the pose of any object imaged by an imaging system according to an embodiment of the present disclosure.
[0040] 1A is a schematic diagram depicting a pose estimation system 1 according to one embodiment of the present disclosure. As shown in FIG 1A, a main camera 10 is positioned such that a field of view 12 of the main camera 10 captures a disposition 20 of objects 22 in a scene. In the embodiment shown in FIG 1A, the main camera 10 is positioned above a support platform 40 (e.g., spaced away from the objects 22 along the direction of gravity), although embodiments of the present disclosure are not limited thereto, and for example, the pose estimator may be positioned such that it has a downward angled field of view of the objects 22.
[0041] In some embodiments, one or more support cameras 30 are positioned at different poses around a scene including the arrangement 20 of objects 22. Thus, each of the support cameras 30, e.g., a first support camera 30a, a second support camera 30b, and a third support camera 30c, captures a different view of the object 22 from a different viewpoint (e.g., a first viewpoint, a second viewpoint, and a third viewpoint, respectively). Although FIG. 1A shows three support cameras 30, embodiments of the present disclosure are not limited thereto and may, for example, include at least one support camera 30 and may include four or more support cameras 30.
[0042] Support platform 40 may be any surface that supports object 22, such as a bench top, a table top, a conveyor belt, etc. Support platform 40 is not required and object 22, or an object 22 whose pose may be estimated by the pose estimation system, may be supported by other devices, such as a movable arm of a robot, an overhead conveyor system, etc.
[0043] The pose estimator 100 according to various embodiments of the present disclosure is configured to calculate or estimate the pose of the object 22 based on information captured by the main camera 10 and the support camera 30 .
[0044] More specifically, the main camera 10 and the support camera 30 are configured to estimate the pose of an object 22 detected within their field of view 12. In the embodiment shown in Fig. 1A, the object 22 is abstractly depicted as a simple three-dimensional solid, such as a sphere, a rectangular prism, and a cylinder, although embodiments of the present disclosure are not so limited and characterization of the pose estimator may be performed using any object for which a pose relative to the camera can be clearly defined.
[0045] In particular, "pose" refers to the position and orientation of an object relative to a reference coordinate system. For example, a reference coordinate system can be defined with the main camera 10 as the origin, the direction along the optical axis of the main camera 10 (e.g., the direction passing through the center of the field of view 12 of the main camera 10) is defined as the z-axis of the coordinate system, and the x-axis and y-axis are defined to be mutually orthogonal and orthogonal to the z-axis. (Embodiments of the present disclosure are not limited to this particular coordinate system, and one skilled in the art will understand that pose can be mathematically transformed to an equivalent representation in a different coordinate system.)
[0046] Each object 22 may also be associated with its own corresponding coordinate system, defined with respect to its particular shape. For example, a right angle prism with unequal side lengths may have a canonical coordinate system defined, where the x-axis is parallel to its shortest direction, the z-axis is parallel to its longest direction, the y-axis is orthogonal to the x-axis and z-axis, and the origin is located at the center of gravity of the object 22.
[0047] Generally, in a three-dimensional coordinate system, object 22 has six degrees of freedom: rotation about three axes (e.g., rotation about the x-axis, y-axis, and z-axis) and translation along three axes (e.g., translation along the x-axis, y-axis, and z-axis). For clarity, symmetry of object 22 is not discussed in detail herein, but may be addressed, for example, by identifying multiple possible poses for different symmetries (e.g., when selecting positive versus negative orientations of the z-axis for a right circular cylinder) or by ignoring some rotational components of the poses (e.g., a right circular cylinder is rotationally symmetric about its axis).
[0048] In some embodiments, it is assumed that a three-dimensional (3D) model or computer-aided design (CAD) model is available that represents a canonical or idealized version of each type of object 22 in the arrangement 20 of objects. For example, in some embodiments of the present disclosure, the objects 22 are individual instances of manufactured parts that have a substantially uniform appearance from one part to the next. Examples of such manufactured parts include screws, bolts, nuts, connectors, and springs, as well as specialty parts such as electronic circuit components (e.g., packaged integrated circuits, light-emitting diodes, switches, resistors, etc.), labware (e.g., test tubes, PCR tubes, bottles, caps, lids, pipette tips, sample plates, etc.), and manufactured parts (e.g., handles, switch caps, light bulbs, etc.). Thus, in these situations, a CAD model that defines the ideal or canonical shape of any particular object 22 in the arrangement 20 can be used to define a coordinate system for the object (e.g., the coordinate system used in the representation of the CAD model).
[0049] Based on the reference coordinate system (or camera space, for example, defined with respect to the pose estimation system) and the object coordinate system (or object space, for example, defined with respect to one of the objects), the pose of the object can be considered as a rigid transformation (rotation and translation) from object space to camera space. The pose of object 1 in camera space 1 is
number
number
number
number
[0050] If two objects, object A and object B, are in the same camera C coordinate frame, let P be the pose of object A relative to camera C. CA is used to denote the pose of object B relative to camera C, and P CB For convenience, we assume that the object pose is expressed based on a reference coordinate system, so that the poses of objects A and B relative to the camera space C are P A and P B It can be expressed as:
[0051] If object A and object B are in fact the same object, but taken during different pose estimation measurements, and the residual pose P err or P AB (P AB =P err ) to set the posture P A From posture P B To show the conversion to P A P err =P B (1) The following relationship must hold:
number
[0052] Ideally, the pose estimate P A and P B Assuming that the object has not moved (e.g., translated or rotated) relative to the pose estimator 10 between the measurements of P A and P B should both be the same, and P err is the identity matrix (i.e., indicating that there is no error between poses)
number
[0053] In a similar manner, the pose of a particular object can be calculated for views from two different cameras. For example, an image of object A captured by a main camera C can be used to calculate the pose P of object A relative to the main camera C. CA Similarly, using the image of object A captured by the first support camera S1, the pose of object A relative to the support camera S1 can be calculated as
number
number
[0054] Ideally, assuming that the known relative poses of the main camera C and the support camera S1 are accurate, and the pose calculated based on the data captured by the two cameras is accurate, then P CA and
number
number
[0055] The difference P between the actual measured values and the identity matrix, calculated based on the estimates calculated by the attitude estimator 10 err is the error R err =||R(P err )|| (3) T err =||T(P err )|| (4) where R err is the rotation error, T err is the translation error. The function R( ) is err to axis angles, where the magnitude is the rotational difference, and the function T( ) extracts the translational component of the attitude matrix.
[0056] The axis angle expression from the rotation matrix R is given by the following formula: Tr(R)=1+2cosθ (5)
number
[0057] Some aspects of embodiments of the present disclosure relate to computing a high accuracy pose estimate of an object 22 in a scene based on a joint estimation of the object's pose across the main camera 10 and the support camera 30. Some aspects of embodiments of the present disclosure also relate to calibrating a camera system based on computing a joint estimation of the pose of an object 22 in a scene.
[0058] Pose Estimation Hardware In the embodiment shown in FIG. 1A, the pose estimation system 1 includes a main camera 10. and one or more support cameras 30. In some embodiments of the present disclosure, the main camera 10 includes a stereo camera. Examples of stereo cameras include a camera system having at least two monocular cameras spaced apart from one another along a baseline, where the monocular cameras have overlapping fields of view and optical axes that are substantially parallel to one another. Although embodiments of the present disclosure are presented herein in embodiments where the main camera 10 and the support camera 30 are passive cameras (e.g., cameras that are not connected to a dedicated light projector, but instead use ambient lighting or other light sources), embodiments of the present disclosure are not limited thereto and may also include situations where one or more active lighting projectors are included in the camera system, thereby forming an active camera system where the active lighting projectors may be configured to project structured light or patterns onto a scene. The support camera 30 may be a stereo camera, a monocular camera, or a combination thereof (e.g., several stereo support cameras and several monocular support cameras).
[0059] The main camera 10 and the support camera 30 can use the same or different imaging modalities. Examples of imaging modalities include monochrome, color, infrared, ultraviolet, thermal, polarized light, and combinations thereof.
[0060] 1B, 1C, 1D, and 1E provide an example of a polarized camera system that may be used as part of a main camera 10 according to some embodiments of the present disclosure.
[0061] The interactions between light and transparent objects are rich and complex, but the material of the object determines how transparent the object is under visible light. In many transparent household objects, most of the visible light passes straight through and a small portion (about 4% to about 8%, depending on the refractive index) is reflected. This is because light in the visible part of the spectrum does not have enough energy to excite the atoms in the transparent object. As a result, the texture (e.g., appearance) of the object behind (or visible through) the transparent object dominates the appearance of the transparent object. For example, when viewing a transparent glass cup or tumbler on a table, the appearance of the object on the other side of the tumbler (e.g., the surface of the table) generally dominates what is seen through the cup. This property creates some problems when trying to detect surface features of transparent objects, such as glass windows and layers of shiny transparent paint, based on intensity images alone.
[0062] FIG. 1B is a high-level depiction of the interaction of light with transparent and opaque (e.g., diffuse and / or reflective) objects. As shown in FIG. 1B, polarization camera 10 captures a polarized raw frame of a scene including a transparent object 41 in front of an opaque background object 42. A light ray 43 that strikes the image sensor 14 of polarization camera 10 contains polarization information from both the transparent object 41 and the background object 42. In contrast to the light 45 that is reflected from the background object 42 and passes through the transparent object 41, a small portion of the light 44 reflected from the transparent object 41 is strongly polarized and therefore has a large impact on the polarization measurement.
[0063] Similarly, light rays that strike an object's surface may interact with the shape of the surface in a variety of ways. For example, a surface with a glossy paint may behave substantially similarly to a transparent object in front of an opaque object, as shown in FIG. 1B, where the interaction between the light rays and the transparent or translucent layer (or clear coat layer) of the glossy paint causes the light reflecting from the surface to be polarized based on the characteristics of the transparent or translucent layer (e.g., based on the layer thickness and surface normal), which are encoded into the light rays that strike the image sensor. Similarly, as discussed in more detail below with respect to shape from polarization (SfP) theory, changes in the shape of a surface (e.g., the direction of the surface normal) may cause significant changes in the polarization of light reflected by the object's surface. For example, a smooth surface may generally exhibit the same polarization properties throughout, but scratches or dents in the surface may change the direction of the surface normal in those areas, and light that strikes the scratches or dents may be polarized, attenuated, or reflected differently than other parts of the object's surface. Models of the interaction between light and matter generally consider three fundamental elements: geometry, illumination, and matter. Geometry is based on the shape of the material. Illumination includes the direction and color of the light. Materials can be parameterized by the refractive index or angular reflection / transmission of light. This angular reflection is known as the Bidirectional Reflectance Distribution Function (BRDF), although other functional forms may more accurately represent certain scenarios. For example, a Bidirectional Subsurface Scattering Function (BSSRDF) would be more accurate in the context of materials that exhibit subsurface scattering (e.g., marble or wax).
[0064] A light ray 43 that strikes the image sensor 14 of the polarization camera 10 has three measurable components: the intensity of the light (Intensity Image / I), the percentage or fraction of the light that is linearly polarized (Degree of Linear Polarization / DOLP / ρ), and the direction of that linear polarization (Angle of Linear Polarization / AOLP / φ). These properties encode information about the surface curvature and material of the imaged object that can be used by the pose estimator 100 to detect transparent objects, as described in more detail below. In some embodiments, by using one or more polarization cameras, the pose estimator 100 can detect other optically challenging objects based on similar polarization properties of light passing through semi-transparent objects and / or interacting with multi-path guiding or non-reflective objects (e.g., matte black objects).
[0065] More specifically, polarization camera 10 may further include a polarizer or polarizing filter or polarizing mask 16 positioned in the optical path between scene 1 and image sensor 14. According to various embodiments of the present disclosure, polarizer or polarizing mask 16 is configured to enable polarization camera 10 to capture images of scene 1 with the polarizer set at various specified angles (e.g., 45° rotation or 60° rotation or non-uniformly spaced rotations).
[0066] As an example, FIG. 1B depicts an embodiment in which the polarizing mask 16 is a polarizing mosaic aligned to the pixel grid of the image sensor 14 in a manner similar to the red-green-blue (RGB) color filters (e.g., Bayer filters) of a color camera. In a manner similar to the way that a color filter mosaic filters incident light based on wavelength so that each pixel in the image sensor 14 receives light in a particular portion of the spectrum (e.g., red, green, or blue) according to the pattern of the color filters of the mosaic, the polarizing mask 16 using the polarizing mosaic filters light based on linear polarization so that different pixels receive light at different angles of linear polarization (e.g., 0°, 45°, 90°, and 135°, or angles of 0°, 60°, and 120°). Thus, a polarizing camera 10 using a polarizing mask 16 as shown in FIG. 1B can capture light in four different linear polarizations in parallel or simultaneously. One example of a polarizing camera is the Blackfly® S polarized camera manufactured by FLIR® Systems, Inc. of Wilsonville, Oregon.
[0067] Although the above description relates to some possible implementations of a polarization camera using a polarization mosaic, embodiments of the present disclosure are not limited thereto and encompass other types of polarization cameras capable of capturing images with multiple different polarizations. For example, the polarization mask 16 can have less than four polarizations or more than four different polarizations, or can have polarizations at angles different from those described above (e.g., at polarization angles of 0°, 60°, and 120°, or at polarization angles of 0°, 30°, 60°, 90°, 120°, and 150°). As another example, the polarization mask 16 can be implemented using an electronically controlled polarization mask, such as an electro-optic modulator (which can include, for example, a liquid crystal layer), in which the polarization angles of individual pixels of the mask can be independently controlled such that different portions of the image sensor 14 receive light with different polarizations. As another example, the electro-optical modulator can be configured to transmit light of different linear polarizations when capturing different frames, for example, such that the camera captures images with sequentially different linear polarizer angles across the polarizing mask (e.g., 0 degrees, 45 degrees, 90 degrees, or 135 degrees). As another example, the polarizing mask 16 can include a mechanically rotating polarizing filter such that different polarized raw frames are captured by the polarizing camera 10 and the polarizing filter is mechanically rotated relative to the lens 12 to transmit light to the image sensor 14 at different polarization angles. Additionally, while the above examples relate to the use of a linear polarizing filter, embodiments of the present disclosure are not limited thereto and include the use of a polarized camera that includes a circular polarizing filter (e.g., a linear polarizing filter with a quarter wave plate). Thus, in various embodiments of the present disclosure, the polarized camera uses a polarizing filter to capture multiple polarized raw frames with different polarizations of light, such as different linear polarization angles and different circular polarizations (e.g., handedness).
[0068] As a result, polarization camera 10 captures multiple input images (or polarization raw frames) of a scene that includes the surface of object 22. In some embodiments, each of the polarization raw frames is imaged at a different polarization angle φ pol(e.g., 0 degrees, 45 degrees, 90 degrees, or 135 degrees). Each of the polarized raw frames is captured from substantially the same pose with respect to the scene 1 (e.g., images captured with a polarizing filter at 0 degrees, 45 degrees, 90 degrees, or 135 degrees are all captured by polarized camera 10 at the same location and orientation), as opposed to capturing the polarized raw frames from different locations and orientations with respect to the scene. Polarized camera 10 may be configured to detect light in various different portions of the electromagnetic spectrum, such as portions of the electromagnetic spectrum that are visible to the human eye, red, green, and blue portions of the spectrum that are visible to the human eye, and non-visible portions of the electromagnetic spectrum, such as infrared and ultraviolet.
[0069] FIG. 1C is a perspective view of a polarized camera module according to an embodiment of the present disclosure. FIG. 1D is a cross-sectional view of a portion of a polarized camera module according to an embodiment of the present disclosure. Some aspects of the embodiments of the present disclosure relate to a polarized camera module in which multiple polarized cameras (e.g., multiple cameras each having a polarized filter in its optical path) are arranged adjacent to one another in an array and can be controlled to capture images in groups (e.g., a single trigger can be used to control all cameras in the system to capture images in parallel or substantially simultaneously). The polarized filters in the optical path of each of the cameras in the array cause differently polarized light to reach the image sensor of the camera. The individual polarized cameras in the camera system have optical axes that are substantially perpendicular to one another, are placed adjacent to one another, and have substantially the same field of view but different polarizations such that the cameras in the camera system capture substantially the same view of the scene 1. In some embodiments, the individual polarized cameras are arranged such that the parallax shift between the cameras is substantially negligible based on the designed working distance of the camera system to objects in the scene, and if the designed working distance is large, a larger spacing between the cameras may be tolerated. In some embodiments of the present disclosure, the polarized camera module includes at least three polarized cameras, each having a polarized filter with a different polarization state (e.g., each being linearly polarized at a different angle, such as 0°, 60°, and 120°).
[0070] For example, in the embodiment of the polarized camera module 10' shown in FIG. 1C, four cameras 10A', 10B', 10C', and 10D' are arranged in a 2×2 grid to form a camera array, with the four cameras having substantially parallel optical axes. The four cameras can be controlled together to capture images substantially simultaneously and using the same exposure settings (e.g., the same aperture, exposure length, and gain or "ISO" settings). In various embodiments of the present disclosure, each of the separate cameras 10A', 10B', 10C', and 10D' includes a different polarizing filter.
[0071] FIG. 1D shows two cross-sectional views of the polarization cameras 10A' and 10B' shown in FIG. 1C. As seen in FIG. 1D, each polarization camera (10A' and 10B') system includes a corresponding lens, a corresponding image sensor, and a corresponding polarization filter. Specifically, the polarization camera 10A' includes a lens 12A', an image sensor 14A', and a polarization filter 16A'. Similarly, the polarization camera 10B' includes a lens 12B', an image sensor 14B', and a polarization filter 16B'. In some embodiments of the present disclosure, the image sensor, the four cameras 10A', 10B', 10C', and 10D', are monolithically formed on the same semiconductor die, and the four cameras are located in the same housing with separate openings for the lenses 12 corresponding to the different image sensors. Similarly, the polarization filters 16 may correspond to different portions of a single physical layer, with different polarization filters (e.g., different linear polarization angles) in different regions of the layer (corresponding to different cameras).
[0072] In some embodiments of the present disclosure, each of the cameras in the camera system 10' has a corresponding polarizing filter configured to filter light polarized differently. For example, in the embodiment shown in Figures 1C and 1D, the polarizing filter 16A' of the camera 10A' may be a linear polarizing filter oriented at an angle of 0°, the polarizing filter 16B' of the camera 10B' may be a linear polarizing filter oriented at an angle of 45°, the polarizing filter 16C' of the camera 10C' may be a linear polarizing filter oriented at an angle of 90°, and the polarizing filter 16D' of the camera 10D' may be a linear polarizing filter oriented at an angle of 135°. In some embodiments, one or more of the cameras may include a circular polarizer. In some embodiments of the present disclosure, the camera system 10' includes polarizing filters configured to filter light with at least two different polarizations. In some embodiments of the present disclosure, the camera system 10' includes polarizing filters configured to filter light with at least three different polarizations. 1D, the polarizing filter 16 is located behind the lens 12 (e.g., between the lens 12 and the image sensor 14), although embodiments of the present disclosure are not limited thereto. In some embodiments, the polarizing filter is located in front of the lens 12.
[0073] In some embodiments, the various individual cameras of the camera array are registered with each other by determining their relative pose (or relative position and orientation) by capturing multiple images of a calibration target such as a checkerboard pattern, an ArUco target (see, e.g., Garrido-Jurado, Sergio, et. al., “Automatic generation and detection of highly reliable fiducial markers under occlusion” Pattern Recognition 47.6(2014):2280-2292.), or a ChArUco target (see, e.g., An, Gwon Hwan, et al., “Charuco board-based omnidirectional camera calibration method” Electronics 7.12(2018):421.). In particular, the process of calibrating the targets may include calculating intrinsic matrices that characterize the internal parameters of each camera (e.g., matrices that characterize the focal length, image sensor format, and principal point of the camera), and extrinsic matrices that characterize the pose of each camera with respect to world coordinates (e.g., matrices for implementing a transformation between the camera coordinate space and the world or scene coordinate space).
[0074] Although not shown in FIG. 1D, in some embodiments of the present disclosure, each polarization camera may also include a color filter having a mosaic pattern, such as a Bayer filter, such that individual pixels of image sensor 14 receive light corresponding to, for example, the red (R), green (G), and blue (B) portions of the spectrum, such that each camera captures light in the visible portion of the electromagnetic spectrum according to the mosaic pattern. In some embodiments, demosaicing is used to calculate separate red, green, and blue channels from the raw data. In some embodiments of the present disclosure, each polarization camera may be used without a color filter or with filters used to transmit or selectively transmit various other portions of the electromagnetic spectrum, such as infrared light.
[0075] FIG. 1E is a perspective view of a stereo polarized camera system according to an embodiment of the present disclosure. In some applications, stereo vision techniques are used to capture multiple images of a scene from different viewpoints. As mentioned above, in some embodiments of the present disclosure, the individual polarized cameras in the camera system are placed adjacent to each other such that the parallax shift between the cameras is substantially negligible based on the camera system's designed working distance to the object being imaged. In a stereo polarized camera system, some of the individual polarized cameras are spaced apart so that the parallax shift is large and objects within the camera system's designed working distance can be detected. This allows the distance to various surfaces in the scene ("depth") to be detected according to the magnitude of the parallax shift (e.g., a larger parallax shift in the location of corresponding parts of the image indicates that those corresponding parts are on a surface closer to the camera system, and a smaller parallax shift indicates that the corresponding parts are on a surface farther from the camera system). These techniques for calculating depth based on parallax shift are sometimes referred to as Depth from Stereo.
[0076] Accordingly, FIG. 1E illustrates a stereo polarized camera system 10″ having a first polarized camera module 10-1″ and a second polarized camera module 10-2″ having substantially parallel optical axes and spaced apart along a baseline 10-B. In the embodiment illustrated in FIG. 1E, the first polarized camera module 10-1″ includes polarized cameras 10A″, 10B″, 10C″, and 10D″ arranged in a 2×2 array similar to that illustrated in FIGS. 1C and 1D. Similarly, the second polarized camera module 10-2″ includes polarized cameras 10E″, 10F″, 10G″, and 10H″ arranged in a 2×2 array such that the entire stereo polarized camera module 10″ includes eight individual polarized cameras (e.g., eight separate image sensors behind eight separate lenses). In some embodiments of the present disclosure, the corresponding polarized cameras of the polarized camera modules 10-1″ and 10-2″ are configured to capture polarized raw frames having substantially the same polarization. For example, cameras 10A'' and 10E'' may both have linear polarizing filters at the same 0° angle, cameras 10B'' and 10F'' may both have linear polarizing filters at the same 45° angle, cameras 10C'' and 10G'' may both have linear polarizing filters at the same 90° angle, and cameras 10D'' and 10H'' may both have linear polarizing filters at the same 135° angle.
[0077] The embodiments of the present disclosure are not limited to the specific embodiment shown in Figure 1E. In some embodiments, the stereo polarized camera system includes three or more polarized camera modules, each of which includes a plurality of polarized cameras arranged in an array and configured to capture polarized raw frames of different polarizations using polarizing filters. As described above, in some embodiments of the present disclosure, one or more of the individual polarized cameras of the polarized camera module can include a color filter, and therefore one or more of the polarized cameras in the stereo polarized camera module can also include a color filter.
[0078] In a similar manner as described for calibrating or registering the cameras within a camera module, the various polarized camera modules of a stereo polarized camera system can also be registered with each other by capturing multiple images of a calibration target and calculating intrinsic and extrinsic matrices for the various camera modules.
[0079] Although the embodiment of the stereo polarized camera system shown in FIG. 1E includes two polarized camera modules each having four polarized cameras, embodiments of the present disclosure are not so limited.
[0080] For example, in some embodiments of the present disclosure, a stereo polarized camera system includes a plurality of polarized camera modules, each of which includes three or more individual polarized cameras, each of which has a polarized filter with a different polarization state (e.g., linear polarization at different angles).
[0081] In some embodiments of the present disclosure, a stereo polarized camera system includes multiple polarized camera modules spaced apart along one or more baselines, each of the polarized camera modules including a single polarized camera configured to capture polarized raw frames with different polarizations, according to an embodiment such as that described above with respect to FIG. 1B. For example, in some embodiments of the present disclosure, the polarized camera of each module may include a polarized mask (e.g., similar to the polarized mask shown in FIG. 1B), such that each individual polarized camera captures images in which pixels detect light according to a mosaic pattern of different polarized filters (e.g., polarized filters at different angles). For example, in the embodiment shown in FIG. 1B, each 2×2 block of pixels of the polarized mask includes linear polarizers with linear polarization angles of 0°, 45°, 90°, and 135°. In other embodiments of the present disclosure, each individual polarized camera may include mechanically or electronically controllable polarized filters, as discussed above with respect to FIG. 1B, to enable the polarized camera to capture polarized raw frames with different polarizations.
[0082] Although the above embodiments describe individual polarized camera modules or polarized cameras spaced apart along one or more baselines in a stereo polarized camera system having substantially parallel optical axes, embodiments of the present disclosure are not so limited. For example, in some embodiments of the present disclosure, the optical axes of the polarized camera modules are angled toward each other such that the polarized camera modules provide different angled views of objects within a designed working distance (e.g., where the optical axes meet or intersect near the designed working distance from the stereo camera system).
[0083] According to various embodiments of the present disclosure, the pose estimator 100 is implemented using one or more processing or electronic circuits configured to perform various operations described in more detail below. Types of electronic circuits may include central processing units (CPUs), graphic processing units (GPUs), artificial intelligence (AI) accelerators (e.g., vector processors that may include vector arithmetic logic units configured to efficiently perform operations common to neural networks, dot products, softmaxes, etc.), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), digital signal processors (DSPs), etc. For example, in some circumstances, aspects of embodiments of the present disclosure are implemented by program instructions stored in non-volatile computer readable memory that, when executed by electronic circuits (e.g., CPUs, GPUs, AI accelerators, or combinations thereof), perform the operations described herein to compute processed outputs 20, such as instance segmentation maps, from input polarized raw frames 18. The operations performed by pose estimator 100 may be performed by a single electronic circuit (e.g., a single CPU, a single GPU, etc.) or may be allocated among multiple electronic circuits (e.g., multiple GPUs, or CPUs in combination with GPUs). The multiple electronic circuits may be local to each other (e.g., located on the same die, located in the same package, or located in the same embedded device or computer system) and / or remote from each other (e.g., may communicate over a network such as a local personal area network such as Bluetooth, over a local area network such as a local wired and / or wireless network, and / or over a wide area network such as the Internet, such as when some operations are performed locally and other operations are performed on servers hosted by a cloud computing service).One or more electronic circuits that operate to implement the pose estimator 100 may be referred to herein as a computer or computer system, which may include memory that stores instructions that, when executed by the one or more electronic circuits, implement the systems and methods described herein.
[0084] Pose Estimation Based on Multi-view Camera System Some aspects of the embodiments of the present disclosure relate to estimating the pose of an object in a scene using a system of multiple independent cameras with a large baseline to perform simultaneous pose estimation. Some aspects of the embodiments of the present disclosure relate to pose estimation that identifies corresponding object instances in an instance segmentation map by applying techniques including passive keypoint matching from multiple views (e.g., from the main camera 10 and the support camera 30) and reprojecting the keypoints to different viewpoints (e.g., corresponding to different support cameras 30). Instance segmentation can then be used to detect more keypoints associated with the same object. These new keypoints can then be used to estimate the pose of the object and further refine the estimated pose according to a multi-view perspective n-point (PnP) algorithm according to embodiments of the present disclosure. Some embodiments also relate to using these additional viewpoints corresponding to different support cameras 30 to refine depth and edge predictions to refine the pose of the object based on iterative closest point (ICP) and shape from polarization, as described in more detail below.
[0085] 2A is a flowchart depicting a method 200 for estimating pose according to one embodiment of the present disclosure. In various embodiments of the present disclosure, the operations of the method 200 shown in FIG. 2A may be performed by one or more processing or electronic circuits that are components of the pose estimator 100.
[0086] In operation 210, the pose estimator 100 controls the main camera 10 and one or more support cameras 30 to obtain a set of multi-viewpoint images or multi-view calibrated images of the scene. In particular, the set of images is multi-viewpoint or multi-view in the sense that the main camera and the support cameras capture images of the scene from different viewpoints. Some individual cameras can capture images from multiple viewpoints, such as in the case of a stereo main camera and / or a stereo support camera that can capture images from at least two viewpoints. For example, the main camera 10 according to the embodiment shown in FIG. 1E can capture eight different images of the scene substantially simultaneously and from substantially two different viewpoints by the first polarized camera module 10-1″ and the second polarized camera module 10-2″. In general, the main camera 10 and the support camera 30 capture overlapping views of the same scene, noting that some objects may be visible from some viewpoints but not from other viewpoints due to occlusion between different objects in the scene, thereby providing different views of substantially the same object or objects in the scene.
[0087] FIG. 2B depicts four views of a scene including an arrangement of objects captured by the cameras of a pose estimation system according to an embodiment of the present disclosure. In the embodiment of FIG. 2B, the four images are captured by two stereo cameras (e.g., the stereo cameras of the main camera 10 and the support camera 30). The first stereo camera captures a left image 211-L and a right image 211-R, and the second stereo camera captures a left image 212-L and a right image 211-R, and the four images depict overlapping views of the same scene. In the particular scene shown in FIG. 2B, a collection of objects is placed in the ChArUco board, which provides a fiducial for providing additional information about the camera pose.
[0088] A set of images is referred to herein as "calibrated" based on the assumption that the system of cameras, including the main camera 10 and one or more support cameras 30, have been calibrated with respect to each other, e.g., the relative poses of the cameras are known and stored by the pose estimator 100. Some aspects of embodiments of the present disclosure relate to automatic calibration of camera systems and are described in more detail in a separate section below.
[0089] In some embodiments of the present disclosure, the pose estimator 100 further performs stereo rectification on the images received from the main camera and one or more support cameras. Stereo rectification may include transforming images to project images from different viewpoints onto the same image plane. For example, the images captured by the first polarized camera module 10-1″ and the second polarized camera module 10-2″ may be inherently rectified due to intrinsic and extrinsic camera parameters. For example, lens distortion, imperfect physical alignment of the optical axes of the camera modules and / or between the main camera and the support camera, etc. may be corrected by the pose estimator 100 based on known calibration data collected during a calibration operation before proceeding to further processing.
[0090] In operation 230, the pose estimator 100 performs instance segmentation on the multi-view image to compute a separate instance segmentation map for each view captured by the main camera 10 and the support camera 30. Instance segmentation is a computer vision approach that identifies, for every pixel in an input image, an instance that belongs to an object as well as the type or class of that object. For example, in an image of 3D geometric blocks of a rectangular prism, a cylinder, and a sphere, each "instance" corresponds to a different block, and the instance segmentation map would label each pixel with a different block identifier and block type (prism, cylinder, or sphere).
[0091] One class of approaches to performing instance segmentation on an input image is to feed the input image to a convolutional neural network (CNN) that is trained to compute an instance segmentation map from the input image. Examples of image segmentation CNNs include Mask R-CNN (He, Kaiming, et al., "Mask R-CNN," in Proceedings of the IEEE International Conference on Computer Vision 2017), AlexNet (see, e.g., Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton, "ImageNet classification with deep convolutional neural networks," Advances in neural information processing systems. 2012), VGG (see, e.g., Simonyan, Karen, and Andrew Zisserman, "Very deep convolutional networks for large-scale image recognition," arXiv preprint arXiv:1409.1556 (2014)), and ResNet-101 (see, e.g., Kaiming He, Xiangyu Zhang, and Shaoqing, "Very deep convolutional networks for large-scale image recognition," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016). Ren, Jian Sun, and others. "Deep residual learning for image recognition"). MobileNet (see, for example, Howard, Andrew G., et al. "Mobilenets: Efficient convolutional neural networks for mobile vision applications" arXiv preprint arXiv:1704.04861 (2017)), MobileNetV2 (see, for example, Sandler, Mark, et al. "MobileNetV2: Inverted residuals and linear bottlenecks" in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018), and MobileNetV3 (see, for example, Howard, Andrew, et al. "Searching for MobileNetV3" in Proceedings of the IEEE International Conference on Computer Vision. 2019).
[0092] In many cases, the performance computed by a convolutional neural network (e.g., the accuracy of a segmentation map) is significantly improved when it is trained or retrained on data representative of the target domain in which it operates.
[0093] Accordingly, certain aspects of embodiments of the present disclosure relate to systems and methods for training a machine learning model, such as a convolutional neural network, to compute an instance segmentation map for a particular use case.
[0094] Furthermore, as mentioned above, some types of materials are difficult to image using standard imaging modalities such as color cameras. However, using a polarization camera system can make these objects and surfaces detectable. Some aspects of the embodiments of the present disclosure relate to performing instance segmentation using polarization cues captured according to the embodiments of the present disclosure. Some techniques for performing instance segmentation using polarization cues are described in more detail in U.S. Provisional Patent Application No. 62 / 942,113, filed November 30, 2019, United States Patent and Trademark Office, U.S. Provisional Patent Application No. 63 / 001,445, filed March 29, 2020, United States Patent and Trademark Office, and PCT U.S. Patent Application No. 2020 / 048604, filed August 28, 2020, United States Patent and Trademark Office, the entire disclosures of which are incorporated herein by reference.
[0095] These aspects of embodiments of the present disclosure relating to training convolutional neural networks to perform instance segmentation are described in more detail in subsequent sections.
[0096] Multiple instance segmentation maps are computed by performing instance segmentation on images captured from each viewpoint of the camera system, including the main camera 10 and the support camera 30. Each instance segmentation map labels distinct instances of the object 22 that are visible in the captured image.
[0097] In operation 250, the pose estimator 100 matches the instance segmentations across multiple stereo images to achieve an approximate depth estimate for each object. As mentioned above, in some embodiments, at least the main camera 10 is a stereo camera configured to capture stereo pair images. Because the cameras (e.g., camera modules) of the main camera are relatively close, the stereo pair of images are similar in that the overall shapes of the object instances detected between the images are relatively similar and the relative locations of the detected instances of the object with respect to each other are approximately the same. Thus, in some embodiments, the depth estimate of each object instance imaged by the stereo pair is calculated by identifying corresponding instances detected in separate instance segmentation maps and measuring the disparity (e.g., position shift) of the location of the instance in the segmentation between the two images in a manner similar to depth from disparity using block matching. Details of the system and method for automatically performing correspondence matching between instances are described in more detail in the following section on object-level correspondence using segmentation.
[0098] In operation 270, the pose estimator 100 computes keypoints for the detected objects. Specifically, each instance of each object detected by the instance segmentation process is extracted (e.g., cropped) from the corresponding images captured by the main camera 10 and the stereo camera 30 to generate a collection of image patches or crops (along with the coordinate locations where they were extracted from the original images). These image patches are fed to a keypoint detector to compute multiple keypoints for each object detected in the scene. Each keypoint detected in an image patch can be expressed in two-dimensional coordinates [u,v], such as the coordinate space of the image captured from the viewpoint of the corresponding camera (e.g., the main camera 10 or the support camera 30).
[0099] Examples of keypoint or feature detectors that can be used in operation 270 include, but are not limited to, scale invariant feature transform (SIFT), speed up robust features (SURF), gradient location and orientation histogram (GLOH), histogram of gradient orientation (HOG), basis coefficients, Haar wavelet coefficients, etc.
[0100] FIG. 2C is a depiction of 2D keypoints detected in four different views of the same instance of an object from four different viewpoints captured by cameras of a pose estimation system, according to one embodiment of the present disclosure. In the embodiment shown in FIG. 2C, an object (e.g., a manufactured part) is viewed by two stereo cameras, Stereo 1 and Stereo 2, each having a left camera and a right camera. Thus, FIG. 2C shows crops depicting the same object from left image 271-L and right image 271-R captured by camera Stereo 1, and crops depicting the same object from left image 272-L and right image 272-R captured by camera Stereo 2. Each dot in FIG. 2C represents a 2D keypoint detected by the keypoint detector, and the number next to the dot represents a keypoint identifier, such that in the case of perfect keypoint detection, the same keypoint identifier across different views represents the same physical location on the surface of the object.
[0101] In some embodiments of the present disclosure, keypoints in an image are detected using a trained keypoint detector, such as a trained convolutional neural network, which is described in more detail in another section below regarding training convolutional neural networks. When applying a trained keypoint detector, each keypoint detected or predicted in an image may be associated with a corresponding confidence level (e.g., the probability that a given point is a keypoint).
[0102] Additionally, as described above, in some embodiments of the present disclosure, one or more of the main camera 10 and the support camera 30 are polarization cameras configured to capture polarization information of the scene. The polarization information enables detection of optically challenging surfaces of an object (e.g., surfaces that may be optically difficult to detect with a comparative color camera), such as transparent surfaces, highly reflective surfaces, and matte black surfaces. Thus, in some embodiments, a trained keypoint detector is trained on polarization data captured from the scene (e.g., polarization raw frames, polarization cues, and / or polarization features in the polarization representation space) to detect keypoints on optically challenging surfaces of an object based on the polarization information. Training a keypoint detector based on polarization data is described in more detail in another section below.
[0103] In operation 290, the pose estimator 100 performs a multi-view joint optimization over key points of an object across multiple views of the object captured by the main camera 10 and the support camera 30 to compute an estimated pose of the object in the scene.
[0104] In some embodiments, the pose estimator 100 independently computes a pose for each detected object in the scene based on distinct keypoints detected for each object detected according to instance segmentation.
[0105] In operation 290, the pose estimator 100 performs multi-view object matching based on the image of the current detected object among the detected objects. In some embodiments, the pose estimator uses keypoint prediction and stereo constraints to estimate object correspondences using a simple Hungarian matching algorithm (e.g., linear sum assignment), and uses a custom cost function to calculate object poses according to a multi-view perspective n-point algorithm according to embodiments of the present disclosure.
[0106] Multi-view perspective n points In some embodiments, the pose estimator 100 is configured to detect and estimate poses for a predefined specific type of object. For example, in the case of a pose estimator 100 configured for use in manufacturing robotics, the pose estimator 100 may be configured to detect the poses of various end effectors of a robot arm and the poses of various parts (e.g., screws, panels, cases, etc.) that are assembled into a product. Thus, the pose estimator 100 stores a set of predefined keypoints for each type of object for which it is configured to detect and calculate poses, where these keypoints correspond to characteristic three-dimensional points on the surface of the object. The i-th 3D keypoint among the plurality of N 3D keypoints associated with a particular object is herein defined as its Cartesian coordinate [x i , y i ,z i As an example, some key points on a screw may correspond to the tip of the screw, the edge of the head of the screw, etc.
[0107] In operation 290, the pose estimator 100 finds predefined 3D keypoints ([x i ,y i ,z i ]), and a rotation transformation R with respect to a global coordinate system (e.g., a coordinate system defined with respect to the main camera 10) based on the 2D keypoints ([u,v]) detected in each view of the object. o and the translation transformation T o We estimate the current object pose, denoted by
[0108] More specifically, in some embodiments, the pose estimator 100 calculates a pose R that minimizes the error or difference across all viewpoints (from the main camera 10 and the support camera 30) between the detected 2D keypoints and the predefined 3D keypoints when the 3D keypoints are projected into the 2D space associated with each viewpoint. o , T o More precisely, in some embodiments, the pose estimator 100 computes a cost function,
number
[0109] More specifically, for each 3D keypoint i and each viewpoint j, the i-th 3D keypoint is assigned a current estimated object pose [R o T o ] and the camera extrinsic parameters R of viewpoint j are j T j (e.g., the relative pose between the cameras of viewpoint j with respect to the global coordinate system) and the camera intrinsic parameters K j(e.g., internal properties of the camera at viewpoint j, such as lens distortion) into the 2D space of viewpoint j, so that the pose estimator 100 estimates the i-th 2D keypoint (keypoint [u ij ,v ij ]) to the corresponding i-th 3D keypoint [x i ,y i ,z i The error or cost is summed over all N distinct 3D keypoints and over all M distinct viewpoints (e.g., one main camera and M−1 support cameras), and the pose estimator 100 applies an optimization function to find the estimated object pose R that minimizes the overall cost or overall error. o T o An example of an optimization function is to apply gradient descent to find the object pose R that minimizes this error. o T o In some embodiments, the optimization is performed across all viewpoints simultaneously, rather than iteratively considering each viewpoint one at a time.
[0110] FIG. 2D depicts a crop depicting a single object in the scene extracted from an image captured from one of the cameras, and the same crop with an overlapping rendering of a 3D model of the object posed in the scene according to an estimated pose of the single object calculated by pose estimator 100 in accordance with one embodiment of the present disclosure.
[0111] 2E depicts an image of a scene including multiple objects captured from one of the cameras, and the objects with overlapping renderings of the objects' 3D models posed in the scene according to the estimated poses of the objects detected by the pose estimator 100 according to an embodiment of the present disclosure, and the poses are due to the pose estimator 100 according to an embodiment of the present disclosure. In the particular case shown in FIG. 2E, the pose estimator 100 has not estimated poses for all objects in the scene, because in some embodiments, objects that are widely occluded (e.g., by other objects) or parts of the scene that do not meet other criteria (e.g., do not match other views well) may be left unanalyzed by the pose estimator 100, but may have their poses estimated later as the objects shift in the scene and / or disappear from the scene.
[0112] Symmetry awareness in keypoint detection. For symmetric objects, for a given pose of the object, P o =S*P o There exists a 4×4 transformation S such that: One example of a symmetric object is a cube, which has eight-way symmetry. This means that for any of these eight given poses, the cube will look the same from a given viewpoint. However, because keypoints are computed (or predicted) deterministically, a specific 2D point on an image captured from one viewpoint can be matched to a specific 3D point on the object's surface. When viewed from another viewpoint, that 2D point may match to a different 3D point on the object's surface. This means that the detected keypoints may not be consistent across images from different viewpoints.
[0113] Therefore, some embodiments of the present disclosure provide a method for determining the pose R of an object. o , T o We define a cost function for estimating (e.g., Eq. 7 above) for a particular view vj With respect to modifying the cost function with an additional term that defines a set of transformations (S) that performs a search for a transformation S that minimizes the reprojection error (e.g., the error associated with reprojecting the i-th 3D keypoint from viewpoint j into 2D space) on x, which allows embodiments of the present disclosure to determine a minimum pose even when the same 3D keypoints are labeled differently in different 2D keypoint views. Equation 8 below provides an example of a cost function modified to account for object symmetry.
number
[0114] In some embodiments, the pose estimator 100 calculates an estimated pose R o , T o Iteratively refine for each viewpoint. Figure 3 is a flowchart depicting a method 300 according to one embodiment of the present disclosure for estimating the pose of an object based on multiple views of the object. The process may begin with computing an estimated pose of the object based on images captured by the main camera 10. Because the main camera 10 is a depth camera (e.g., a stereo camera), the pose estimator 100 can compute the depth of surfaces in the scene based on disparity, and thus the depth of surfaces of instances of the object detected in the scene.
[0115] In some embodiments, the pose estimator 100 aligns a 3D model (e.g., a CAD model) of the detected object with the portion of the depth map that corresponds to the instance of the object to generate an initial estimated pose (R o , T o ) to determine
[0116] In operation 330, pose estimator 100 selects the next viewpoint from among the remaining viewpoints (eg, from among the viewpoints corresponding to the supporting cameras) and sets that viewpoint as the "current viewpoint."
[0117] In operation 350, the pose estimator 100 calculates an updated pose (R o , T o ) to minimize a cost function (e.g., Equation 7 or Equation 8 above). Because the expected pose of the object is expected to be close to the current estimate of the pose (e.g., the initial pose of the object calculated in operation 310), in some embodiments the minimization process is limited to searching a local window of values, which improves the performance of the optimization process.
[0118] In operation 390, the pose estimator 100 determines whether there are additional viewpoints to consider. If there are additional viewpoints, the process returns to operation 330 to select the next keypoint and update the estimate based on the 2D keypoints detected from the next viewpoint. If there are no additional keypoints, the current estimate of the pose is updated based on the estimated pose (R o , T o )
[0119] In some embodiments, the combination of the main camera 10 and the support camera 30 includes multiple stereo pairs (e.g., where the support camera 30 includes at least one stereo pair of cameras). In some of these embodiments, the entire segmentation, keypoint detection, and multi-view perspective n-point processing (e.g., of FIG. 2) is performed independently for each stereo pair, and then the pose estimator 100 matches objects based on 3D locations and epipolar lines across the multiple cameras. This can result in a trade-off between more accurate matching at a larger distance but a higher computational load, which can lead to longer execution times.
[0120] Polarization-based refinement Some aspects of embodiments of the present disclosure relate to further refining the pose estimate calculated in operation 290 based on polarization information. This process may be referred to herein as polarization-based multi-view edge-based refinement, where one or more of the main camera 10 and / or support cameras 30 are polarization cameras configured to capture polarized raw frames of objects in the scene.
[0121] As discussed above with respect to FIG. 1B, a light ray 43 striking the image sensor 14 of the polarization camera 10 has three measurable components: the intensity of the light (Intensity Image / I), the percentage or fraction of the light that is linearly polarized (Degree of Linear Polarization / DOLP / ρ), and the direction of that linear polarization (Angle of Linear Polarization / AOLP / φ).
[0122] To measure the intensity I, DOLPρ, and AOLPφ at each pixel, we first measure the intensity I at each pixel by observing it at different angles φ behind a polarizing filter (or polarizer). pol Three or more polarization raw frames of the scene captured at φ are needed (e.g., because there are three unknown values to be determined: intensity I, DOLPρ, and AOLPφ. For example, a polarization camera such as that described above with respect to Figures 1B, 1C, 1D, and 1E can capture four different polarization angles φ pol, for example, capture polarized raw frames at 0 degrees, 45 degrees, 90 degrees, or 135 degrees, thereby defining I0, I 45 , I 90 , and I 135 Four polarized raw frames shown as
number
[0123]
number
number
[0124] Therefore, four different polarized raw frames
number
[0125] Shape from polarization (SfP) theory (see, for example, Gary A Atkinson and Edwin R Hancock, Recovery of surface orientation from diffuse polarization, IEEE transactions on image processing, 15(6):1653-1664, 2006) determines the refractive index of the object's surface normal (n), the azimuthal angle (θ a ), and the zenith angle (θ z ) and the φ and ρ components of the light coming from the object, if diffuse reflection is dominant, the following property is obtained:
number
number
number
[0126] In either case, ρ is θ z Note that it increases exponentially as θ increases, and for the same refractive index, specular reflection is much more polarized than diffuse reflection.
[0127] Accordingly, some aspects of embodiments of the present disclosure relate to applying SfP theory to detect or measure surface gradients (e.g., surface orientation, or surface normal, or direction perpendicular to the surface) based on raw polarization frames of the object captured by the polarization cameras of the main camera 10 and support camera 30, and to update the estimated pose of the object by aligning an existing 3D model (e.g., a CAD model) of the object with the measured surface normal of the object.
[0128] Leveraging polarized light, the pose estimator 100 can accurately detect geometry and albedo edges of surfaces in the images. In situations where the pose estimator 100 stores a textured 3D model (or CAD model) of the detected object, the pose estimator 100 also determines geometry and albedo-based edges for a given object pose estimated by the multi-view pose estimation process discussed above (e.g., method 300). To refine the pose, the pose estimator 100 perturbs the pose in a manner similar to the iterative closest point (ICP) algorithm until the edges of the 3D model are aligned with the object's shape (e.g., its edges and surface normals) calculated from the images.
[0129] More specifically, FIG. 4 is a flowchart of a method for pose-based edge-based refinement according to one embodiment of the present disclosure.
[0130] In operation 410, pose estimator 100 computes a multiview edge image from a given multiview input image. The multiview edge image corresponds to the detected edges of the current object (the object whose pose is being estimated). These edges may include the outer boundary of the object relative to the rest of the scene (e.g., relative to the background, or relative to other objects detected or not detected in the scene) and may include internal edges, such as the edges of a cube in a frontal view from the viewpoint of the image.
[0131] In some embodiments of the present disclosure, edges of an object are detected based on shape from polarized light, as described above, where the edges correspond to the locations of discontinuities in the calculated surface normal. These discontinuities can be calculated by applying an edge detection filter (such as a Canny edge detector or a differential edge detector, but embodiments of the present disclosure are not limited thereto). For example, in some embodiments of the present disclosure, a deep convolutional neural network is trained to detect edges in the input image and / or features in the polarization representation space (e.g., intensity, AOLPφ, and DOLPρ). Details of training the neural network based on polarization data are described in more detail in another section below.
[0132] In some embodiments of the present disclosure, a neural network trained to compute a segmentation map based on input images provided by the main camera 10 and the support camera 30 computes its output segmentation map with accurate and precise boundaries of shapes. More specifically, some aspects of the embodiments of the present disclosure relate to training a deep learning model to perform semantic segmentation with sharp (e.g., low noise) and accurate boundaries in the instance segmentation map. Some techniques for performing semantic boundary prediction are described in "Devil is in the edges: Learning semantic boundaries from noisy annotations" by Acuna, David, Amlan Kar, and Sanja Fidler in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition 2019. By training such deep learning models using labeled polarization information (e.g., polarization raw frames and / or features in polarization representation space such as intensity, AOLPφ, and DOLPρ), embodiments of the present disclosure enable accurate detection of boundaries of optically challenging object instances in addition to objects that are detectable according to images captured by a comparison color camera.
[0133] Thus, in some embodiments of the present disclosure, edges in an image are detected (e.g., calculated directly or indirectly based on the image) based on the locations of boundaries of instances in a segmentation map corresponding to the image.
[0134] In some embodiments of the present disclosure, the pose estimator 100 detects edges based on a combination of shape from polarization and instance segmentation, as discussed above.
[0135] In operation 430, the pose estimator 100 detects a correspondence between the measured edge images detected in operation 410 from the multi-viewpoint images of the object with the rendered edges of a 3D model of the object placed in the virtual scene according to the current pose estimate. More specifically, the current pose estimate R o , T o The pose estimator 100 then renders edge images of the 3D model from viewpoints associated with the cameras (e.g., main camera 10 and support camera 30) as well as the depth of each pixel in the rendered scene (e.g., distance from the virtual camera at the pose corresponding to the corresponding viewpoint of the corresponding real camera) and searches for correspondence (within a distance threshold) between edge pixels in the rendered and measured edge images.
[0136] In operation 450, pose estimator 100 estimates a new pose based on the correspondences (e.g., based on minimizing the position difference between the correspondences as described above using a multi-view perspective n-point similar to that described above with respect to FIG. 3) and evaluates the new pose based on a loss calculated using a cost function (e.g., based on a Gaussian pyramid, performing a multi-scale evaluation of the quality of the alignment between the edges based on the current estimated pose and the measured edge positions). If the new pose is better (e.g., has less loss than the previous best pose according to the cost function), the new pose is stored as the current best pose.
[0137] In operation 490, the pose estimator 100 determines whether the change in the cost function (or current loss) is less than a threshold (e.g., 10 -4If the pose has not converged, the process returns to operation 430 to render images of the 3D model of the object based on the current best pose, and calculates a new pose based on these rendered images in operation 450. If the pose has converged, the current best pose R o , T o is returned as the refined pose of the object.
[0138] The following table presents, as pseudocode, the edge alignment process described above with respect to FIG. Table: Pseudocode for edge alignment prev_loss=inf P_0=initial_pose# (e.g., from the pose estimation pipeline in Figure 3) converged=false While not converged: all_correspondences=[] j counts up from view_edges to edge_images: rendered_edge_pixels=render_edges(Cam_j,P_i-1,cad_model) rendered_depth=render_depth(Cam_j,P_i-1,cad_model) potential_correspondences=[] Until (x,y) is rendered_edge_pixels: correspondences=find_corr(x,y,view_edges,distance_threshold) Until (x2,y2) become correspondences: potential_correspondences.append([x2,y2,depth2xyz(rendered_depth[x,y])]) all_correspondences.append(potential_correspondences) best_loss=inf best_pose=None Until i becomes iters: correspondences=random.sample(all_correspondences) pose_new=multi_view_pnp(correspondences) Loss=0 j counts up from view_edges to edge_images: rendered_edge_pixels=render_edges(Cam_j,P_new,cad_model) Loss+=GaussianPyramid(view_edges,rendered_edge_pixels) If Loss < best_loss: best_loss=Loss best_pose=pose_new If best_loss-prev_loss<1e-4: converged=true otherwise, prev_loss=best_loss
[0139] 5 is a flow chart illustrating a method 500 for surface normal-based refinement of pose from polarization data, according to one embodiment of the present disclosure. In the embodiment shown in FIG. 5, the inputs to method 500 include a polarization raw frame of the current object (or features extracted in the polarization representation space), an initial estimated pose of the object (e.g., previously calculated based on other techniques such as multi-view perspective n-point), and an existing 3D model of the object (e.g., a 3D model of the object stored by pose estimator 100, which identifies a 3D model corresponding to the object based on the class of the object identified through instance segmentation of the input image).
[0140] In operation 510, the pose estimator 100 calculates or measures the surface normal of the object based on the polarization raw frames captured from the object. More specifically, the pose estimator 100 may take as input a portion of the polarization raw frames (captured by one or more polarization cameras among the main camera 10 and the support camera 30) that corresponds to a region (detected by instance segmentation) corresponding to the current object. In some embodiments, only one camera among the main camera 10 and the support camera 30 is a polarization camera. In some embodiments, more than one camera among the main camera and the support camera 30 is (or includes) a polarization camera. The polarization raw frames are used to calculate properties in the polarization representation space (e.g., intensity I, DOLPρ, and AOLPφ) that correspond to the object, and these polarization properties are used to calculate the surface normal of the object (e.g., a direction normal to the surface of the object, etc.).
[0141] In operation 530, pose estimator 100 places the 3D model in a global coordinate system according to a previously estimated pose, such as a pose estimate calculated based on a multi-view perspective n-point, as discussed above.
[0142] The 3D model of the object includes a geometric definition of the location of the object's surfaces, and in some embodiments, pose estimator 100 calculates a surface normal of the posed 3D model. In operation 550, pose estimator 100 iteratively updates the pose of the 3D model to align the surface normal of the 3D model with the surface normal measured in operation 510, in a manner similar to an iterative closest point (ICP) algorithm. In some embodiments, pose estimator 100 calculates a cost function that compares (e.g., calculates the difference between) the surface normal of the 3D model with a measured surface normal based on polarized raw frames captured from each viewpoint with a polarized camera, and pose estimator 100 updates the pose of the 3D model of the object to minimize the difference between the surface normal of the posed 3D model and the measured surface normal (e.g., calculates the pose R that minimizes the cost calculated by the cost function). o , T o In some embodiments, pose estimator 100 determines a correspondence between portions of the surface appearing in the image of the object and a rendering of the 3D model posed according to the current pose, and calculates the difference between the measured surface normals and the surface normals of the corresponding portions of the posed 3D model.
[0143] In some embodiments of the present disclosure, refinement of the pose estimate is performed based on both detected edge-based alignment (such as by method 400 shown in FIG. 4) and surface normal-based alignment (such as by method 500 shown in FIG. 5). For example, the pose R of the object is o , T o Updating θ may simultaneously optimize a cost function that includes terms for both the error in matching the measured edges of the object with the edges of a 3D model posed according to the current estimated pose, and the error in matching the measured surface normals of the object with the surface normals of a 3D model posed according to the current estimated pose.
[0144] Training a deep neural network based on generated training data, including polarized training data In general, a computer vision system for computing a segmentation map that classifies objects depicted in a scene may include a trained convolutional neural network that takes as input two-dimensional images (e.g., captured by a color camera) and outputs a segmentation map based on those images. Such a convolutional neural network may be pre-trained on an existing dataset such as ImageNet (see, e.g., ImageNet: A Large-Scale Hierarchical Image Database, by J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, in IEEE Computer Vision and Pattern Recognition (CVPR), 2009). However, these existing datasets may contain images that are not representative of images expected to be encountered in a particular application of the computer vision system, which may cause these pre-trained models to perform poorly on the particular computer vision task that the computer vision system is intended to perform. For example, a computer vision system for a manufacturing environment is more likely to encounter images of tools, partially assembled products, manufacturing parts, etc. than images of people, animals, household items, and outdoor environments that might be found in more "general-purpose" datasets.
[0145] Thus, "retraining" refers to updating the parameters (e.g., connection weights) of a pre-trained model based on additional training data from a particular target domain associated with the task to be performed by the re-trained model. Continuing with the above example, labeled images of tools, partially assembled products, parts, etc. from a particular manufacturing environment can be used as training data to re-train a pre-trained model (e.g., a pre-trained convolutional neural network) to improve its performance in detecting and classifying objects encountered in that manufacturing environment. However, manually collecting various images of typical scenes in that manufacturing environment and labeling these images based on ground truth values (e.g., identifying pixels corresponding to different classes of objects) is generally a time-consuming and expensive task.
[0146] As mentioned above, three-dimensional (3D) rendering computer graphics software can be used to generate training data for training machine learning models to perform computer vision tasks. For example, existing 3D models of these tools, partially assembled products, and manufacturing parts can be arranged in a virtual scene according to various ways in which such objects are encountered in the real world (e.g., including lighting conditions in the environment and 3D models of support surfaces and equipment). For example, partially assembled products can be placed on a 3D model of a conveyor belt, parts can be located in a parts bin, tools can be placed on a tool bench and / or in the scene during the process of positioning parts in the partially assembled product. Thus, 3D computer graphics rendering systems are used to generate photorealistic images of a range of typical arrangements of objects in a particular environment. These generated images can also be automatically labeled. In particular, if the particular 3D models used to depict each of the different types of objects are already associated with class labels (e.g., screws of various sizes, pre-assembled parts, products in various stages of assembly, specific types of tools, etc.), segmentation maps can be automatically generated (e.g., by mapping the surfaces of the objects to specific class labels).
[0147] However, 3D rendering computer graphics software systems are generally tuned to generate images representing typical imaging modalities based on the intensity of visible light (e.g., red, green, and blue light intensities). 3D rendering software, such as Blender® by the Blender Foundation, typically does not consider the behavior of electromagnetic radiation that is invisible to the eye or may otherwise be negligible when rendering photorealistic scenes. Examples of these additional behaviors include polarization of light (e.g., polarized light interacting with transparent and reflective objects in the scene, as detected by a camera with a polarizing filter in the light path), thermal or infrared radiation (e.g., emitted by warm objects in the scene and detected by a camera system sensitive to detecting infrared light), ultraviolet radiation (e.g., detected by a camera system sensitive to ultraviolet light), combinations thereof (e.g., polarization by thermal radiation, polarization by visible light, polarization by ultraviolet light, etc.), etc.
[0148] Accordingly, aspects of embodiments of the present disclosure relate to systems and methods for modeling the behavior of various materials when imaged based on polarized light or other imaging modalities. Data (e.g., images) generated according to embodiments of the present disclosure can then be used as training data to train a deep learning model, such as a deep convolutional neural network, to calculate predictions based on imaging modalities other than standard imaging modalities (e.g., visible light or light intensity in the visible portion of the electromagnetic spectrum).
[0149] As a motivational example, embodiments of the present disclosure are described in the context of generating synthetic images of objects captured through a polarizing filter (referred to herein as "polarized light raw frames"), which are used to train a deep neural network, such as a convolutional neural network, to perform tasks based on the polarized light raw frames. However, embodiments of the present disclosure are not limited to generating synthetic polarized light raw frames for training a convolutional neural network that takes the polarized light raw frames (or features extracted therefrom) as input data.
[0150] FIG. 6 is a block diagram depicting a system for training a statistical model to perform a computer vision task based on images of various modalities, where the training is performed using data generated according to an embodiment of the present disclosure. As shown in FIG. 6, training data 605 is provided to a model training system 607, which takes a model 630 (e.g., a pre-trained model or a model architecture with initialized weights) and uses the training data 605 to generate a trained model (or retrained model) 632. The model 630 and the trained model 632 may be statistical models such as deep neural networks (deep neural networks include convolutional neural networks). A synthetic data generator 640 according to an embodiment of the present disclosure generates synthesized data 642 that may be included in the training data 605 to generate the trained model 632. The model training system 607 may apply an iterative process to update parameters of the model 630 to generate the trained model 632 according to the provided training data 605 (e.g., including the synthesized data 642). Updating the parameters of the model 630 may include, for example, applying gradient descent (and, in the case of neural networks, backpropagation) according to a loss function that measures the difference between the labels and the model's output as a function of the training data. The model training system 607 and the synthetic data generator 640 may be implemented using one or more electronic circuits.
[0151] According to various embodiments of the present disclosure, the model training system 607 and / or the synthetic data generator 640 are implemented using one or more electronic circuits configured to perform various operations described in more detail below. Types of electronic circuits may include a central processing unit (CPU), a graphics processing unit (GPU), an artificial intelligence (AI) accelerator (e.g., a vector processor that may include a vector arithmetic logic unit configured to efficiently perform operations common to neural networks, dot products, softmaxes, etc.), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a digital signal processor (DSP), etc. For example, in some circumstances, aspects of embodiments of the present disclosure are implemented by program instructions stored in a non-volatile computer readable memory that, when executed by an electronic circuit (e.g., a CPU, a GPU, an AI accelerator, or a combination thereof), perform the operations described herein to compute a segmentation map from input polarization raw frames or input polarization features (e.g., intensity, AOLP, and DOLP) in a polarization representation space. The operations performed by the model training system 607 and the synthetic data generator 640 may be performed by a single electronic circuit (e.g., a single CPU, a single GPU, etc.) or may be allocated among multiple electronic circuits (e.g., multiple GPUs, or CPUs in combination with GPUs). The multiple electronic circuits may be local to each other (e.g., located on the same die, located in the same package, or located in the same embedded device or computer system) and / or remote from each other (e.g., may communicate over a network such as a local personal area network such as Bluetooth, over a local area network such as a local wired and / or wireless network, and / or over a wide area network such as the Internet, such as when some operations are performed locally and other operations are performed on servers hosted by a cloud computing service).One or more electronic circuits that operate to implement the model training system 607 and the synthetic data generator 640 may be referred to herein as a computer or computer system, which may include memory that stores instructions that, when executed by the one or more electronic circuits, implement the systems and methods described herein.
[0152] 7A, 7B, 7C, and 7D provide background illustrating the segmentation maps computed by the comparative approach and semantic or instance segmentation according to an embodiment of the present disclosure. More specifically, FIG. 7A is an image or intensity image of a scene in which one real transparent ball is placed on top of a photographic printout depicting another scene containing two transparent balls ("spoofs") and some background clutter. FIG. 7B shows a segmentation mask computed by a comparative mask region-based convolutional neural network (Mask R-CNN) that uses a different pattern of lines to identify instances of the transparent ball superimposed on the intensity image of FIG. 7A, with the real transparent ball correctly identified as an instance and the two spoofs incorrectly identified as instances. In other words, the Mask R-CNN algorithm was tricked into labeling the two spoof transparent balls as instances of the real transparent ball in the scene.
[0153] Figure 7C is a linear polarization angle (AOLP) image calculated from polarized raw frames captured of a scene according to an embodiment of the present invention. As shown in Figure 7C, transparent objects have very unique textures in polarization space, such as the AOLP domain, where there are geometry-dependent signatures on edges and clear, or unique, or specific patterns that occur on the surface of transparent objects at linear polarization angles. In other words, the internal texture of a transparent object (as opposed to, for example, the external texture picked up from the background surface seen through the transparent object) is more visible in the polarization angle image of Figure 7C than in the intensity image of Figure 7A.
[0154] 7D depicts the intensity image of FIG. 7A overlaid with a segmentation mask computed using polarization data according to an embodiment of the present invention, where the real transparent ball is correctly identified as an instance using a superimposed pattern of lines and the two spoofs are correctly excluded as instances (e.g., in contrast to FIG. 7B, FIG. 7D does not include a superimposed pattern of lines on the two spoofs). Although FIGS. 7A, 7B, 7C, and 7D show examples relating to detecting real transparent objects in the presence of spoof transparent objects, embodiments of the present disclosure are not limited thereto and may also be applied to other optically challenging objects such as transparent, semi-transparent, and non-matte or non-Lambertian objects, as well as non-reflective (e.g., matte black objects) and multi-path guiding objects.
[0155] FIG. 8 is a graph of transmitted versus reflected light energy over a range of angles of incidence on a surface having a refractive index of about 1.5. As shown in FIG. 8, the slopes of the transmitted energy line (shown as a solid line in FIG. 8) and the reflected energy line (shown as a dotted line in FIG. 8) are relatively small for low angles of incidence (e.g., angles close to normal to the plane of the surface). Thus, for low angles of incidence (e.g., close to normal to the surface, in other words, close to the surface normal), it may be difficult to detect small differences in the angle of the surface in the polarization pattern (low contrast). Meanwhile, the slope of the reflected energy increases from flat as the angle of incidence increases, and the slope of the transmitted energy decreases from flat (becomes larger in absolute value) as the angle of incidence increases. In the example shown in FIG. 8, which has a refractive index of 1.5, the slopes of both lines begin to become substantially steeper at an angle of incidence of about 60°, and their slopes become very steep at an angle of incidence of about 80°. The particular shapes of the curves may vary for different materials according to the refractive index of the materials. Capturing an image of the surface under inspection at an angle of incidence that corresponds to a steeper portion of the curve (e.g., an angle closer to parallel to the surface, such as approximately 80° for a refractive index of 1.5 as shown in FIG. 8 ) can improve the contrast and detectability of changes in the surface topography in the polarization raw frame 18 and improve the detectability of such features in the tensors of the polarization representation space, because small changes in the angle of incidence (due to small changes in the surface normal) can cause large changes in the captured polarization raw frame.
[0156] 9 is a flow chart depicting a pipeline for generating a synthetic image according to one embodiment of the present disclosure. In some embodiments of the present disclosure, the operations of FIG. 9 are implemented by the synthetic data generator 40, e.g., in dedicated program instructions stored in a memory of the synthetic data generator 40, which when executed by a processor of the synthetic data generator 40, cause the synthetic data generator 40 to perform the dedicated operations described herein to generate a synthetic image based on a physical simulation of optical phenomena. For convenience, aspects of embodiments of the present disclosure are described in the context of applying polarized imaging in a manufacturing context to perform computer vision tasks on optically challenging manufacturing parts and tools, such as objects having transparent surfaces, shiny metallic surfaces, and / or dark matte surfaces.
[0157] In operation 910, the synthetic data generator 40 places a 3D model of the object into the virtual scene. In the context of generating a synthetic image of a scene in a manufacturing environment, the 3D model of the object may be readily available from computer-aided design (CAD) models of parts and partially or fully assembled products. These CAD models may have been created previously during the design phase of the product and may be obtained, for example, from a vendor of the part (e.g., a vendor who supplied the part to the manufacturer), publicly available information (e.g., data sheets), or an internal product designer employed by the manufacturer. In some circumstances, the CAD model may be manually generated based on the part specifications.
[0158] In some embodiments of the present disclosure, 3D models of objects are placed in a virtual scene in a manner that resembles the arrangement of those objects expected to be encountered in the particular computer vision task that the machine learning model is trained to perform.
[0159] In the above example of computer vision in the manufacturing context, one task is to perform instance segmentation on a box of parts, where the parts can be homogenous (e.g., all parts in the box are the same, such as a box of springs or a box of screws) or heterogeneous (e.g., a mix of different types of parts, such as different size screws, or screws mixed with matching nuts). Objects can be randomly placed in the box, where parts can be oriented in many different directions in the box, and in a box of heterogeneous parts, different types of parts are mixed together, as opposed to being separated into different parts of the box. A computer vision system can be trained to compute a segmentation map of the box and identify the location and orientation of individual parts in the box (and in the case of a box of heterogeneous parts, also the type of object). This segmentation map can then be used by an actuator system, such as a robotic arm, to pick parts from the box and add them to a partially assembled product.
[0160] Thus, in some embodiments of the present disclosure, the synthetic data generator 40 generates a scene of parts in a box by placing a 3D model of the virtual box in the scene and dropping the 3D model of the part into the virtual box as simulated using a physics simulation engine, such as a physics engine built into a 3D computer graphics rendering system. For example, Blender® 3D rendering software includes a physics system that simulates various physical real-world phenomena, such as the movement, collisions, and potential deformations of rigid bodies, cloth, soft bodies, fluids, etc., due to the influence of gravity or other forces. Thus, rigid body simulation can be used to simulate dropping rigid parts (e.g., screws, bolts, relatively stiff springs) into a rigid virtual box, and soft body simulation can be used to place elastic or deformable parts (e.g., strings, wires, plastic sheets, etc.) into a rigid virtual box.
[0161] More specifically, various differential scenes can be generated that represent different potential states of the box, such as by dropping various numbers of instances of the 3D model of the part into the virtual box. For example, if a typical box has a maximum capacity of 1,000 screws, various scenes can be generated by dropping 1,000 screws, 900 screws, 500 screws, 100 screws, and 10 screws into the virtual box to generate different scenes that represent different potential full states of the virtual box. In addition, multiple scenes can be generated for any given number of screws (or the number of screws can be randomized between the generation of different scenes), where the placement of the parts within the box is also randomized, such as by dropping the parts into the box one at a time from different random locations on the box.
[0162] Thus, in operation 910, synthetic data generator 40 generates a scene including an array of representative objects.
[0163] In operation 930, the synthetic data generator 40 adds lighting to the virtual scene generated in operation 910. Specifically, the synthetic data generator 40 adds one or more light sources to the virtual scene, where the light sources illuminate some or all of the surfaces of the objects in the box. In some embodiments, to improve the robustness of the training, the position of one or more light sources is randomized, and multiple scenes are generated with light sources at different locations (e.g., different angles and distances) relative to the part box. In some embodiments of the present disclosure, the virtual lighting includes virtual light sources that represent light sources that would be found in the environment in which the computer vision system is trained to operate. Examples of potential representative light sources include, for example, different color temperatures corresponding to incandescent, fluorescent, light-emitting diode (LED) bulbs, natural light from simulated windows in the environment, and other forms of lighting technology, and the shape of the virtual lighting (e.g., the direction of the light rays emitted by the lighting) can range from direct light to diffuse light. In some embodiments of the present disclosure, the light features (e.g., color temperature and shape) are also randomized to generate different scenes with different types of lighting.
[0164] At operation 950, the synthetic data generator 40 applies modality-specific materials to objects in the 3D virtual scene. For example, when generating synthetic polarization imaging data, polarization-specific materials are applied to objects in the virtual scene, whereas when generating synthetic thermal imaging data, thermal imaging-specific materials are applied to objects in the virtual scene. For illustrative purposes, polarization-specific materials are described in detail herein, however, embodiments of the present disclosure are not limited thereto and may also be applied to the generation and application of materials specific to multi-modal and / or plenoptic imaging modalities.
[0165] Some aspects of the embodiments of the present disclosure relate to domain randomization, where the material appearance of objects in a scene is randomized beyond the typical appearance of the object. For example, in some embodiments, a large number of materials with random colors (e.g., thousands of different materials with different colors selected randomly) are applied to different objects in a virtual scene. In a real-world environment, objects in a scene generally have distinct colors (e.g., rubber washers generally all appear matte black, while screws may be a particular shade of shiny black, matte black, gold, or shiny metal). However, real-world objects may have different appearances due to changes in lighting conditions such as light color temperature, reflections, specular highlights, etc. Thus, applying randomization to the colors of materials applied to objects when generating training data expands the domain of the training data to encompass unrealistic colors, which increases the diversity of the training data and allows training of more robust machine learning models that can make accurate predictions (e.g., more accurate instance segmentation maps) in a greater variety of real-world conditions.
[0166] Some aspects of the embodiments of the present disclosure relate to performing texture mapping to generate a model of a material that depends on one or more parameters (parameterized material) according to an imaging modality. For example, as discussed above, the appearance of a given surface in a scene imaged by a polarization camera system may vary based on the material properties of the surface, the spectral profile and polarization parameters of the illumination source or sources (light sources) in the scene, the angle of incidence of the light on the surface, and the viewpoint angle of the observer (e.g., the polarization camera system). Therefore, simulating the physical properties of the polarization of different materials is a complex and computationally intensive task.
[0167] Thus, some aspects of embodiments of the present disclosure relate to emulating the physical properties of various imaging modalities based on empirical data, such as real-world images captured of real-world materials. More specifically, an imaging system (e.g., a polarized camera system) implementing a particular imaging modality of interest is used to collect sample images from an object made of a particular material of interest. In some embodiments, the collected sample images are used to calculate an empirical model of the material, such as a light-field function (e.g., a bidirectional reflectance density function or BRDF) of the material's surface.
[0168] In general, some techniques for generating an empirical model of a material involve capturing a sample of images of a representative material from multiple viewing angles and at multiple different illumination angles.
[0169] FIG. 10 is a schematic diagram of sampling a real material from multiple angles using a polarization camera system according to an embodiment of the present disclosure. As shown in FIG. 10, a surface 1002 of a physical object (e.g., a washer, a screw, etc.) is made of a material of interest (e.g., black rubber, chrome-plated stainless steel, etc., respectively). In operation 810, the material is placed in a physical scene (e.g., a laboratory benchtop). A physical illumination source 1004, such as an LED lamp or a fluorescent lamp, is placed in the scene and positioned to illuminate at least a portion of the surface 1002. For example, as shown in FIG. 10, a light ray 1006 emitted from the physical illumination source 1004 is incident on a particular point 1008 of the surface 1002 at an incidence angle α at the particular point 1008 on the surface 1002 with respect to a normal direction 1014 of the surface 1002 at the particular point 1008 of the surface 1002.
[0170] An imaging system is used to obtain images of an object's surface 1002 from multiple poses relative to the surface's normal direction. In the embodiment shown in Figure 10, a polarized camera system 1010 is used as the imaging system to capture images of the surface 1002 including a portion (e.g., including a particular point 1008) that is illuminated by a physical illumination source 1004. The polarized camera system 1010 captures images of the surface 1002 from different poses 1012, such as by moving the polarized camera system 1010 from one pose to the next and capturing polarized raw frames from each pose. In the embodiment shown in FIG. 10, the polarized camera system 1010 images the surface 1002 at a front-on parallel observer angle β of 0° (e.g., a front-on parallel view from directly above or aligned with the surface normal 1014 at point 1008) in a first pose 1012A, at an intermediate observer angle β such as an angle of 45° with respect to the surface normal 1014 in a second pose 1012B, and at a shallow observer angle β with respect to the surface normal 1014 (e.g., slightly less than 90°, such as 89°) in a third pose 1012C.
[0171] As discussed above, the polarized camera system 1010 is generally configured to capture polarized raw frames using polarized filters at different angles (e.g., using a polarized mosaic with four different polarization angles in the optical path of a single lens and sensor system, using an array of four cameras where each of the cameras has a linear polarizing filter at a different angle, using polarized filters set at different angles for different frames captured from the same pose at different times, etc.).
[0172] Each of these viewpoints or poses 1012 gives a different polarization signal due to the nature of the physics of polarized light. Thus, by capturing images of the surface 1002 from different observer angles, a model of the material's BRDF can be estimated based on interpolation between images captured by the camera system at one or more poses 1012 with the closest corresponding observer angle β, and using one or more physical illumination sources 1004 with the closest corresponding angles of incidence α.
[0173] Methods for generating empirical plenoptic models (e.g., polarization models) of materials based on sampling are described in more detail in PCT U.S. Patent Application No. 21 / 12073, filed with the United States Patent and Trademark Office on January 4, 2021, the entire disclosure of which is incorporated herein by reference.
[0174] Referring again to FIG. 9, in some embodiments, at operation 970, the synthetic data generator 40 sets a virtual background for the scene. In some embodiments, the virtual background is an image captured using the same imaging modality as the modality being simulated by the synthetic data generator 40. For example, in some embodiments, when generating a synthetic polarized image, the virtual background is a real image captured using a polarized camera, and when generating a synthetic thermal image, the virtual background is a real image captured using a thermal camera. In some embodiments, the virtual background is an image of an environment similar to the environment in which the trained machine learning model is intended to operate (e.g., a manufacturing facility or factory in the case of a computer vision system for manufacturing robots). In some embodiments, the virtual background is randomized, thereby increasing the diversity of the synthetic training dataset.
[0175] In operation 990, the synthetic data generator 40 renders the 3D scene based on the specified imaging modality (e.g., polarized light, thermal, etc.) using one or more of the empirically derived modality-specific models of the material. Some aspects of an embodiment of the present disclosure relate to rendering an image based on an empirical model of the material according to an embodiment of the present disclosure. The empirical model of the material can be developed as discussed above based on samples collected from images captured of a real-world object made of the material of interest.
[0176] Generally, a 3D computer graphics rendering engine generates a 2D rendering of a virtual scene by calculating the color of each pixel of an output image according to the color of the surface of the virtual scene that the pixel depicts. For example, in a ray-tracing rendering engine, virtual rays are emitted from a virtual camera into the virtual scene (opposite the typical path of light in the real world), where they interact with the surfaces of 3D models of objects in the virtual scene. These 3D models are typically represented using geometric shapes, such as meshes of points that define flat surfaces (e.g., triangles), which may have materials that describe how the virtual rays interact with the surfaces, such as reflection, refraction, scattering, dispersion, and other optical effects, as well as textures that represent the color of the surfaces (e.g., the textures may be solid colors or, for example, bitmap images applied to the surfaces). The path of each virtual ray is followed (or "tracked") through the virtual scene until it reaches a light source (e.g., a virtual lighting fixture) in the virtual scene, and the accumulated changes in the textures encountered along the path from the camera to the light source are combined with the properties of the light source (e.g., the color temperature of the light source) to calculate the color of the pixel. This general process can be modified, as will be appreciated by those skilled in the art, such as by performing anti-aliasing (or smoothing) by tracing multiple rays through different parts of each pixel and calculating the color of the pixel based on a combination (e.g., an average) of different colors calculated by tracing the different rays that interact with the scene.
[0177] Systems and methods for rendering plenoptic images (e.g., polarization images or images in polarization representation space) based on these empirical plenoptic models are described in more detail in PCT U.S. Patent Application No. 21 / 12073, filed with the United States Patent and Trademark Office on January 4, 2021, the entire disclosure of which is incorporated herein by reference.
[0178] Accordingly, aspects of an embodiment of the present disclosure relate to a simulation environment configured to simulate the 3D geometry, lighting conditions, and material properties of an object of interest (e.g., a type of object whose pose is to be estimated by pose estimator 100). The simulation of material properties includes the ability to simulate various physical phenomena, including specular reflection, material color, roughness, and polarization of light and its reflection from different materials, where at least some plenoptic material properties, such as polarization of light, are simulated based on an empirical model of the material.
[0179] Data Generation Using Simulation Some aspects of the embodiments of the present disclosure relate to a combination of the above techniques to generate synthetic images for training machine learning models. FIG. 11 is a flowchart depicting a method for generating a training dataset according to one embodiment of the present disclosure. One or more virtual scenes representing a target domain may be generated as discussed above (e.g., by selecting one or more 3D models of the parts and dropping an instance of the 3D model into a container to generate an image of a box of the parts). Some aspects of the embodiments of the present disclosure relate to forming a training dataset based on (1) images generated purely by domain randomization in operation 1110, (2) images generated purely through texture mapping (e.g., applying a plenoptic material to the surface of an object) in operation 1112, and (3) images generated purely through normal mapping (e.g., polarization spatial feature maps such as DOLP ρ images and AOL P φ images generated based on the surface normals of an object) in operation 1114.
[0180] Additionally, the training dataset may include images generated using models of materials generated by interpolating between different empirically generated models as parameterized in the embedding space, as discussed above.
[0181] In some embodiments of the present disclosure, the images generated according to (1) domain randomization, (2) texture mapping, and (3) normal mapping are further processed by applying style transfer or other filters to the generated images, respectively, in operation 1120, before adding the images to the training dataset. Applying style transfer makes images that appear somewhat different, such as those generated using the three techniques described above, appear more consistent in appearance. In some embodiments, the style transfer process transforms the synthesized input images to look more similar to the captured images based on the target imaging modality (e.g., making the images generated using (1) domain randomization and (3) feature maps generated using normal mapping look like polarized raw frames) or makes the synthesized input images look more artificial, such as by applying an unrealistic painting style to the input images (e.g., making the images generated using (1) domain randomization and (2) rendering with texture mapping and (3) feature maps generated using normal mapping look like paintbrushes on a canvas).
[0182] In some embodiments, neural style transfer networks such as SytleGAN for complex global style transfer (see, e.g., Karras, Tero, et al., “Analyzing and improving the image quality of stylegan,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition 2020), patch-based networks for local style transfer (see, e.g., Chen, Tian Qi, and Mark Schmidt, “Fast patch-based style transfer of arbitrary style,” arXiv preprint arXiv:1612.04337 (2016)), and networks using domain adaptation (see, e.g., Dundar, Aysegul, et al., “Domain stylization: A strong, simple baseline for synthetic to real image domain adaptation,” arXiv preprint arXiv:1807.09384 (2018)) are trained and used to perform style transfer in operation 1120 on the images selected for the training dataset. As a result, all images in the training dataset may have a similar style or appearance, regardless of how the images were acquired (e.g., whether (1) domain randomization, (2) texture mapping, (3) normal mapping, or other sources such as real images of objects captured using an imaging system that implements the modality of interest such as polarized or thermal imaging).
[0183] Additionally, some aspects of embodiments of the present disclosure relate to performing additional or other data augmentations to the generated images, including, but not limited to, style randomization, Gaussian noise, affine transformations, edge blurring, and distortion.
[0184] When generating training data for supervised learning, the synthetic data generator 640 also automatically generates labels (e.g., desired outputs) for the synthesized images in operation 1124. For example, when generating training data for training a machine learning model to perform an image classification task, the generated labels for a given image may include a class of object depicted in the image. These classification labels may be generated by identifying the unique type of each of the objects visible in the virtual scene. As another example, when generating training data for training a machine learning model to perform an instance segmentation task, the generated labels may include a segmentation map in which each instance of each object is uniquely identified (e.g., using different instance identifiers), along with a classification (e.g., objects of the same type have the same classification identifier). For example, the segmentation map may be generated by tracing rays entering the virtual scene from the camera, where each ray may intersect with some first surface of the virtual scene. Each pixel of the segmentation map is labeled accordingly based on the instance identifier and classification identifier of the object, including the surface hit by the ray emitted from the camera through the pixel. As yet another example, when generating training data for training a keypoint detector, keypoints are selected using standard 3D keypoint selection methods. In some embodiments, a farthest point algorithm is used to select keypoints on the 3D model that are as far away from each other as possible, forcing them to be in different parts of the object.
[0185] In some embodiments of the present disclosure, images of the training dataset are sampled from the synthesized datasets (1), (2), and (3) based on hard example mining (see, e.g., Smirnov, Evgeny, et al., “Hard example mining with auxiliary embeddings,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops 2018) in operation 1124. Sampling the synthesized dataset using hard example mining can reduce the size of the training set and improve the efficiency of the training process by removing substantially redundant images that would not have much impact on the training process, while retaining “hard samples” that have a greater impact on the resulting trained model.
[0186] 1, the resulting training dataset of synthesized data 642 generated by synthetic data generator 640 is then used as training data 605 by model training system 607 to train model 630, such as a pre-trained model or a model initialized with random parameters, to generate trained model 632. Continuing with the example presented above of generating training data according to a polarization imaging modality, training dataset 605 can be used to train model 630 to operate on modified input features such as polarization raw frames (e.g., images generated by texture mapping) and tensors in polarization representation space (e.g., images generated by normal mapping).
[0187] Thus, training data 605 including synthesized data 642 is used to train or retrain a machine learning model 630 to perform a computer vision task based on a particular imaging modality. For example, synthetic data from a polarization imaging modality can be used to retrain a convolutional neural network that has been pre-trained to perform instance segmentation based on standard color images to perform instance segmentation based on polarization input features.
[0188] In deployment, a trained model 632 trained based on training data generated according to embodiments of the present disclosure is then configured to take inputs similar to the training data, such as polarization raw frames and / or tensors in polarization representation space (which input images are further modified by the same style transfer, if style transfer was applied when generating the training data), and generate predicted outputs, such as segmentation maps. For example, in some embodiments, as shown in FIG. 2A, models trained to perform tasks such as instance segmentation and keypoint detection are used by pose estimator 100 to compute segmentation maps and detect keypoints, respectively, in operations 230 and 270. Examples of architectures suitable for these computer vision tasks include, but are not limited to, Mask R-CNN for segmentation and keypoints (see, e.g., He, Kaiming, et al., “Mask R-CNN,” in Proceedings of the IEEE International Conference on Computer Vision. 2017), U-Net (see, e.g., Ronneberger, Olaf, Philipp Fischer, and Thomas Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, 2015), and AdaptIS (see, e.g., Sofiiuk, Konstantin, Olga Barinova, and Anton Konushin, “Adaptis: Adaptive Instance Selection Network,” in Proceedings of the IEEE / CVF International Conference on Computer Vision 2019).
[0189] Thus, while the comparison system for computing pose estimation relies on either real-world training data or a specific depth sensor to achieve generalized 6DoF performance, the embodiments of the present disclosure can be applied to any object type for which an accurate 3D model (e.g., a CAD model) is available. The present system and method can also be applied to any imaging technology as long as the output resembles what is achievable with the generation of simulated data. Furthermore, the embodiments of the present disclosure can also be used to train neural networks to perform computer vision tasks based on high-dimensional optical data, such as information from polarization cues. Thus, some embodiments of the present disclosure relate to the combination of deep learning models (e.g., convolutional neural networks) trained on training data generated by a rendering engine according to embodiments of the present disclosure, including training based on polarization cues to perform 6 degrees of freedom (6DoF) pose estimation as described above.
[0190] Object-level support using segmentation When spaced apart cameras are used for image capture, each of the multiple cameras (e.g., the main camera 10 and the support camera 30) can capture images of one or more objects from different viewpoints. An instance segmentation map (also called an instance segmentation mask) can be generated to identify and depict objects in the captured images. Although the same object is depicted in various images, the instance segmentation maps generated based on the various images may be different if the images depict the object from different viewpoints. Current technology mechanisms can perform global pixel-level correspondence to identify blocks of pixels in a first image that correspond to groups of pixels in a second image. Such correspondence may be desirable to perform depth estimation for each object instance. However, computing pixel-level correspondence at a global level (e.g., for the entire image) can be computationally expensive. It is therefore desirable to have a more efficient way than finding correspondences for portions of images depicting different viewpoints of the same object.
[0191] Broadly speaking, embodiments of the present disclosure are directed to reducing a search space for performing image processing tasks such as pixel-level correspondence. In one embodiment, instance segmentation is performed to identify different instances of objects in images depicting a scene viewed from different viewpoints, and an instance segmentation map / mask may be generated in response to the instance segmentation operation. The instance segmentation mask may then be employed to compute object-level correspondence.
[0192] In one embodiment, object-level correspondence allows a first instance of an object appearing in a first image depicting a first viewpoint to be matched to a second instance of the same object appearing in a second image depicting a second viewpoint. Once object-level correspondence is performed, the search space for performing pixel-level correspondence, for example, may be limited to regions of the image that correspond to the same object. Reducing the search space in this manner may speed up the processing of pixel-level correspondence and other similar tasks.
[0193] 12 is a flow diagram of a process for object-level correspondence according to one embodiment. The process may be implemented by one or more processing or electronic circuits that are components of pose estimator 100. As will be appreciated by those skilled in the art, it will be understood that the sequence of steps in the process is not fixed and can be modified, reordered, performed differently, sequentially, in parallel or simultaneously, or in any desired order.
[0194] Processing begins at block 1200, where pose estimator 100 receives multi-view images from main camera 10 and support camera 30. A first image captured by one of the cameras may depict one or more objects in the scene from a first viewpoint, and a second image captured by the second camera may depict one or more objects in the scene from a second viewpoint different from the first viewpoint. The images captured by the cameras may be, for example, polarized images and / or images that are not subject to any polarized light.
[0195] At block 1202, the pose estimator 100 performs instance segmentation and mask generation based on the captured images. In this regard, the pose estimator 100 classifies various regions (e.g., pixels) of an image captured by a particular camera 10, 30 as belonging to a particular classification of objects. Each of the different instances of the object in the image is also identified, and a unique label is applied to each of the different instances of the object, such as by individually labeling each object in the image with a different identifier.
[0196] In one embodiment, segmentation masks are also generated to depict various object instances. Each segmentation mask may be a 2D image having the same dimensions as the input image, and the value of each pixel may correspond to a label (e.g., a particular instance of the object depicted by the pixel). Different segmentation masks may be generated for different images depicting different viewpoints of the object of interest. For example, a first segmentation mask may be generated to depict the object instances in a first image captured by a first camera, and a second segmentation mask may be generated to depict the object instances in a second image captured by a second camera. A convolutional neural network, such as Mask R-CNN, may be employed to generate the segmentation masks.
[0197] At block 1204, the pose estimator 100 engages in object-level correspondence of objects identified in the segmentation mask. In this regard, the pose estimator may invoke a matching algorithm to identify a segmented instance of a particular object in one image as corresponding (or matching) to a segmented instance of the same object in another image. The matching algorithm may be constrained to search for matching object instances along an epipolar line passing through the object instances in one image to find corresponding object instances in a different image. In one embodiment, the matching algorithm compares different features of regions corresponding to the segmented object instances to estimate object correspondence. Matching object instances from one image to another may narrow the search space of other image processing tasks, such as, for example, performing pixel-level or keypoint correspondence. The search space may be narrowed to identified regions of the images identified as corresponding to the same object.
[0198] At block 1206, the pose estimator 100 generates an output based on the object-level correspondence. The output can be, for example, a measure of disparity or an estimated depth (e.g., distance from the cameras 10, 30) of the object based on the disparity between corresponding instances depicted in various images. In one embodiment, the output is a three-dimensional relative pose between the first instance of the object and the second instance of the object, calculated, for example, based on the perspective n-points of multi-view keypoints and / or subsequent pose refinement as discussed above with respect to FIGS. 2A, 3, 4, and 5.
[0199] 13 is a block diagram of an architecture for instance segmentation and mask generation of step 1002, according to one embodiment. Input images 1300 captured by the various cameras 10, 30 are provided to a deep learning network 1302, such as a CNN backbone. In embodiments where the images include polarized images, the deep learning network may be implemented as a polarized CNN backbone, as described in PCT US Patent Application No. 2020 / 048604, the contents of which are incorporated herein by reference.
[0200] In one embodiment, the deep learning network 1302 is configured to generate a feature map based on the input image 1300 and employ a region proposal network (RPN) to propose regions of interest from the generated feature map. The CNN backbone proposals may be provided to a box head 1304 to perform classification and bounding box regression. In one embodiment, the classification outputs a classification label 1306 for each of the object instances in the input image 1300, and the bounding box regression predicts a bounding box 1308 for the classified object. In one embodiment, a different classification label 1306 is provided for each instance of the object.
[0201] The suggestions from the CNN backbone may also be provided to a mask head 1310 to generate an instance segmentation mask. The mask head 1306 may be implemented as a fully convolutional network (FCN). In one embodiment, the mask head 1310 is configured to encode a binary mask for each of the object instances in the input image 1300. Figure 15 depicts an example of an instance segmentation mask.
[0202] Figure 14 is a more detailed flow diagram of the matching algorithm employed in step 1204 (Figure 12) to identify object level correspondences for particular object instances in the first segmentation mask, according to one embodiment. The process can be repeated for all object instances identified in the first segmentation mask. As will be appreciated by those skilled in the art, the sequence of steps in the process of Figure 14 is not fixed and can be modified, reordered, performed differently, sequentially, in parallel or simultaneously, or in any desired order.
[0203] At block 1420, the matching algorithm identifies features of the first object instance in the first segmentation mask. The features identified for the first object instance may include a shape of the object instance's region, a feature vector within the region, and / or keypoint predictions within the region. The shape of the first object instance's region may be represented via a set of points sampled along the contour of the region. If a feature vector within the region is used as a feature descriptor, the feature vector may be an average deep learning feature vector extracted via a convolutional neural network.
[0204] At block 1422, the matching algorithm identifies an epipolar line that passes through the first object instance in the first segmentation mask.
[0205] At block 1424, the matching algorithm identifies one or more second object instances in the second segmentation mask that may correspond to the first object instance. The search for the second object instance may be limited to an epipolar line between the first and second segmentation maps that passes through the first object instance. In one embodiment, the matching algorithm searches approximately along the identified epipolar line to identify object instances in the second segmentation mask that have the same classification identifier as the first object instance. For example, if the first object instance belongs to a "dog" classification, the matching algorithm evaluates object instances in the second segmentation mask that also belong to the "dog" classification and ignores objects that belong to a different classification (e.g., a "cat" classification).
[0206] At block 1426, the matching algorithm identifies features of the second object instance that belong to the same classification. As with the first object instance, the features of a particular second object instance may include the shape of the region of the second object instance, a feature vector representing the region, and / or keypoint predictions within the region.
[0207] At block 1428, a matching algorithm compares the features of the first object instance to the features of the second object instance to determine a match. In one embodiment, the matching algorithm identifies matches between the features of the first object instance and the features of the second object instance to select the best match. In one embodiment, the best match may be identified via a matching function such as the Hungarian matching function. In one embodiment, the features of the object instances are represented as probability distributions, and the matching function attempts to find a match of the probability distributions that minimizes the Kullback-Leibler (KL) divergence.
[0208] At block 1430, a determination is made as to whether a match was found. If the answer is yes, then output is generated at block 1432. The output may include, for example, information (e.g., object IDs) of the second object instance that matched with the first object instance.
[0209] If the answer is no, then at block 1434 an output may be generated indicating a failed match.
[0210] Figure 15 is an instance segmentation mask generated for image 211-L of Figure 2B according to one embodiment. The segmentation mask may identify regions 1600, 1602 of the image that contain instances of identified objects, along with their unique boundaries. Each identified mask region may take the shape of the identified object instance. Pixels within the identified mask region may be associated with one or more values that identify a particular object instance and object type.
[0211] Figures 16A and 16B are images depicting the results of a process for determining object-level correspondence described with respect to Figure 12, according to one embodiment. In the example of Figures 16A and B, the match is for objects depicted in images 1500a, 1500b. For example, as a result of performing the process of Figure 12, object instance 1600 in Figure 16A can be identified as a corresponding / matching object instance 1602 in Figure 16B. On the other hand, certain object instances may end up not matching. For example, no match is found for object instance 1604 in Figure 16A.
[0212] Automatic calibration of hardware systems Some aspects of embodiments of the present disclosure relate to systems and methods for automatic calibration of a camera system including a main camera 10 and a support camera 30. Some aspects of embodiments of the present disclosure relate to optimizing cost functions (e.g., as shown in Equations 7 and 8 above) when estimating the pose of an object in a scene imaged by the main camera 10 and the support camera 30. These cost functions assume that the relative poses associated with each of the viewpoints are known, i.e., the calibration parameters of each of the cameras are known.
[0213] Camera calibration generally falls into three categories: camera-internal, stereo-internal, and device or system-internal. Initial estimates for all three can be calculated using standard camera calibration, but these initial estimates may be inaccurate. Thus, some embodiments of the present disclosure relate to active calibration from the part itself. Specifically, some aspects of the embodiments of the present disclosure involve the determination of the pose of objects in the scene and the calibration of the cameras (extrinsic parameters R for the jth camera). j , T j and the internal parameter K j ) simultaneously by simultaneously optimizing the cost function of Equation 14 below:
number
[0214] Equation 14 expresses the method to find the keypoints [x i ,y i ,z i ] (indexed by i) ij ,v ij] (indexed by the current keypoint i seen by the camera at the jth viewpoint). This active calibration using deep learning-based keypoints with arbitrary objects (as opposed to specialized calibration targets, for example) enables calibration of M cameras using a cluttered part 22 box, even with a passive camera system (e.g., without the use of an active lighting projector).
[0215] FIG. 17 is a schematic depiction of camera auto-calibration according to one embodiment of the present disclosure. As shown in FIG. 17, an object 1210 may have a pose in a scene and may have corresponding 3D keypoints [x1,y1,z1], [x2,y2,z2], [x3,y3,z3], and [x4,y4,z4]. These keypoints are projected into 2D image space corresponding to the viewpoints (or poses) of Cam1 and Cam2, and these projected points are shown to be near the corresponding deep learning estimated points. These deep learning estimated points correspond to the 2D keypoints detected in the 2D images captured by Cam1 and Cam2, and are shown in FIG. 17 as [u 11 ,v 11 ], [u 21 ,v 21 ], [u 31 ,v 31 ] and [u 41 ,v 41 ], and [u 12 ,v 12 ], [u 22 ,v 22 ], [u 32 ,v 32 ] and [u 42 ,v 42 The pose estimator 100 estimates the object pose R o , T o ,Automatically calibrate the camera system (e.g., determine the extrinsic and intrinsic parameters of the cameras) by simultaneously updating R1, T1, K1 of estimated camera pose 1 and R2, T2, K2 of estimated camera pose 2.
[0216] Correlating camera space to robot space As mentioned above, some aspects of embodiments of the present disclosure relate to pose estimation in the context of robotics, such as providing pose estimates to a robotics system to detect the pose of an end effector (and other parts) of a robot arm and objects within the robot's vicinity. These pose estimates can then be used by a robot controller to control the movement of the robotic system and to manipulate and / or navigate detected objects within a workspace or environment.
[0217] For example, in some embodiments, once the six degrees of freedom (6DoF) pose of the object has been calculated (e.g., according to FIG. 2A above), the object's pose is correlated in some manner to the robot's workspace, allowing the end effector to be efficiently transitioned to the position required to grasp the object, such as by transforming the pose into a coordinate system used by the robot controller.
[0218] In some embodiments, fiducials (e.g., visually detectable markers such as ArUco markers) are incorporated into the robot end effector to facilitate tracking the pose of the end effector. Tracking these fiducials using the multi-view vision system 1 allows for easy connection of the vision system 1 to the robot workspace and mitigates positional errors associated with moving the end effector of the robot arm to the location of an object to be grasped by the end effector based on the object's position calculated by the pose estimator 100.
[0219] FIG. 18 is a schematic diagram depicting that the precision and repeatability of a robot arm is a function of both the robot's pose and position. As shown in FIG. 18, a robot arm attempting to move an end effector to a desired position in several different attempts may actually move the end effector to a cluster of different acquired positions. The difference between the centroid of the acquired positions and the desired position corresponds to the precision (or error) of the robot system, the smaller the difference, the higher the precision. The radius or size of a circle (or sphere or ball) that contains all of the acquired positions corresponds to the repeatability of the movement (e.g., the smaller the radius, the higher the repeatability).
[0220] Thus, aspects of an embodiment of the present disclosure include using a multi-view vision system 1 according to an embodiment of the present disclosure to track the end effector and / or other joints of the robot arm (e.g., by pose estimation based on a fiducial and / or a 3D model of the end effector and / or other joints of the robot arm) and predict position errors associated with moving the end effector of the robot arm to a desired position based on both the initial pose and position of the end effector. In particular, in some embodiments, a multi-view vision system 1 according to an embodiment of the present disclosure is combined with a deep learning backend to predict errors associated with every movement of the end effector based on the detected pose of the end effector and the joints of the robot arm. These predictions are provided as real-time feedback to the robot controller to improve the accuracy and repeatability of moving the end effector of the robot arm to a desired position.
[0221] Thus, aspects of embodiments of the present disclosure relate to a fully passive multi-view vision system (e.g., no additional light projectors) that applies deep learning-based keypoint detectors to perform simultaneous optimization across multiple views to achieve high accuracy six degree of freedom (6DoF) pose estimation that would not be possible with passive vision systems. Passive vision systems can be very economical because they do not require additional light projectors and their associated calibration, and instead can achieve high performance depth accuracy using deep learning and simultaneous optimization while calibrating using fiducials and / or using automatic calibration using objects detected in the scene.
[0222] In some embodiments where the main camera 10 and one support camera are spaced 30 cm apart from each other and placed about 1 meter above the object 22, and both the main camera 10 and the support camera 30 have two camera modules with 30 degree FoV lenses (e.g., a set of four lenses in the arrangement of FIG. 1E) and 5 megapixel image sensors, the pose estimation system can achieve a pose detection accuracy of about 200 μm in the depth direction (e.g., approximately parallel to the optical axis of the main camera 10 and the support camera), and a pose detection translation accuracy (e.g., perpendicular to the optical axis) of less than 100 μm, and a rotational accuracy of 1 degree, along with a point cloud accuracy of 500 μm for a variety of highly cluttered objects and a variety of lighting conditions. These exemplary embodiments of the present disclosure have an experimentally measured cycle time of less than 500 milliseconds (ms), measured from the time of image capture by the main camera 10 and the support camera 30 to the time of output of the 6DoF pose of the object and point cloud by the pose estimator 100.
[0223] Although the invention has been described in connection with specific exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims and equivalents thereof.
Claims
1. 1. A method for estimating a pose of an object, comprising: receiving a plurality of images of the object captured from a plurality of viewpoints relative to the object, the plurality of images including polarization information; initializing a current pose of the object based on calculating an initial estimated pose of the object from at least one of the plurality of images; performing an instance segmentation process on the plurality of images captured from a plurality of viewpoints; extracting image patches of the object from the instances of the object detected by the instance segmentation process; predicting a plurality of 2D keypoints associated with the object from each of the plurality of images, the predicting comprising providing the extracted image patches to a deep learning keypoint detector, the deep learning keypoint detector configured to compute 2D keypoints based on the polarization information of the images; and calculating an updated pose that minimizes a cost function based on a plurality of differences between the 2D keypoints and a plurality of 3D keypoints associated with a 3D model of the object positioned according to the current pose and projected onto each of the viewpoints.
2. each of the plurality of differences corresponds to a different one of the viewpoints; Each of the differences, with respect to a viewpoint of the viewpoints, the plurality of 2D keypoints associated with an image of the plurality of images corresponding to the viewpoint; The method of claim 1 , wherein the 3D key points of the 3D model positioned in the current pose are calculated based on a difference between projecting the 3D key points onto the viewpoint.
3. The cost function is: [0010] Wherein R o , T o is the pose of the object, i is an index for iterating through N 3D keypoints, j is an index for iterating through M viewpoints, and c ij is the corresponding keypoint prediction [u ij , v ij ] is the reliability of j , T j , K j are the extrinsic and intrinsic parameters of the jth camera used to capture the image from the jth viewpoint, and [x i , y i , z i 3. The method of claim 2, wherein i,j,i ...
4. The cost function is: [0025] Further describing the symmetry in the object according to o , T o is the pose of the object, i is an index for iterating through N 3D keypoints, j is an index for iterating through M viewpoints, and c ij is the corresponding keypoint prediction [u ij , v ij ] is the reliability of j , T j , K j are the extrinsic and intrinsic parameters of the jth camera used to capture the image from the jth viewpoint, and [x i , y i , z i ] is the i-th 3D keypoint, S is the transformation between different symmetries of the object, and v j The method of claim 2 , wherein j is the jth view.
5. 10. The method of claim 1, wherein the deep learning keypoint detector is trained based on synthesized training data, the training data including synthesized polarization information and synthesized keypoint labels.
6. A method for estimating a pose of an object, comprising: receiving a plurality of images of the object captured from a plurality of viewpoints relative to the object, the plurality of images including polarization information; initializing a current pose of the object based on calculating an initial estimated pose of the object from at least one of the plurality of images; performing an instance segmentation process on the plurality of images captured from a plurality of viewpoints; extracting image patches of the object from the instances of the object detected by the instance segmentation process; predicting a plurality of 2D keypoints associated with the object from each of the plurality of images, the predicting comprising providing the extracted image patches to a keypoint detector; calculating an updated pose that minimizes a cost function based on a plurality of differences between the 2D keypoints and a plurality of 3D keypoints associated with a 3D model of the object positioned according to the current pose and projected onto each of the viewpoints; The method further includes refining the updated pose by aligning the 3D model with the polarization information.
7. The refining of the updated pose comprises: Detecting one or more edges of the object in the image to generate one or more measured edge images; Finding a correspondence between rendered edge images of the 3D model positioned at the current pose of the one or more measured edges; and iteratively refining the updated pose using multi-view perspective n-points based on the detected correspondences.
8. The edge is calculating a surface normal based on the polarization information; and detecting edges of the object at points of discontinuity in the surface normal.
9. The edge is Computing a plurality of instance segmentation maps from the images; and detecting edges of the object at boundaries of instances of the object in the instance segmentation map.
10. The refining of the updated pose comprises: calculating a measured surface normal of the object based on the polarization information; positioning the 3D model of the object based on the updated pose; and iteratively updating the pose of the 3D model to align the surface normal of the 3D model with the measured surface normal.
11. 1. A system comprising: The main camera and one or more support cameras spaced apart from the main camera, the main camera and the one or more support cameras having overlapping fields of view; and a pose estimator including a processor and a memory, the memory storing instructions that, when executed by the processor, cause the processor to control the main camera, the one or more support cameras, and to perform the method of any one of claims 1 to 10.
12. The system of claim 11 , wherein the main camera and the one or more support cameras form a passive camera system.
13. The system of claim 11 , further comprising an active projector configured to project structured light onto an area within the overlapping fields of view of the main camera and the support camera.
14. 12. A method for automatically calibrating the system of claim 11, comprising: receiving a plurality of images of the object includes receiving a plurality of images of an object within the overlapping fields of view of the main camera and the one or more support cameras; The method, wherein calculating the updated pose includes calculating updated viewpoints of the main camera and the one or more support cameras.
Citation Information
Patent Citations
Method for object pose estimation, apparatus for object pose estimation, method for object estimation pose refinement and computer readable medium
JP2013050947A
Image processor, and image processing method
JP2015115041A
Image processing device, image processing method and program
JP2017102622A
Image retrieving apparatus, image retrieving method, and setting screen used therefor
JP2019091138A
Mobile body behavior prediction device and mobile body behavior prediction method
JP2019109691A
