Systems and methods for surface modeling using polarization cues

Abstract

A computer-implemented method for surface modeling includes: receiving one or more polarization raw frames of a surface of a physical object, wherein the polarization raw frames are captured at different linear polarization angles by means of a polarization filter; extracting one or more first tensors in one or more polarization representation spaces from the polarization raw frames; and detecting surface properties of the surface of the physical object based on the one or more first tensors in the one or more polarization representation spaces.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. Provisional Patent Application No. 62 / 901,731, filed in the U.S. Patent and Trademark Office on September 17, 2019, and U.S. Provisional Patent Application No. 63 / 001,445, filed in the U.S. Patent and Trademark Office on March 29, 2020, the entire disclosures of which are incorporated herein by reference. Technical Field

[0003] Aspects of embodiments of the present disclosure relate to the field of computer vision and modeling the surfaces of objects using machine vision. Background Art

[0004] Large-scale surface modeling is often required in manufacturing for various reasons. One application area is the manufacturing of automobiles and automotive components, where surface modeling using computer vision or machine vision provides a method for automatically inspecting scanned surfaces, which can increase efficiency and reduce manufacturing costs.

[0005] Large-scale surface modeling also has other applications, such as laboratory work and inspection of individual workpieces outside of large-scale manufacturing. Summary of the Invention

[0006] Aspects of embodiments of the present disclosure relate to surface modeling that uses light polarization (e.g., the rotation of light waves) to provide an additional channel of information to the process of characterizing the surface of an object. Aspects of embodiments of the present disclosure may be applied in scenarios such as manufacturing, where surface characterization is used to perform object inspection as part of a quality assurance process, such as detecting defective goods produced on a production line and removing or repairing those defective objects.

[0007] According to one embodiment of the present disclosure, a computer-implemented method for surface modeling includes: receiving one or more polarization raw frames of a surface of a physical object, the polarization raw frames being captured at different polarizations by a polarization camera including a polarization filter; extracting one or more first tensors in one or more polarization representation spaces from the polarization raw frames; and detecting surface characteristics of the surface of the physical object based on the one or more first tensors in the one or more polarization representation spaces.

[0008] The one or more first tensors in the one or more polarization representation spaces may include: a degree of linear polarization (DOLP) image in a DOLP representation space; and an angle of linear polarization (AOLP) image in an AOLP representation space.

[0009] The one or more first tensors may further include one or more non-polarized tensors in one or more non-polarized representation spaces, and the one or more non-polarized tensors may include one or more intensity images in an intensity representation space.

[0010] The one or more intensity images may include: a first color intensity image; a second color intensity image; and a third color intensity image.

[0011] The surface characterization may include detection of defects in the surface of the physical object.

[0012] Detecting the surface characteristic may include: loading a stored model corresponding to the location of the surface of the physical object; and calculating the surface characteristic based on the stored model and the one or more first tensors in the one or more polarization representation spaces.

[0013] The stored model may include one or more reference tensors in the one or more polarization representation spaces, and calculating the surface properties may include calculating a difference between the one or more reference tensors in the one or more polarization representation spaces and the one or more first tensors.

[0014] The difference can be calculated using the Fresnel distance.

[0015] The stored model may include a reference three-dimensional grid, and calculating the surface characteristics may include: calculating a three-dimensional point cloud of the surface of the physical object based on the one or more first tensors in the one or more polarization representation spaces; and calculating the difference between the three-dimensional point cloud and the reference three-dimensional grid.

[0016] The stored model may include a trained statistical model configured to compute a prediction of a surface characteristic based on the one or more first tensors in the one or more polarization representation spaces.

[0017] The trained statistical model may include an anomaly detection model.

[0018] The trained statistical model may include a convolutional neural network trained to detect defects in the surface of the physical object.

[0019] The trained statistical model may include a trained classifier trained to detect defects.

[0020] According to one embodiment of the present disclosure, a system for surface modeling includes: a polarization camera including a polarization filter, the polarization camera being configured to capture polarization raw frames at different polarizations; and a processing system including a processor and a memory storing instructions, the instructions, when executed by the processor, causing the processor to: receive one or more polarization raw frames of a surface of a physical object, the polarization raw frames corresponding to different polarizations of light; extract one or more first tensors in one or more polarization representation spaces from the polarization raw frames; and detect surface properties of the surface of the physical object based on the one or more first tensors in the one or more polarization representation spaces.

[0021] The one or more first tensors in the one or more polarization representation spaces may include: a degree of linear polarization (DOLP) image in a DOLP representation space; and an angle of linear polarization (AOLP) image in an AOLP representation space.

[0022] The one or more first tensors may also include one or more non-polarization tensors in one or more non-polarization representation spaces, and the one or more non-polarization tensors may include one or more intensity images in an intensity representation space.

[0023] The one or more intensity images may include: a first color intensity image; a second color intensity image; and a third color intensity image.

[0024] The surface characterization may include detection of defects in the surface of the physical object.

[0025] The memory may also store instructions that, when executed by the processor, cause the processor to detect the surface characteristics by: loading a storage model corresponding to the position of the surface of the physical object; and calculating the surface characteristics based on the storage model and the one or more first tensors in the one or more polarization representation spaces.

[0026] The stored model may include one or more reference tensors in the one or more polarization representation spaces, and the memory may also store instructions that, when executed by the processor, cause the processor to calculate the surface properties by calculating the difference between the one or more reference tensors in the one or more polarization representation spaces and the one or more first tensors.

[0027] The difference can be calculated using the Fresnel distance.

[0028] The stored model may include a reference three-dimensional grid, and the memory may also store instructions that, when executed by the processor, cause the processor to calculate the surface characteristics by: calculating a three-dimensional point cloud of the surface of the physical object based on the one or more first tensors in the one or more polarization representation spaces; and calculating a difference between the three-dimensional point cloud and the reference three-dimensional grid.

[0029] The stored model may include a trained statistical model configured to compute a prediction of the surface characteristic based on the one or more first tensors in the one or more polarization representation spaces.

[0030] The trained statistical model may include an anomaly detection model.

[0031] The trained statistical model may include a convolutional neural network trained to detect defects in the surface of the physical object.

[0032] The trained statistical model may include a trained classifier trained to detect defects. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The drawings, together with the specification, illustrate exemplary embodiments of the present invention, and, together with the description, serve to explain the principles of the present invention.

[0034] Figure 1A is a schematic depiction of the surface of an object (eg, a car) being inspected by a surface characterization system according to one embodiment of the present disclosure.

[0035] Figure 1B is a schematic block diagram of a surface characterization system according to one embodiment of the present disclosure.

[0036] Figure 2A is an image or intensity image of a scene in which a real transparent ball is placed on a photographic print depicting another scene containing two transparent balls (the "cheater") and some background clutter.

[0037] Figure 2B Describes the use of overlaid segmentation masks computed with a Mask Region-based Convolutional Neural Network (Mask R-CNN). Figure 2A The intensity image identifies instances of a transparent ball, where the real transparent ball is correctly identified as an instance and the two deceivers are incorrectly identified as instances.

[0038] Figure 2C is a polarization angle image calculated from a captured polarization raw frame of a scene according to one embodiment of the present invention.

[0039] Figure 2D FIG. 1 shows an embodiment of the present invention using overlapping segmentation masks calculated using polarization data. Figure 2A Intensity image of , where the real transparent ball is correctly identified as an instance, and the two spoofs are correctly rejected as instances.

[0040] Figure 3 It is a high-level description of the interaction of light with transparent and non-transparent objects (e.g., diffusion and / or reflection).

[0041] Figure 4 is a graph of the energy of light transmitted and reflected over a range of angles of incidence for a surface with a refractive index of approximately 1.5.

[0042] Figure 5 is a block diagram of a processing circuit 100 for computing a surface characterization output based on polarization data, according to one embodiment of the present invention.

[0043] Figure 6 is a flow chart of a method for performing surface characterization based on an input image to calculate a surface characterization output according to one embodiment of the present invention.

[0044] Figure 7A is a block diagram of a feature extractor according to one embodiment of the present invention.

[0045] Figure 7B FIG. 4 is a flow chart illustrating a method for extracting features from a polarization raw frame according to an embodiment of the present invention.

[0046] Figure 8A is a block diagram of a predictor according to one embodiment of the present invention.

[0047] Figure 8B is a flow chart describing a method for detecting surface characteristics of an object according to one embodiment of the present invention. DETAILED DESCRIPTION

[0048] In the following detailed description, only certain exemplary embodiments of the present invention are shown and described by way of illustration. As will be appreciated by those skilled in the art, the present invention can be implemented in many different forms and should not be construed as being limited to the embodiments set forth herein. Throughout the specification, similar reference numerals represent similar elements.

[0049] As used herein, the term "surface modeling" refers to capturing information about the surface of a real-world object, such as the three-dimensional shape of the surface, and may also include capturing color (or "texture") information about the surface and other information about the reflectivity of the surface (e.g., bidirectional reflectance distribution function or BRDF).

[0050] Surface profile inspection is important in analyzing the inherent shape and curvature properties or characteristics of a surface. Surface modeling of real-world objects has applications in many fields where surface characterization is required. For example, in manufacturing, surface modeling can be used to perform inspections of objects produced through a manufacturing process, enabling the detection of defects in objects (or manufactured goods or workpieces) and the removal of those defective objects from the manufacturing stream. One application area is the manufacture of automobiles and automobile parts, such as in the automated inspection of defective automobile parts, where a computer vision or machine vision system (e.g., using one or more cameras) captures images of automobile parts and generates classification results and / or other inspection information about the quality of the part, such as whether a window is scratched or a door panel has a dent. Applications using surface modeling techniques from computer vision to perform automated inspections of scanned surfaces improve efficiency and reduce manufacturing costs, such as by detecting errors early in the manufacturing or assembly process.

[0051] Compared to contact-based three-dimensional (3-D) scanners, for example, which detect objects by physical touch, computer vision and machine vision technologies enable fast and contactless surface modeling. However, comparable computer vision techniques, whether performed passively (e.g., without additional lighting) or actively (e.g., with active lighting that emits structured light), may not be able to reliably determine what could be considered "optically challenging" surface properties. These can be situations where the color of a defect is very similar to the background color of the surface on which the defect appears. For example, defects such as scratches in window glass or in the clear coat of a glossy paint, and shallow dents in painted or unpainted metal surfaces, can often be difficult to see in a standard color image of the surface because the color (or texture) change caused by these defects may be relatively small. In other words, the contrast between the color of the defect and the color of the defect-free (or "clean") surface may be relatively small, such as when a dent in a painted door panel has the same color as the undented portion.

[0052] Thus, some aspects of embodiments of the present disclosure relate to detecting defects in objects based on the polarization signature of the object, as calculated based on raw polarization frames captured of the inspected object using one or more polarization cameras (e.g., cameras that include a polarization filter in the optical path). In some embodiments, polarization-enhanced imaging can provide orders of magnitude improvements in the characterization of surface shape, including the accuracy of the detected direction of the surface normal. A smooth surface that is aesthetically pleasing cannot have bumps or depressions, which are essentially localized curvature variations, which are defined by their surface normal representation. Thus, some embodiments of the present disclosure can be applied to smoothness detection and shape fidelity in the high-precision manufacturing of industrial parts. One use case involves inspecting manufactured parts before they leave an assembly line for delivery to the end customer. In many manufacturing systems, manufactured parts leave the assembly line at a high rate on a conveyor system (e.g., on a conveyor belt), and to increase throughput, it is required that the parts be inspected while they are still moving, with very short time between parts.

[0053] Thus, some aspects of embodiments of the present disclosure relate to systems and methods for surface characterization, including by capturing polarization raw frames of the surface to be characterized, and computing a characterization based on those polarization raw frames, such as detecting defects in the surface.

[0054] Figure 1A is a schematic depiction of the surface of an object (e.g., a car) being inspected by a surface characterization system according to one embodiment of the present disclosure. Figure 1A In the arrangement shown, the object to be inspected 1 may be within a scene or environment. For example, in a factory or other manufacturing environment, the object to be inspected 1 may be located on an assembly line and may be moved on a conveyor system 40 such as a conveyor belt or an overhead conveyor (e.g., an overhead chain conveyor). The object to be inspected 1 may have one or more surfaces ( Figure 1A Surfaces 2, 3, and 4 are labeled in the figure, which are imaged by one or more polarization cameras 10. The polarization cameras 10 can be mounted on a mount, which can be a movable mount, such as on the end effector of a robotic arm 32, or a fixed mount, such as a gantry 34 that is above or part of a conveyor system. The polarization cameras 10 capture polarization raw frames (images) 18 of the respective surfaces 2, 3, and 4 of the inspected object 1, wherein each polarization camera 10 includes a polarization filter in its optical path.

[0055] Figure 1B is a schematic block diagram of a surface characterization system according to one embodiment of the present invention. In particular, Figure 1BThe polarization camera 10 is described as being configured to image the surface 2 of the inspected object 1. Figure 1B In the illustrated embodiment, the polarization camera 10 has a lens 12 with a field of view, wherein the lens 12 and the camera 10 are oriented such that the field of view includes an inspected surface (e.g., inspected surface 2 of inspected object 1). The lens 12 is configured to direct (e.g., focus) light from the scene (e.g., from the inspected surface) onto a light-sensitive medium such as an image sensor 14 (e.g., a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor).

[0056] The polarization camera 10 further includes a polarizer, a polarization filter, or a polarization mask 16 placed in the optical path between the scene 1 and the image sensor 14. According to various embodiments of the present disclosure, the polarizer or polarization mask 16 is configured to enable the polarization camera 10 to capture images of the scene 1 using the polarizer set at various specified angles (e.g., rotated at 45°, 60°, or at non-uniform intervals).

[0057] As an example, Figure 1B An embodiment is shown in which the polarization mask 16 is a polarization mosaic that is aligned with the pixel grid of the image sensor 14 in a manner similar to the red-green-blue (RGB) color filters (e.g., a Bayer filter) of a color camera. In a manner similar to how a color filter mosaic filters incident light based on wavelength so that each pixel in the image sensor 14 receives light in a specific portion of the spectrum (e.g., red, green, or blue) according to the pattern of the color filters of the mosaic, the polarization mask 16 using the polarization mosaic filters light based on linear polarization so that different pixels receive light at different linear polarization angles (e.g., at 0°, 45°, 90°, and 135°, or at 0°, 60°, and 120°). Thus, a polarization camera 10 using a polarization mask 16 such as that shown in FIG. 1 is capable of simultaneously or synchronously capturing light of four different linear polarizations. An example of a polarization camera is manufactured by the Wilsonville, Oregon, company. Systems Company Systems, Inc. S polarization camera ( S Polarization Camera).

[0058] While the above description relates to some possible implementations of polarization cameras using polarization mosaics, embodiments of the present disclosure are not limited thereto and encompass other types of polarization cameras capable of capturing images at multiple different polarizations. For example, polarization mask 16 may have fewer than four polarizations or more than four different polarizations, or may have polarizations at angles different from those described above (e.g., at polarization angles of 0°, 60°, and 120°, or at polarization angles of 0°, 30°, 60°, 90°, 120°, and 150°). As another example, polarization mask 16 may be implemented using an electronically controlled polarization mask, such as an electro-optical modulator (e.g., which may include a liquid crystal layer), in which the polarization angles of individual pixels of the mask can be independently controlled, such that different portions of image sensor 14 receive light having different polarizations. As another example, the electro-optical modulator may be configured to transmit light of different linear polarizations when capturing different frames, for example, so that the camera captures images with the entire polarization mask sequentially set to different linear polarizer angles (e.g., sequentially set to: 0°; 45°; 90°; or 135°). As another example, the polarization mask 16 may include a mechanically rotatable polarization filter, allowing the polarization camera 10 to capture different polarization raw frames, wherein the polarization filter is mechanically rotated relative to the lens 12 to transmit light of different polarization angles to the image sensor 14. Furthermore, while the above examples involve the use of linear polarization filters, embodiments of the present disclosure are not limited thereto and also include the use of a polarization camera including a circular polarization filter (e.g., a linear polarization filter with a quarter-wave plate). Thus, in various embodiments of the present disclosure, the polarization camera uses a polarization filter to capture multiple polarization raw frames with different light polarizations, such as different linear polarization angles and different circular polarizations (e.g., handedness).

[0059] As a result, the polarization camera 10 captures a plurality of input images 18 (or polarization raw frames) of a scene including the inspected surface 2 of the inspected object 1. In some embodiments, each polarization raw frame 18 corresponds to a polarization image obtained after a polarization filter or polarizer at a different polarization angle φ. pol In one embodiment, the polarization camera 10 is configured to detect light in various portions of the electromagnetic spectrum, such as the human-visible portion of the electromagnetic spectrum, the red, green, and blue portions of the human-visible spectrum, and the invisible portions of the electromagnetic spectrum, such as infrared and ultraviolet.

[0060] In some embodiments of the present disclosure, such as some of the embodiments described above, different polarization raw frames are captured by the same polarization camera 10 and, therefore, may be captured from substantially the same pose (e.g., position and orientation) relative to the scene 1. However, the embodiments of the present disclosure are not limited in this regard. For example, the polarization camera 10 may move relative to the scene 1 between different polarization raw frames (e.g., when capturing different raw polarization raw frames corresponding to different polarization angles at different times, such as in the case of a mechanically rotated polarization filter) because the polarization camera 10 has moved or because the object 1 has moved (e.g., if the object is on a moving conveyor system). In some embodiments, different polarization cameras capture images of the object at different times, but at substantially the same pose relative to the object (e.g., different cameras capture images of the same surface of the object at different points in the conveyor system). Therefore, in some embodiments of the present disclosure, different polarization raw frames are captured using the polarization camera 10 at different poses relative to the inspected object 1 and / or inspected surface 2, or at the same relative pose.

[0061] The polarization raw frame 18 is provided to a processing circuit 100, described in more detail below, which calculates a characterization output 20 based on the polarization raw frame 18. Figure 1B In the embodiment shown, the characterization output 20 includes regions 21 in the image of the surface 2 where defects (eg, dents in a car door) are detected.

[0062] Figure 2A 、 2B , 2C and 2D provide background for illustrating a segmentation map calculated based on polarized raw frames by a comparative approach and semantic segmentation or instance segmentation according to an embodiment of the present disclosure. In more detail, Figure 2A is an image or intensity image of a scene in which a real transparent ball is placed on a photographic print depicting another scene containing two transparent balls (the "cheater") and some background clutter. Figure 2B Shows the overlap Figure 2AThe segmentation mask calculated by the Mask Region-based Convolutional Neural Network (Mask R-CNN) on the intensity image uses different line patterns to identify instances of the transparent ball, where the real transparent ball is correctly identified as an instance and the two spoofed transparent balls are incorrectly identified as instances. In other words, the Mask R-CNN algorithm is fooled into labeling the two spoofed transparent balls as instances of the actual transparent ball in the scene.

[0063] Figure 2C is an angle of linear polarization (AOLP) image calculated from a captured polarization raw frame of a scene according to one embodiment of the present invention. Figure 2C As shown, transparent objects have very unique textures in polarization space such as the AOLP domain, where geometrically related signatures are present on edges and distinct or unique or specific patterns appear on the surface of the transparent object under linear polarization angles. In other words, the intrinsic texture of the transparent object (e.g., as opposed to the extrinsic texture taken from the background surface visible through the transparent object) is very distinct in polarization space. Figure 2C The polarization angle image is better than Figure 2A The intensity is more visible in the image.

[0064] Figure 2D FIG. 1 shows an embodiment of the present invention using overlapping segmentation masks calculated using polarization data. Figure 2A Intensity images of where the real transparent ball is correctly identified as an instance using the overlapping line pattern, and the two deceivers are correctly excluded as instances (e.g., with Figure 2B on the contrary, Figure 2D Excluding overlapping line patterns on the two cheaters). Although Figure 2A 、 2B , 2C and 2D show examples related to detecting a real transparent object in the presence of a deceptive transparent object, but the embodiments of the present disclosure are not limited thereto and can also be applied to other optically challenging objects, such as transparent, translucent and non-matte or non-Lambertian objects, as well as non-reflective (e.g., matte black objects) and multipath inducing objects.

[0065] Thus, some aspects of embodiments of the present disclosure involve extracting tensors in a representation space (or first tensors in a first representation space, such as polarization feature maps) from a polarization raw frame to provide as input to a surface characterization algorithm or other computer vision algorithm. These first tensors in the first representation space may include polarization feature maps that encode information about the polarization of light received from the scene, such as Figure 2C AOLP images, degree of linear polarization (DOLP) feature maps, etc. (e.g., Stokes vectors or other combinations of transformations from respective polarization raw frames). In some embodiments, these polarization feature maps are combined with non-polarization feature maps (e.g., Figure 2A ) to provide an additional information channel used by semantic segmentation algorithms.

[0066] While embodiments of the present invention are not limited to use with specific surface characterization algorithms, some aspects of embodiments of the present invention relate to a deep learning framework for polarization-based surface characterization of transparent objects (e.g., glass windows of vehicles and glossy layers of transparent paint) or other optically challenging objects (e.g., transparent, semi-transparent, non-Lambertian, multipath-introducing objects, and non-reflective (e.g., very dark) objects), where these frameworks may be referred to as polarization convolutional neural networks (Polarized CNNs). The Polarization CNN framework includes a backbone adapted to process polarization-specific textures and can be coupled with other computer vision architectures such as Mask R-CNN (e.g., to form a Polarized Mask R-CNN architecture) to produce an accurate and robust characterization solution for transparent and other optically challenging objects. Furthermore, the approach is applicable to scenes with a mixture of transparent and non-transparent (e.g., opaque objects) and can be used to characterize transparent, semi-transparent, non-Lambertian, multipath-introducing, dark, and turbid surfaces of one or more objects being inspected.

[0067] Polarization feature representation space

[0068] Some aspects of the embodiments of the present disclosure relate to systems and methods for extracting features from the polarization raw frame in operation 650, wherein these extracted features are used for robust detection of optically challenging features in the surface of the object in operation 690. In contrast, comparison techniques that rely solely on intensity images may not be able to detect these optically challenging features or surfaces (e.g., comparing Figure 2A The intensity image and Figure 2CAOLP image, as discussed above). The term “first tensor” in “first representation space” will be used herein to refer to features computed (e.g., extracted) from the polarization raw frames 18 captured by the polarization camera, where these first representation spaces include at least polarization feature spaces (e.g., feature spaces such as AOLP and DOPL that contain information about the polarization of light detected by the image sensor), and may also include non-polarization feature spaces (e.g., feature spaces that do not require information about the polarization of light reaching the image sensor, such as images computed solely based on intensity images captured without any polarization filter).

[0069] The interaction between light and transparent objects is rich and complex, but it is the material of an object that determines its transparency in visible light. For many transparent household items, most of the visible light passes directly through, while a small portion (about 4% to about 8%, depending on the refractive index) is reflected. This is because light in the visible part of the spectrum has insufficient energy to excite atoms in transparent objects. As a result, the texture (e.g., appearance) of objects behind (or visible through) the transparent object dominates the appearance of the transparent object. For example, when looking at a clear glass cup or tumbler on a table, the appearance of objects on the other side of the glass (e.g., the surface of the table) typically dominates what is seen through the cup. This property leads to several difficulties when trying to detect surface properties of transparent objects such as window glasses and glossy transparent layers of paint based solely on intensity images:

[0070] Figure 3 It is a high-level description of the interaction of light with transparent and non-transparent objects (e.g., diffusion and / or reflection). Figure 3 As shown, polarization camera 10 captures a polarization raw frame of a scene including a transparent object 302 in front of an opaque background object 303. Light 310 reaching image sensor 14 of polarization camera 10 contains polarization information from both transparent object 302 and background object 303. Compared to light 313 that reflects from background object 303 and passes through transparent object 302, a small portion of reflected light 312 from transparent object 302 is heavily polarized and therefore has a large impact on polarization measurements.

[0071] Similarly, light that strikes a surface can interact with the shape of the surface in various ways. For example, a surface with a glossy paint can behave essentially like Figure 3A transparent object is shown in front of an opaque object, where the interaction between light and a transparent or translucent layer (or clear coating) of glossy paint causes the light reflected from the surface to be polarized based on the properties of the transparent or translucent layer (e.g., based on the layer's thickness and the surface normal), which is encoded in the light reaching the image sensor. Similarly, as discussed in more detail below with respect to shape from polarization (SfP) theory, changes in surface shape (e.g., the direction of the surface normal) can cause significant changes in the polarization of light reflected from the object's surface. For example, a smooth surface may generally exhibit the same polarization characteristics throughout, but scratches or dents in the surface change the direction of the surface normal in those areas, and light reaching the scratches or dents may be polarized, attenuated, or reflected differently from other parts of the object's surface. Models of the interaction between light and matter typically consider three fundamentals: geometry, illumination, and material. Geometry is based on the shape of the material. Illumination includes the direction and color of the illumination. Materials can be parameterized by the refractive index or angular reflectance / transmission of light. This angular reflectance is called the bidirectional reflectance distribution function (BRDF), although other functional forms can more accurately represent some situations. For example, the bidirectional subsurface scattering distribution function (BSSRDF) will be more accurate in the context of materials that exhibit subsurface scattering, such as marble or wax.

[0072] Light 310 reaching the image sensor 14 of the polarization camera 10 has three measurable components: the intensity of the light (intensity image / I), the percentage or proportion of linearly polarized light (degree of linear polarization / DOLP / ρ), and the direction of linear polarization (angle of linear polarization / AOLP / φ). These properties encode information about the surface curvature and material of the imaged object, which can be used by the predictor 800 to detect transparent objects, as described in more detail below. In some embodiments, the predictor 800 can detect other optically challenging objects based on similar polarization properties of light passing through translucent objects and / or light interacting with multipath-inducing objects or passing through non-reflective objects (e.g., matte black objects).

[0073] Thus, some aspects of embodiments of the present invention relate to using a feature extractor 700 to compute a first tensor in one or more first representation spaces, which may include derived feature maps based on intensity I, DOLPρ, and AOLPφ. The feature extractor 700 may generally extract information into a first representation space (or first feature space) including a polarization representation space (or polarization feature space) such as a "polarization image," in other words, an image extracted based on a polarization raw frame that is not otherwise computable from an intensity image (e.g., an image captured by a camera that does not include a polarization filter or other mechanism for detecting the polarization of light reaching its image sensor), wherein these polarization images may include a DOLPρ image (in a DOLP representation space or feature space), an AOLPφ image (in an AOLP representation space or feature space), other combinations of polarization raw frames computed from Stokes vectors, and other images (or more generally, a first tensor or first feature tensor) of information computed from the polarization raw frames. The first representation space may include a non-polarization representation space, such as an intensity I representation space.

[0074] Measuring the intensity I, DOLPρ and AOLPφ at each pixel requires polarization at different angles φ after the polarization filter (or polarizer). pol 3 or more polarization raw frames of the captured scene (e.g., because there are three unknown values to determine: intensity I, DOLPρ, and AOLPφ). For example, S polarization camera with polarization angle φ pol Capture polarization raw frames for 0, 45, 90, or 135 degrees, resulting in four polarization raw frames Here it is represented as I0, I 45 , I 90 and I 135 .

[0075] At each pixel The relationship between the intensity I, DOLPρ and AOLPφ can be expressed as:

[0076]

[0077] Therefore, by using four different polarization original frames (I0,I 45 ,I 90 and I 135 ), a system of four equations can be used to solve for the intensity I, DOLPρ, and AOLPφ.

[0078] The shape-from-polarization (SfP) theory (see, for example, Gary A Atkinson and Edwin R Hancock, Recovery of surface orientation from diffuse polarization. IEEE transactions on image processing, 15(6):1653-1664, 2006) states that when diffuse reflection dominates, the refractive index (n) of the surface normal, the azimuth (θ a ) and the zenith angle (θ z ) and the φ and ρ components of light from the object follow the following properties:

[0079]

[0080] φ=θ a (3)

[0081] And when specular reflections dominate:

[0082]

[0083]

[0084] where, in both cases, ρ increases with θ z increases exponentially, and if the refractive index is the same, the specular reflection is much more polarized than the diffuse reflection.

[0085] Thus, some aspects of embodiments of the present disclosure involve applying SfP theory to detect the shape of a surface (e.g., the orientation of the surface) based on its raw polarization frame 18. This approach enables characterizing the shape of an object without using other computer vision techniques for determining the shape of the object, such as time-of-flight (ToF) depth sensing and / or stereo vision techniques, although embodiments of the present disclosure may be used in conjunction with such techniques.

[0086] More formally, aspects of embodiments of the present disclosure relate to computing a first tensor 50 in a first representation space, including extracting the first tensor in a polarization representation space based on a polarization raw frame captured by a polarization camera 10 in operation 650, such as forming a polarization image (or extracting a derived polarization signature map).

[0087] The light from a transparent object has two components: the reflected intensity I r , reflection DOLPρ r and reflection AOLPφ r The reflected part, including the refractive intensity I t , refraction DOLPρt and Refraction AOLPφ t The intensity of a single pixel in the resulting image can be written as:

[0088] I=I r +I t (6)

[0089] When the linear polarization angle φ pol When a polarization filter is placed in front of the camera, the value at a given pixel is:

[0090]

[0091] According to L r ,ρ r ,φ r ,I t ,ρ t and φ t Solve the above expressions to find the pixel values in the DOLPρ image and the pixel values in the AOLPφ image:

[0092]

[0093]

[0094] Therefore, according to one embodiment of the present disclosure, the above equations (7), (8) and (9) provide a model for forming a first tensor 50 in a first representation space, including the intensity image I, the DOLP image ρ and the AOLP image φ, wherein the use of a polarization image or a tensor in a polarization representation space (including the DOLP image ρ and the AOLP image φ based on equations (8) and (9)) enables reliable detection of optically challenging surface features of an object that are typically not detectable by a comparison system that uses only the intensity I image as input.

[0095] In more detail, the first tensor in the polarization representation space (in the derived feature map 50), such as the DOLP image ρ and the AOLP image φ, can reveal surface properties of an object that might otherwise appear textureless in the intensity domain I. Transparent objects may have texture that is not visible in the intensity domain I because the intensity is strictly dependent on I r / I t The ratio of (see equation (6)). tUnlike opaque objects with an intensity of λ = 0, transparent objects transmit most of the incident light and reflect only a small portion of it. As another example, thin or small deviations in the shape of an otherwise smooth surface (or smooth portions of other rough surfaces) may be essentially invisible or have low contrast in the intensity I domain (e.g., a domain that does not take into account the polarization of light), but may be very visible or have high contrast in a polarization representation space such as DOLPρ or AOLPφ.

[0096] Therefore, one exemplary method of acquiring surface topography is to use polarization cues in conjunction with geometric regularization. The Fresnel equation relates AOLPφ and DOLPρ to the surface normal. These equations may be useful for anomaly detection by utilizing the so-called polarization pattern of the surface. A polarization pattern is a tensor of size [M, N, K], where M and N are the horizontal and vertical pixel dimensions, respectively, and where K is the polarization data channel, the size of which may vary. For example, if circular polarization is ignored and only linear polarization is considered, then K will be equal to two, since linear polarization has both a polarization angle and a degree of polarization (AOLPφ and DOLPρ). Similar to the Moire pattern, in some embodiments of the present disclosure, the feature extraction module 700 extracts polarization patterns in polarization representation space (e.g., AOLP space and DOLP space). In the above shown Figure 1A and Figure 1B In the exemplary characterization output 20 shown in FIG, the horizontal and vertical dimensions correspond to the lateral field of view of a narrow strip or patch of the surface 2 captured by the polarization camera 10. However, this is an exemplary case: in various embodiments, the strip or patch of the surface may be vertical (e.g., much taller than wide), horizontal (e.g., much wider than tall), or have a more conventional field of view (FoV) that tends to be closer to a square (e.g., a width-to-height ratio of 4:3 or 16:9).

[0097] While the foregoing discussion provides a specific example of a linear polarization-based polarization representation space in the case of using a polarization camera with one or more linear polarization filters to capture polarization raw frames corresponding to different linear polarization angles and calculating tensors in linear polarization representation spaces such as DOLP and AOLP, embodiments of the present disclosure are not limited thereto. For example, in some embodiments of the present disclosure, the polarization camera includes one or more circular polarization filters configured to pass only circularly polarized light, and wherein a polarization pattern or first tensor in the circular polarization representation space is further extracted from the polarization raw frames. In some embodiments, these additional tensors in the circular polarization representation space are used alone, while in other embodiments, they are used in conjunction with tensors in linear polarization representation spaces such as AOLP and DOLP. For example, a polarization pattern comprising a tensor in the polarization representation space may include tensors in the circular polarization space, AOLP, and DOLP, wherein the polarization pattern may have dimensions [M, N, k], where K is three, to further comprise a tensor in the circular polarization representation space.

[0098] Figure 4 is a graph of the energy of light transmitted and reflected over a range of angles of incidence for a surface with a refractive index of approximately 1.5. Figure 4 As shown, the transmitted energy ( Figure 4 ) and reflected energy ( Figure 4 The slope of the line (shown as a dashed line in FIG) is relatively small at low angles of incidence (e.g., at angles closer to perpendicular to the surface plane). Thus, when the angle of incidence is low (e.g., close to perpendicular to the surface, in other words, close to the surface normal), small differences in surface angle may be difficult to detect in the polarization pattern (low contrast). On the other hand, the slope of the reflected energy increases from flat as the angle of incidence increases, and the slope of the transmitted energy decreases from flat (to have a larger absolute value) as the angle of incidence increases. Figure 4 In the example shown for a refractive index of 1.5, the slopes of the two lines become substantially steeper starting at an angle of incidence of approximately 60°, and their slopes become very steep at an angle of incidence of approximately 80°. For different materials, the specific shape of the curves may vary depending on the refractive index of the material. Thus, at an angle of incidence corresponding to the steeper portion of the curve (e.g., an angle close to parallel to the surface, such as approximately 80° in the case of a refractive index of 1.5, the slopes of the two lines become substantially steeper starting at an angle of incidence of approximately 60°, and their slopes become very steep at an angle of incidence of approximately 80°. Figure 4 Capturing an image of the inspected surface (as shown) can improve the contrast and detectability of surface shape changes in the polarization raw frame 18, and can improve the detectability of such features in the tensor in the polarization representation space, because small changes in the angle of incidence (due to small changes in the surface normal) can cause large changes in the captured polarization raw frame.

[0099] Some aspects of embodiments according to the present disclosure relate to providing first tensors in a first representation space extracted from a polarization raw frame (e.g., including feature maps in the polarization representation space) as input to a predictor for calculating or detecting surface properties of transparent objects and / or other optically challenging surface properties of inspected objects. These first tensors may include derived feature maps, which may include an intensity feature map I, a degree of linear polarization (DOLP) ρ feature map, and an angle of linear polarization (AOLP) φ feature map, where the degree of linear polarization (DOLP) ρ feature map and the angle of linear polarization (AOLP) φ feature map are examples of polarization feature maps or tensors in the polarization representation space, with reference to feature maps that encode information related to the polarization of light detected by a polarization camera. In some embodiments, the feature maps or tensors in the polarization representation space are provided as input to, for example, a detection algorithm that utilizes the SfP theory to characterize the shape of the surface of the object imaged by the polarization camera 10.

[0100] Polarization-based surface characterization

[0101] like Figure 1A and 1B As shown, various aspects of embodiments of the present invention relate to systems and methods for performing surface characterization of an inspected object by capturing images of the surface of the object 1 using one or more polarization cameras 10, which capture polarization raw frames 18 that are analyzed by a processing system or processing circuit 100. Characterization of the surface can include detecting optically challenging surface features, such as surface features that may be difficult or impossible to detect using comparative computer vision or machine vision techniques that do not use polarization information. While some aspects of embodiments of the present disclosure relate to surface features corresponding to defects in manufactured products (e.g., defects such as cracks, tears, uneven application of paint or dye, the presence of surface contaminants, unintentional surface irregularities, or other geometric deviations from a reference model), embodiments of the present disclosure are not limited thereto and can be applied to detecting other surface features, such as detecting the positional boundaries between different types of materials, measuring the uniformity of the refractive index of a material across an area, characterizing the geometry of a surface treatment applied to a portion of a material (e.g., etching of a material and / or deposition of a material on a surface), and the like.

[0102] Figure 5 is a block diagram of a processing circuit 100 for computing a surface characterization output based on polarization data, according to one embodiment of the present invention. Figure 6 is a flow chart of a method 600 for performing surface characterization based on an input image to compute a surface characterization output, according to one embodiment of the present invention.

[0103] According to various embodiments of the present disclosure, processing circuit 100 is implemented using one or more electronic circuits configured to perform various operations described in greater detail below. Types of electronic circuits may include a central processing unit (CPU), a graphics processing unit (GPU), an artificial intelligence (AI) accelerator (e.g., a vector processor, which may include a vector arithmetic logic unit configured to efficiently perform operations common to neural networks, such as dot products and softmax (normalized exponential function)), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), and the like. For example, in some cases, aspects of the embodiments of the present disclosure are implemented as program instructions stored in a non-volatile computer-readable memory, which, when executed by an electronic circuit (e.g., a CPU, a GPU, an AI accelerator, or a combination thereof), perform the operations described herein to calculate the characterized output 20 from the input polarization raw frame 18. The operations performed by processing circuit 100 may be performed by a single electronic circuit (e.g., a single CPU, a single GPU, etc.), or may be distributed among multiple electronic circuits (e.g., multiple GPUs or a CPU in combination with a GPU). The multiple electronic circuits may be local to each other (e.g., on the same die, in the same package, or in the same embedded device or computer system) and / or may be remote from each other (e.g., communicating over a network such as a personal area network (PAN) such as Bluetooth, over a local area network such as a local wired and / or wireless network, and / or over a wide area network such as the Internet, in which case some operations are performed locally while other operations are performed on servers hosted by a cloud computing service). The one or more electronic circuits operating to implement the processing circuit 100 may be referred to herein as a computer or computer system, which may include memory storing instructions that, when executed by the one or more electronic circuits, implement the systems and methods described herein.

[0104] like Figure 5As shown, in some embodiments, the processing circuit 100 includes a feature extractor or feature extraction system 700 and a predictor 800 (e.g., a classical computer vision prediction algorithm and / or a trained statistical model such as a trained neural network), which is configured to calculate a prediction output 20 (e.g., a statistical prediction) regarding the surface characteristics of the object based on the output of the feature extraction system 700. Although some embodiments of the present disclosure are described herein in the context of a surface characterization system for detecting defects in the surface of a manufactured object, where detecting those surface defects may be optically challenging, the embodiments of the present disclosure are not limited thereto. For example, some aspects of the embodiments of the present disclosure may be applied to techniques for characterizing the surface of objects made of materials that are optically challenging to detect or have surface characteristics that are optically challenging to detect, such as the surfaces of translucent objects, multipath-introduced objects, objects that are not completely or substantially matte or Lambertian, and / or very dark objects. These optically challenging objects include objects and their surface features that are difficult to resolve or detect using images captured by a camera system that is insensitive to the polarization of light (e.g., based on images captured by a camera that does not have a polarization filter in the optical path or where different images are captured based on different polarization angles). For example, these surface features may have a surface appearance or color that is very similar to the surface on which the features appear (e.g., a dent has the same color as the underlying material, and a scratch on a transparent material such as glass may also be substantially transparent). Additionally, while embodiments of the present disclosure are described herein in the context of detecting optically challenging surface features, embodiments of the present disclosure are not limited to detecting only optically challenging surface defects. For example, in some embodiments, the predictor 800 is configured to detect both optically challenging surface features and surface features that are robustly detectable without the use of polarization information (e.g., using training a statistical model using training data).

[0105] Polarization can be used to detect surface features or characteristics that would otherwise be optically challenging to detect using intensity information alone (e.g., color intensity information). For example, polarization information can detect geometric changes and material changes in an object's surface. Material changes (or material variations), such as boundaries between different types of materials (e.g., a ferrous metal object on a dark road or a colorless liquid on a surface may both be essentially invisible in color space, but will have corresponding polarization signatures in polarization space), may be more visible in polarization space because differences in the refractive indices of different materials cause changes in the polarization of light. Similarly, differences in the specularity of various materials result in different changes in the polarization rotation phase angle, also leading to detectable features in polarization space that would otherwise be optically challenging to detect without the use of a polarization filter. Consequently, this results in contrast appearing in an image or tensor in a polarization representation space, where corresponding regions of a tensor computed in intensity space (e.g., a color representation space that does not account for the polarization of light) may not capture these surface features (e.g., where these surface features have low contrast or may be invisible in these spaces). Examples of optically challenging surface characteristics include: the specific shape of a surface (e.g., surface smoothness and deviations from ideal or acceptable physical design tolerances); the shape of surface roughness and surface roughness patterns (e.g., intentional etchings, scratches, and edges in the surfaces of transparent objects and machined parts); burrs and flash at the edges of machined and molded parts; etc. Polarization is also useful for detecting objects that have the same color but different material properties, such as scattering or refractive index.

[0106] like Figure 6 As shown, for example, reference Figure 1B In operation 610, the processing circuit 100 captures a polarization raw frame 18 of the surface 2 of the inspected object 1. For example, in some embodiments, the processing circuit 100 controls one or more polarization cameras 10 to capture a polarization raw frame 18 depicting a specific surface 2 of the object. In various embodiments of the present disclosure, one or more detection systems may be used to trigger the capture of a specific surface of the inspected object, such as a mechanical switch trigger (e.g., when a portion of the object or a conveyor system closes an electronic switch to signal the current position of the object), a laser trigger (e.g., when a portion of the object 1 blocks a laser beam from reaching a detector), or an optical trigger (e.g., when a camera system detects the presence of an object at a specific location).

[0107] Return Reference Figure 1AA polarization-enhanced imaging system or surface characterization system according to an embodiment of the present disclosure may use a polarization camera 10 mounted on a gantry positioned around a conveyor belt or mounted on an end effector of a robotic arm, which may be used to provide a patch-based image of an object 1 (e.g., an image of a patch or strip of the surface of the object 1) as the object 1 moves on the conveyor belt. In some embodiments of the present disclosure, the system automatically repositions the polarization camera 10 attached to a movable mount to place the polarization camera 10 such that the incident angle of light on the surface is at Figure 4 The position of the steeper or higher contrast portion of the curve shown (e.g., based on the general orientation of the inspected surface and the light source in the scene). In some embodiments of the present disclosure, an illumination source (e.g., a running light or a flashlight) can also be placed in a fixed position or attached to a movable mount (e.g., rigidly attached to a corresponding polarization camera or attached to an independently movable mount) to illuminate the surface of the object with light at an angle of incidence that makes the surface shape features of the object more easily detectable (e.g., at a high angle of incidence).

[0108] Thus, in some embodiments of the present disclosure, capturing the polarization raw frame 18 of the surface 2 of the inspected object 1 in operation 610 includes moving the polarization camera 10 and / or the illumination source to a position relative to the inspected surface 2 according to the specific characteristics of the surface 2 to be characterized. For example, in some embodiments, this involves automatically positioning the polarization camera 10 and / or the illumination source so that light from the illumination source illuminates the surface 2 at a high angle of incidence (e.g., approximately 80 degrees). In some embodiments of the present disclosure, the specific location where a high angle of incidence may be feasible will vary based on the specific shape of the surface to be inspected (e.g., the design of a car door may include different portions with significantly different surface normals, such as a recess at the door handle, the edge where the door meets the window, and a recess in the main surface of the door for style and / or aerodynamics).

[0109] In some embodiments of the present disclosure, the processing circuit 100 loads a profile associated with a type or category of object being inspected, wherein the profile includes a set of one or more poses to which the polarization camera 10 is to be moved relative to the inspected object 1. Different types or categories of objects having different shapes may be associated with different profiles, while manufactured objects of the same type or category are expected to have the same shape. (For example, different models of vehicles may have different shapes, and these different models of vehicles may be mixed on an assembly line. Therefore, the processing circuit 100 may select a profile corresponding to the type of vehicle currently being inspected from the set of different profiles.) Thus, the polarization camera 10 can automatically move through a series of poses stored in the profile to capture polarization raw frames 18 of the surface of the inspected object 1.

[0110] exist Figure 5 and Figure 6 In the illustrated embodiment, in operation 650, the feature extraction system 700 of the processing circuit 100 extracts one or more first feature maps 50 in one or more first representation spaces (including polarization images or polarization feature maps in various polarization representation spaces) from the input polarization raw frame 18 of the scene.

[0111] Figure 7A is a block diagram of a feature extractor 700 according to one embodiment of the present invention.

[0112] Figure 7B FIG. 1 is a flow chart illustrating a method for extracting features from a polarization raw frame according to an embodiment of the present invention. Figure 7A In the illustrated embodiment, the feature extractor 700 includes an intensity extractor 720 configured to extract an intensity image I52 in an intensity representation space (e.g., according to Equation (7), as an example of a non-polarized representation space), and a polarization feature extractor 730 configured to extract features in one or more polarized representation spaces. In some embodiments of the present disclosure, the intensity extractor 720 is omitted, and the feature extractor does not extract the intensity image I52.

[0113] like Figure 7B As shown, extracting the polarization image in operation 650 may include extracting a first tensor in a first polarization representation space from the polarization raw frame according to the first Stokes vector in operation 651. In operation 652, the feature extractor 700 further extracts a second tensor in a second polarization representation space from the polarization raw frame. For example, the polarization feature extractor 730 may include a DOLP extractor 740 configured to extract a DOLPρ image 54 (e.g., a first polarization image or a first tensor according to equation (8), where DOLP is used as the first polarization representation space) and an AOLP extractor 760 configured to extract an AOLPφ image 56 (e.g., a second polarization image or a second tensor according to equation (9), where AOLP is used as the second polarization representation space) from the provided polarization raw frame 18. In addition, in various embodiments, the feature extraction system 700 extracts two or more different tensors (e.g., n different tensors) in two or more representation spaces (e.g., n representation spaces), wherein the nth tensor is extracted in operation 614. As described above, in some embodiments of the present disclosure, the polarization feature extractor 730 extracts polarization features in a polarization representation space including a linear polarization representation space (e.g., extracting tensors in the aforementioned AOLP and DOLP representation spaces from a polarization raw frame captured using a linear polarization filter) and a circular polarization representation space (e.g., extracting tensors from a polarization raw frame captured using a circular polarization filter). In various embodiments, the representation space includes, but is not limited to, a polarization representation space.

[0114] The polarization representation space may include a combination of polarization raw frames according to Stokes vectors. As a further example, the polarization representation may include a modification or transformation of the polarization raw frames according to one or more image processing filters (e.g., a filter for increasing image contrast or a denoising filter). The feature maps 52, 54, and 56 in the first polarization representation space may then be provided to the predictor 800 for detecting surface characteristics based on the feature map 50.

[0115] Although Figure 7B The case of extracting two or more different tensors from the polarization raw frame 18 in more than two different representation spaces is shown, but embodiments of the present invention are not limited thereto. For example, in some embodiments of the present disclosure, only one tensor in a polarization representation space is extracted from the polarization raw frame 18. For example, one polarization representation space of the raw frame is AOLPφ and the other is DOLPρ (e.g., in some applications, AOLP may be sufficient to detect surface characteristics of transparent objects or other optically challenging objects such as translucent, non-Lambertian, multipath-introduced, and / or non-reflective objects).

[0116] Thus, extracting features such as polarization feature maps or polarization images from the polarization raw frame 18 produces a first tensor 50 from which optically challenging surface properties can be detected from the surface image of the inspected object. In some embodiments, the first tensor extracted by the feature extractor 700 can be an explicit derived feature (e.g., manually crafted by a human designer) associated with the underlying physical phenomena that may be exhibited in the polarization raw frame (e.g., the calculation of AOLP and DOLP images in linear polarization space and the calculation of tensors in circular polarization space, as described above). In some additional embodiments of the present disclosure, the feature extractor 700 extracts other non-polarization feature maps or non-polarization images, such as intensity maps of different colors of light (e.g., red, green, and blue light) and transformations of intensity maps (e.g., applying image processing filters to intensity maps). In some embodiments of the present disclosure, the feature extractor 700 can be configured to extract one or more features (e.g., features that are not manually specified by a human) that are automatically learned through an end-to-end supervised training process based on labeled training data. In some embodiments, these learned feature extractors may include deep convolutional neural networks, which may be used in conjunction with traditional computer vision filters (e.g., Haar wavelet transform, Canny edge detector, etc.).

[0117] Surface characterization based on tensors in representation spaces including polarization representation spaces

[0118] The feature map 50 (including the polarization image) in the first representation space extracted by the feature extraction system 700 is provided as input to the predictor 800 of the processing circuit 100 , which implements one or more prediction models to calculate the surface characterization output 20 in operation 690 .

[0119] In the case where the predictor 800 is a defect detection system, the prediction can be an image 20 (e.g., an intensity image) of the surface 2, in which a portion of the image is labeled 21 or highlighted as containing a defect. In some embodiments, the output of the defect detection system is a segmentation map, in which each pixel can be associated with one or more confidences that the pixel corresponds to the location of various possible classes (or types) of surface features (e.g., defects) that may be found in the objects that the surface characterization system is trained to inspect, or a confidence that the pixel corresponds to an anomaly in the image of the surface of the inspected object. In the case where the predictor is a classification system, the prediction can include multiple classes and corresponding confidences that the image depicts instances of each class (e.g., images depicting various types of defects or different types of surface features, such as smooth glass, etched glass, scratched glass, etc.). In the case where the predictor 800 is a classical computer vision prediction algorithm, the predictor can compute a detection result (e.g., detecting defects by comparing an extracted feature map in a first representation space with a model feature map in the first representation space, or identifying edges or regions with sharp or discontinuous changes in a feature map in an area that is expected to be smooth).

[0120] exist Figure 5 In the illustrated embodiment, the predictor 800 implements the defect detection system and calculates, at operation 690, the surface characteristic output 20 including the location of the detected defect, which is calculated based on the first tensor 50 in the extracted first representation space extracted from the input polarized raw frame 18. As described above, the feature extraction system 700 and the predictor 800 are implemented using one or more electronic circuits configured to perform their operations, as described in more detail below.

[0121] According to various embodiments of the present disclosure, a surface 2 of an object 1 imaged by one or more polarization cameras 10 is characterized according to a model associated with the surface. The specific details of the surface characterization performed by a surface characterization system according to embodiments of the present invention depend on the particular application and the surface being characterized.

[0122] Continuing with the above example of detecting defects on the surface of an automobile, different types of defects may appear on different surfaces of the automobile due to the location and method of manufacturing the various components and due to the types of materials used in the different components. For example, a painted metal door panel may exhibit different types of defects (e.g., scratches, dents) than a glass window (e.g., scratches, chips, and cracks), which in turn may exhibit defects that are different from those found in plastic parts (e.g., headlight covers, which may also exhibit scratches, chips, and cracks, but may also contain intended and intentional surface irregularities, including, for example, surface ridges and bumps and ejector pin marks).

[0123] As another example, in a machined metal part, some surfaces may be expected to be smooth and shiny, while other surfaces may be expected to be rough or have a particular physical pattern (e.g., a pattern of grooves, protrusions, or a random texture), where different surfaces of the machined part may have different tolerances.

[0124] Figure 8A FIG. 1 is a block diagram of a predictor according to an embodiment of the present invention. Figure 8A As shown, the predictor 800 receives an input tensor 50 in a first representation space. The predictor 800 may include a set of models 810 associated with different types of surfaces expected to be analyzed by the surface characterization system. Figure 8A In the illustrated embodiment, the predictor 800 may access m different models (e.g., different models stored in the memory of the processing circuit 100). For example, a first model 811 may be associated with the main surface of the door panel, a second model 812 may be associated with the handle portion of the door panel, and an mth model 814 may be associated with the taillight.

[0125] Figure 8B 6 is a flow chart describing a method 690 for detecting surface characteristics of an object according to one embodiment of the present invention. At operation 691, the processing system 100 selects a model corresponding to a current surface from a collection of models 810. In some embodiments, the particular model is selected based on metadata stored in a configuration file associated with the inspected object 1 and associated with a particular pose in which the polarization camera 10 captured the polarization raw frame 18.

[0126] In some embodiments of the present disclosure, the orientation of the inspected object is consistent from one object to the next. For example, in the case of automobile manufacturing, each assembled car may be moved along a conveyor system with its front end first (e.g., as opposed to some moves with the driver's side first and some moves with the rear of the vehicle first). Thus, based on known information about the position of the car on the conveyor system and its speed, images of different surfaces of the inspected object 1 can be reliably captured. For example, a camera located at a particular height on the driver's side of the car can be expected to image specific portions of the car's bumper, fender, wheel well, driver's side door, tailgate, and rear bumper. Based on the speed of the conveyor system and the trigger time at which the car enters the field of view of the surface characterization system, various surfaces of the car will be expected to be imaged at different times according to a profile associated with the type of object (e.g., the type, class, or model of car).

[0127] In some embodiments, the orientation of the inspected object may be inconsistent, and therefore a separate registration process may be used to determine which surfaces are being imaged by the polarization camera 10. In these embodiments, the configuration file may include a three-dimensional (3-D) model of the inspected object (e.g., a computer-aided design or CAD model of the physical object, or a three-dimensional mesh or point cloud model). Therefore, in some embodiments, a simultaneous localization and mapping (SLAM) algorithm is applied to determine which portions of the inspected object are being imaged by the polarization camera 10, and the determined positions are used to identify corresponding positions on the 3-D model, thereby enabling determination of which surfaces of the 3-D model are being imaged by the polarization camera 10. For example, a keypoint detection algorithm may be used to detect unique portions of the object, and the keypoints are used to match the orientation of the 3-D model to the orientation of the physical object 1 being inspected.

[0128] Thus, in some embodiments of the present disclosure, the surface registration module 820 of the prediction system 800 aligns the polarization raw frame 18 (and / or the tensor 50 in the representation space) captured by the polarization camera with a specific portion of the inspected object based on a profile associated with the object to select a model associated with the current surface imaged by the polarization raw frame 18 from the set of models 810.

[0129] In operation 693, the processing system applies the selected model using the surface analyzer 830 to calculate the surface characterization output 20 of the current surface. Details of various types of models and specific operations performed by the surface analyzer 830 based on these different types of models according to various embodiments of the present disclosure are described in more detail below.

[0130] Surface characterization by comparison with design and representational models

[0131] In some embodiments of the present disclosure, the stored model includes a feature map in a representation space calculated from a representative model of the inspected object (e.g., a design model), and the surface analyzer compares the feature map calculated from the captured polarization raw frame 18 with the stored representative (e.g., ideal) feature map in the same representation space.

[0132] For example, as described above, in some embodiments of the present disclosure, the representation space includes the degree of linear polarization (DOLP) ρ and the angle of linear polarization (AOLP) φ. In some such embodiments, the model 810 includes a reference 2-D and / or 3-D model of the surface (e.g., a CAD model) with its intrinsic surface normals. These intrinsic surface reference models are sometimes referred to as design surface normals and are the design target of the surface (e.g., the ideal shape of the surface), and therefore they represent the ground truth of the patch being inspected (e.g., the patch of the surface imaged by the set of polarization raw frames 18).

[0133] In such an embodiment, the feature extraction system 700 extracts surface normals using shape by polarization (SfP), and these surface normals are aligned by the surface registration module 820 with a reference 2-D and / or 3-D model (e.g., a CAD model) of the corresponding portion of the surface.

[0134] In this embodiment, the surface analyzer 830 performs a comparison between the surface normal represented by the tensor 50 in the representation space calculated from the polarization raw frame 18 and the design surface normal of the corresponding tensor 50 in the model 810 to find different areas, thereby identifying and marking different areas. For example, portions of the tensor 50 in the representation space calculated from the raw polarization frame 18 that differ from corresponding portions of the design surface normal (in the same representation space as the tensor 50) by more than a threshold amount are marked as different or potential defects, while other portions that differ by less than the threshold amount are marked as clean (e.g., non-defective). In various embodiments of the present disclosure, this threshold can be set based on, for example, the design tolerance for the surface being inspected and the sensitivity of the system (e.g., based on the noise level in the system, such as sensor noise in the image sensor 14 of the polarization camera 10).

[0135] Additionally, assuming that the region of interest has both a calculated surface normal and 3D coordinates of a surface of a design object loaded from a model selected from model 810, in some embodiments, the surface analyzer 830 converts the region into a 3D point cloud representing the shape of the imaged surface (e.g., using a polarization-determined shape equation), and the surface analyzer 830 performs further inspection and analysis on the generated 3D point cloud, such as by comparing the shape of the 3D point cloud with the shape of a corresponding surface in a reference 3D model. The comparison may include iteratively reorienting the point cloud to minimize the distance between points in the point cloud and the surface of the reference 3-D model, wherein points in the point cloud that are greater than a threshold distance from the surface of the reference 3-D model region of the inspected surface deviate from the reference model and may correspond to geometric defects (e.g., dents, burrs, or other surface irregularities).

[0136] As another example, manufactured parts that meet the same tolerances will have substantially the same polarization pattern under similar illumination (e.g., the same polarization pattern with variations due to manufacturing tolerances). The polarization pattern of an ideal or expected or reference part will be referred to as a template polarization pattern or reference tensor (which will correspond to a model selected from the set of models 810). In these embodiments, the feature extraction system 700 extracts a measured polarization pattern of the surface of the inspected object (e.g., the measured tensor in the first representation space of the AOLP and DOLP feature maps described above). If the surface of the object contains an anomaly, such as a tiny dent in the surface, the anomaly will appear in the measured polarization pattern, resulting in its classification as an anomalous polarization pattern that is different from the template polarization pattern or reference tensor in the first representation space (or has an area containing an anomaly, such as a tiny dent in the surface). Figure 1B 21 shown in FIG. ). On the other hand, a defect-free surface will generate a measured polarization pattern that matches the template polarization pattern or reference tensor (within tolerance) (eg, when the measured polarization pattern matches, then it is classified as a clean polarization pattern).

[0137] Some aspects of the embodiments of the present disclosure relate to mathematical operations for comparing the template polarization pattern and the measured polarization pattern. In some embodiments, a subtraction or arithmetic difference between the template and the anomalous polarization pattern is calculated to compare the patterns. However, as Figure 4 As shown, the Fresnel equations model the nonlinear relationship between the angle of incidence and the amount of energy transmitted and reflected, where the shape of the curve shifts depending on the refractive index ( Figure 4An example curve is shown for a refractive index of 1.5.) This nonlinear change in reflected energy for a similar change in surface normal at different angles of incidence may make it difficult to perform comparisons between polarization patterns (e.g., comparing a template polarization pattern to a measured polarization pattern). For example, a 1 degree change in angle of incidence near 60 degrees (e.g., an average angle of incidence of 60 degrees and a change in surface normal that causes a 0.5 degree change in angle of incidence to 60.5 degrees) will have a larger change in reflected energy than a similar change near 10 degrees (e.g., an average angle of incidence of 0 degrees and a change in surface normal that causes a 0.5 degree change in angle of incidence to 0.5 degrees). In other words, these embodiments will use a linear metric to compare nonlinear phenomena, which may cause detectability issues in the flatter neighborhood of the curve (e.g., portions of the curve with smaller first-order derivatives) or may cause saturation or overflow of the signal in the steeper neighborhood of the curve (e.g., portions of the curve with larger first-order derivatives).

[0138] Therefore, some aspects of the embodiments of the present disclosure involve using Fresnel subtraction to calculate the Fresnel distance to compare the template polarization pattern and the measured polarization pattern in a manner that takes into account the nonlinear relationship between the angle of incidence and the reflected or transmitted energy. Therefore, according to some aspects of the embodiments of the present disclosure, Fresnel subtraction is a nonlinear operator that allows linear comparison of surface normals. In fact, Fresnel subtraction allows Figure 4 The curve shown is linearized so that a relative microsurface deviation of 30 degrees can be represented by a consistent anomaly score (e.g., an anomaly score calculated based on the Fresnel distance) regardless of whether the original orientation is 0° or 60° (e.g., the average angle of incidence on the surface). In other words, according to embodiments of the present disclosure, the Fresnel distance is calculated using Fresnel subtraction, where the Fresnel distance between two polarization patterns is substantially independent of the original orientation of the surface (e.g., substantially independent of the average angle of incidence on the surface). In some embodiments of the present disclosure, the Fresnel subtraction function is parametrically learned using a pattern matching technique using symbolic regression. In some embodiments of the present disclosure, the Fresnel subtraction function is numerically approximated based on known Fresnel equations according to the refractive index of the material and the orientation of the surface, such as by dividing the measured reflected light by the percentage of energy reflected at the approximate angle of incidence of the light on the surface (e.g., the average angle of incidence over a substantially planar local patch of the surface), based on the assumption that the variation in the surface normal is small enough to be within a substantially or sufficiently linear neighborhood of the curve. In some embodiments of the present disclosure, closed-form equations are derived based on a priori knowledge of material properties, such as the refractive index of the material.

[0139] Because the Fresnel equations are refractive index dependent, the Fresnel subtraction also depends on the refractive index of the material (e.g. Figure 4The shape of the curve shown in is shifted depending on the refractive index. A manufactured component may have different refractive indices in different pieces (e.g., on different surfaces). In some embodiments of the present disclosure, a standard refractive index is selected based on the application's need to balance sensitivity with respect to different surfaces of the object (e.g., the contact surface of a manufactured component may be more important than the non-contact surfaces of those manufactured components, and therefore a refractive index closer to the contact surface may be selected). For example, the standard refractive index may be set to 1.5 and is assumed to be sufficiently close.

[0140] In some embodiments of the present disclosure, local calibration is performed using the design surface normal to determine the local smoothed refractive index for each patch, enabling a more accurate Fresnel subtraction method customized for each patch. In some embodiments, local calibration is performed by assuming that the refractive index is a scalar constant that is constant across different pixels and using information from different pixels to estimate the refractive index value for a given material. In some embodiments, local calibration is performed by estimating the refractive index value using the techniques described in the "refractive distortion" section of "Polarized 3D: High-quality depth sensing with polarization cues," Proceedings of the IEEE International Conference on Computer Vision, 2015, by Kadambi, Achuta, et al.

[0141] Thus, some aspects of embodiments of the present disclosure relate to detecting defects by comparing measured feature maps or tensors extracted from polarization raw frames captured from an inspected object with reference tensors or reference feature maps or template feature maps corresponding to a reference or template object (e.g., based on an ideal surface from a design such as a CAD model, or based on measurements of a known good object).

[0142] Surface feature detection using anomaly detection algorithms

[0143] In some embodiments of the present disclosure, anomaly detection is used to detect surface features. For example, in some cases, significant variations from one instance of the inspected object to the next can be expected. For example, manufacturing processes can cause irregular and uneven variations in the polarization pattern exhibited by a material. While these variations may be within manufacturing tolerances, they may not align with specific physical locations relative to the overall object. For example, a glass window may exhibit inconsistent polarization patterns from one window to the next, depending on the cooling process of a particular piece of glass. However, this inconsistency in the polarization pattern can make it difficult to detect defects. For example, if a "reference" window is used to generate a template polarization pattern, then if the threshold is set too low, the difference between this template polarization pattern and the measured polarization pattern from another window may result in the detection of a defect, but if the threshold is set too high, the defect may go undetected. Some embodiments use adaptive thresholds and / or thresholds set based on physics-based priors. For example, if a surface is curved, areas with high curvature are more likely to have a stronger polarization signal. Therefore, in some embodiments, the threshold set for these areas is different from the threshold for areas that are estimated or expected to be flat. This adaptive thresholding can be quite large (eg, the threshold can differ by several orders of magnitude between different surfaces), since the polarization intensity can vary by two orders of magnitude between what appears to be a mostly flat versus curved surface.

[0144] Thus, some aspects of embodiments of the present disclosure relate to anomaly detection methods for detecting surface features in objects. For example, in some embodiments of the present disclosure, tensors in a representation space are extracted from a large set of known good reference samples. These reference tensors in the representation space can differ from each other according to natural variations (e.g., natural variations in their polarization patterns). Thus, one or more summary metrics can be computed for these reference tensors in the representation space to cluster the various reference tensors, such as computing the maximum and minimum values of DOLP, or the distribution of AOLP on different parts of the characterized surface, or the smoothness of transitions in different levels of DOLP. The statistical distribution of these summary metrics for the set of known good objects can then be stored as part of a storage model 810 for the characterized surface.

[0145] In these embodiments of the present disclosure, based on this approach, the stored model 810 includes an anomaly detection model as a statistical model of generally expected properties for a particular surface of an object, which is loaded based on the registration of the original polarization frame 18 (or the computational tensor 50 in the representation space), and similar summary metrics are calculated based on measurements performed on the computational tensor 50 from the inspected surface. If these summary metrics for the inspected surface are within the distribution of metrics from known good samples represented in the anomaly detection model, then this particular portion of the surface can be marked as clean or defect-free. On the other hand, if one or more of these measurements are outside the measurement distribution (e.g., greater than a threshold distance from the distribution of known good samples, such as greater than two standard deviations from the mean), then the surface can be marked as containing defects.

[0146] Surface property detection using trained convolutional neural networks

[0147] In some embodiments of the present disclosure, the stored model 810 includes trained convolutional neural networks (CNNs) trained to detect one or more defects in a surface of an object based on provided tensors in a representation space. These CNNs can be trained based on labeled training data (e.g., data in which training tensors in a representation space are used to train weights of connections of a neural network to compute outputs that label defect portions based on the labeled training data).

[0148] In some embodiments of the present disclosure, the model is implemented using one or more of the following: encoder-decoder neural networks or a U-net architecture for semantic segmentation of defects. U-net enables the propagation of multiscale information. In some embodiments of the present disclosure, a CNN architecture for semantic segmentation and / or instance segmentation is trained using polarized training data (e.g., training data comprising polarized raw frames as training inputs and segmentation masks as labeled training outputs).

[0149] One embodiment of the present disclosure using deep instance segmentation is based on a modification of the Mask Region Based Convolutional Neural Network (Mask R-CNN) architecture to form the Polarized Mask R-CNN architecture. Mask R-CNN works by taking an input image x, which is an H×W×3 tensor of image intensity values (e.g., height by width by color intensity in the red, green, and blue channels), and passing it through a backbone network: C=B(x). The backbone network B(x) is responsible for extracting useful learning features from the input image and can be any standard CNN architecture, such as AlexNet (see, e.g., Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. “ImageNet classification with deep convolutional neural networks.” Advances in neural information processing systems. 2012.), VGG (see, e.g., Simonyan, Karen, and Andrew Zisserman. “Very deep convolutional networks for large-scale image recognition.” arXiv preprint arXiv:1409.1556 (2014).), ResNet-101 (see, e.g., Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778, 2016.), MobileNet (see, e.g., Howard, Andrew G. et al., “Mobilenets: Efficient convolutional neural networks for mobile vision applications.” arXiv preprint arXiv:1704.04861 (2017).), MobileNetV2 (see, e.g., Sandler, Mark et al., “MobileNetV2: Inverted residuals and linear bottlenecks.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.) and MobileNetV3 (see, for example, Howard, Andrew et al. “Searching for MobileNetV3.” Proceedings of the IEEE International Conference on Computer Vision. 2019.).

[0150] The backbone network B(x) outputs a set of tensors, for example, C = {C1, C2, C3, C4, C5}, where each tensor c i Representing feature maps of different resolutions. These feature maps are then combined in a feature pyramid network (FPN) (see, for example, Tsung-Yi Lin, Piotr Doll'ar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie's Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2117–2125, 2017.), a region proposal network (RPN) (see, for example, Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems, pp. 91–99, 2015.) and finally passed through an output subnetwork (see, e.g., Ren et al. and He et al. above) to produce class, bounding box, and pixel-wise segmentation. These are combined with non-maximum suppression for instance segmentation.

[0151] In some embodiments, a Mask R-CNN architecture is used as a component of a Polarization Mask R-CNN architecture that is configured to take several input tensors, including tensors in a polarization representation space, and compute a multi-scale second tensor in a second representation space. In some embodiments, the tensors in different first representation spaces are referred to as being in different "modes," and the tensors for each mode can be provided to a separate Mask R-CNN backbone for each mode. Each of these backbones computes a mode tensor at multiple scales or resolutions (e.g., corresponding to a different scaled version of the input first tensor), and fuses the mode tensors computed at each scale of the different modes to generate a fused tensor for each scale. The fused tensor or second tensor can then be provided to a prediction module that is trained to compute predictions (e.g., recognition of surface features) based on the fused tensor or second tensor. The Polarization Mask R-CNN structure is described in more detail in U.S. Provisional Patent Application No. 63 / 001,445, filed with the U.S. Patent and Trademark Office on March 29, 2020, and International Patent Application No. PCT / US20 / 48604, filed with the U.S. Patent and Trademark Office on August 28, 2020, the entire disclosures of which are incorporated herein by reference.

[0152] Although some embodiments of the present disclosure relate to surface characterization using a polarized CNN architecture including a Mask R-CNN backbone, embodiments of the present disclosure are not limited thereto, and other backbones such as AlexNet, VGG, MobileNet, MobileNetV2, MobileNetV3, etc. can be modified in a similar manner to replace one or more (e.g., replace all) Mask R-CNN backbones.

[0153] Therefore, in some embodiments of the present disclosure, a surface characterization result 20 is calculated by providing a first tensor comprising tensors in a polarization feature representation space to a trained convolutional neural network (CNN) such as a Polarization Mask R-CNN architecture to calculate a segmentation map, wherein the segmentation map identifies locations or portions of an input image (e.g., an input polarization raw frame) corresponding to specific surface characteristics (e.g., surface defects such as cracks, dents, uneven paint, the presence of surface contaminants, or surface features such as surface smoothness versus roughness, surface flatness versus curvature, etc.).

[0154] Surface feature detection using classifiers

[0155] In some embodiments of the present disclosure, rather than using a convolutional neural network to identify areas of the inspected surface that contain various surface features of interest (e.g., containing defects), the model 810 includes a trained classifier that classifies a given input into one or more categories. For example, the trained classifier may compute a feature output 20 comprising a vector of length equal to the number of different possible surface features that the classifier is trained to detect, where each value in the vector corresponds to a confidence level that the input image depicts the corresponding surface feature.

[0156] The classifier can be trained to obtain an input image of a fixed size, wherein the input can be calculated by, for example, extracting a first tensor in a first representation space from the original polarization frame and providing the entire first tensor as input to the classifier or dividing the first tensor into fixed-size blocks. In various embodiments of the present disclosure, the classifier may include, for example, a support vector machine, a deep neural network (e.g., a deep fully connected neural network), etc.

[0157] Training data for training statistical models

[0158] Some aspects of embodiments of the present disclosure relate to preparing training data for training a statistical model for detecting surface features. In some cases, manually labeled (e.g., human-labeled) training data may be available, such as in the form of manually capturing polarization raw frames of an object's surface using a polarization camera and marking the areas in the image as containing surface features of interest (e.g., boundaries between different types of materials, locations of defects such as dents and cracks, or surface irregularities such as rough portions of a surface that is expected to be smooth). This manually labeled training data can be used as part of a training set for training a statistical model, such as an anomaly detector or convolutional neural network as described above.

[0159] While manually labeled training data is generally considered good training data, there may be situations where the manually labeled data may not be large enough to train a good statistical model. Therefore, some aspects of the embodiments of the present disclosure also relate to augmenting the training dataset, which may include synthesizing additional training data.

[0160] In some embodiments of the present disclosure, computer graphics techniques are used to synthesize training object data with and without surface characteristics of interest. For example, when training a detector to detect surface defects, a polarization raw frame of a defect-free surface can be combined with a polarization raw frame depicting defects such as cracks, debris, burrs, uneven paint, and the like. These independent images can be combined using computer graphics techniques (e.g., image editing tools to programmatically clone or synthesize polarization raw frame images of defects onto polarization raw frames of defect-free surfaces to simulate or synthesize polarization raw frames of surfaces containing defects), and the synthesized defects can be placed in physically reasonable locations on the clean surface (e.g., an image of a dent in a door panel is synthesized into an image of the portion of the door panel that would be dented, and not placed in a physically unrealistic area, such as a glass window. Similarly, fragments in a glass surface can be synthesized into the glass surface, but not onto an image of plastic decoration).

[0161] As another example, in some embodiments of the present disclosure, a generative adversarial network (GAN) is trained to generate synthetic data, where a generative network is trained to synthesize a polarization raw frame of a surface depicting a defect, and a judgment network is trained to determine whether its input is a real polarization raw frame or synthesized (e.g., by the generative network).

[0162] In some embodiments of the present disclosure, a technique called "domain randomization" is used to add "random" image-based perturbations to simulated or synthetic training data to make the synthetic training data more closely resemble real-world data. For example, in some embodiments of the present disclosure, rotation augmentation is applied to the training data to augment the training data with rotated versions of various features. This can be particularly beneficial for improving the accuracy of detection of defects with extreme aspect ratios (e.g., scratches) that are not well represented in natural images.

[0163] In various embodiments of the present disclosure, statistical models are trained using training data based on corresponding techniques. For example, in embodiments using anomaly detection methods, various statistics are calculated for a set of good data, such as the mean and variance of good data points, to determine a threshold distance (e.g., two standard deviations) for determining whether a given sample is acceptable or abnormal (e.g., defective). In embodiments using a neural network such as a convolutional neural network (e.g., Polarized Mask R-CNN), the training process may include updating the weights of the connections between neurons in each layer of the neural network according to a backpropagation algorithm, and iteratively adjusting the weights using gradient descent to minimize the error (or loss) between the output of the neural network and the labeled training data.

[0164] Thus, aspects of embodiments of the present disclosure provide systems and methods for automatically characterizing surfaces, such as for automatically inspecting manufactured parts as they roll off an assembly line. These automated processes enable cost savings for manufacturers, not only through automation in inspection and the resulting reduction in manual labor, but also through robust and accurate handling of anomalies in the products themselves (e.g., automatically removing defective products from the manufacturing stream).

[0165] While the invention has been described in conjunction with certain exemplary embodiments, it should be understood that the invention is not limited to the disclosed embodiments, but on the contrary, the invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims and their equivalents.

Claims

1. A computer-implemented method for detecting surface characteristics for surface modeling, the method comprising: receiving one or more polarization raw frames of a surface of a physical object, the polarization raw frames being captured at different polarizations by a polarization camera including a polarization filter or a polarizer, wherein each polarization raw frame corresponds to an image taken at a different polarization angle behind the polarization filter or the polarizer; extracting one or more first tensors in one or more polarization representation spaces from the polarization raw frame; as well as detecting surface properties of the surface of the physical object based on the one or more first tensors in the one or more polarization representation spaces, Wherein, detecting the surface characteristics includes: loading a stored model corresponding to the location of the surface of the physical object; and computing the surface property based on the stored model and the one or more first tensors in the one or more polarization representation spaces, wherein the storage model comprises one or more reference tensors in the one or more polarization representation spaces, and Wherein, calculating the surface characteristics comprises calculating a difference between the one or more reference tensors and the one or more first tensors in the one or more polarization representation spaces.

2. The computer-implemented method of claim 1 , wherein the one or more first tensors in the one or more polarization representation spaces comprise: The degree of linear polarization (DOLP) represents the DOLP image in space; as well as The Angle of Linear Polarization (AOLP) represents the AOLP image in space.

3. The computer-implemented method of claim 1 , wherein the one or more first tensors further comprise one or more non-polarized tensors in one or more non-polarized representation spaces, and in, The one or more unpolarized tensors include one or more intensity images in an intensity representation space.

4. The computer-implemented method of claim 3, wherein: The one or more intensity images include: a first color intensity image; a second color intensity image; and A third color intensity image.

5. The computer-implemented method of claim 1, 2, 3, or 4, wherein the surface characteristics comprise defects in the surface of the physical object.

6. The computer-implemented method of claim 1 , wherein: The difference is calculated using the Fresnel distance.

7. The computer-implemented method of claim 1 , wherein: The stored model includes a reference three-dimensional grid, and Wherein, calculating the surface characteristics includes: computing a three-dimensional point cloud of the surface of the physical object based on the one or more first tensors in the one or more polarization representation spaces; and The difference between the three-dimensional point cloud and the reference three-dimensional mesh is calculated.

8. The computer-implemented method of claim 1 , wherein: The stored model includes a trained statistical model configured to compute a prediction of the surface characteristic based on the one or more first tensors in the one or more polarization representation spaces.

9. The computer-implemented method of claim 8, wherein the trained statistical model comprises an anomaly detection model.

10. The computer-implemented method of claim 8, wherein the trained statistical model comprises a convolutional neural network trained to detect defects in the surface of the physical object.

11. The computer-implemented method of claim 8, wherein the trained statistical model comprises a trained classifier trained to detect defects.

12. A system for detecting surface characteristics for surface modeling, the system comprising: a polarization camera including a polarization filter, the polarization camera configured to capture polarization raw frames, wherein each polarization raw frame is an image captured at a different polarization angle; and A processing system comprising a processor and a memory storing instructions that, when executed by the processor, cause the processor to: receiving one or more polarization raw frames of a surface of a physical object, the polarization raw frames corresponding to different polarizations of light; Extracting one or more first tensors in one or more polarization representation spaces from the polarization raw frame; and detecting surface properties of the surface of the physical object based on the one or more first tensors in the one or more polarization representation spaces, The memory further stores instructions that, when executed by the processor, cause the processor to detect the surface characteristics by: loading a stored model corresponding to the location of the surface of the physical object; and computing the surface property based on the stored model and the one or more first tensors in the one or more polarization representation spaces, wherein the storage model comprises one or more reference tensors in the one or more polarization representation spaces, and The memory further stores instructions that, when executed by the processor, cause the processor to calculate the surface characteristic by calculating a difference between the one or more reference tensors and the one or more first tensors in the one or more polarization representation spaces.

13. The system of claim 12, wherein the one or more first tensors in the one or more polarization representation spaces comprise: The degree of linear polarization (DOLP) represents the DOLP image in space; as well as The Angle of Linear Polarization (AOLP) represents the AOLP image in space.

14. The system of claim 12, wherein the one or more first tensors further comprise one or more non-polarized tensors in one or more non-polarized representation spaces, and in, The one or more unpolarized tensors include one or more intensity images in an intensity representation space.

15. The system according to claim 14, wherein: The one or more intensity images include: a first color intensity image; a second color intensity image; and A third color intensity image.

16. The system of claim 12, 13, 14 or 15, wherein the surface characteristics comprise defects in the surface of the physical object.

17. The system of claim 12, wherein: The difference is calculated using the Fresnel distance.

18. The system of claim 12, wherein: The stored model includes a reference three-dimensional grid, and The memory further stores instructions, which, when executed by the processor, cause the processor to calculate the surface characteristics through the following steps: computing a three-dimensional point cloud of the surface of the physical object based on the one or more first tensors in the one or more polarization representation spaces; and The difference between the three-dimensional point cloud and the reference three-dimensional mesh is calculated.

19. The system of claim 12, wherein: The stored model includes a trained statistical model configured to compute a prediction of the surface characteristic based on the one or more first tensors in the one or more polarization representation spaces.

20. The system of claim 19, wherein the trained statistical model comprises an anomaly detection model.

21. The system of claim 19, wherein the trained statistical model comprises a convolutional neural network trained to detect defects in the surface of the physical object.

22. The system of claim 19, wherein the trained statistical model comprises a trained classifier trained to detect defects.

Images