Multi-channel LED (light-emitting diode) lighting equipment for realizing three-dimensional reconstruction by using photometric stereo technology

Abstract

The invention discloses multi-channel LED lighting equipment for realizing three-dimensional reconstruction by starting a photometric stereo technology. In one example, an illumination system is provided that improves the reliability of directly detecting component defects by capturing multiple images of a component under different illumination conditions and reconstructing a three-dimensional (3D) model of the component using photometric stereo techniques. The 3D model can be used to perform other operations to further improve the reliability of defect detection, such as measuring the height of irregular changes on the surface of a component.

Description

Technical Field

[0001] The present application relates generally to inspection cameras and more particularly to using a multi-channel LED light source to enable photometric stereo techniques for three-dimensional reconstruction. Background Art

[0002] Inspection cameras are used in industrial products during the manufacturing quality control process to help detect defects in manufactured products. For example, if a manufacturer is producing metal castings, one or more inspection cameras may be placed on the manufacturing and / or assembly line to inspect the metal castings or parts thereof being produced to detect any quality control issues. However, when capturing images, a particular light source may not be suitable for imaging products with specific surfaces, specific defects, and / or specific environments. For example, product surface materials or characteristics affect the quality of light used to capture images, including the reflective, transparent, or black / opacity properties of the product. Another example is that certain types of defects may be difficult to detect, such as scratches or dirt. Another example is that product defects are more difficult to detect in certain environments. Summary of the invention

[0003] In one aspect, the present application provides a system, comprising:

[0004] a lighting device comprising a plurality of independently controllable light sources and at least one camera;

[0005] A computer system comprising at least one hardware processor and a non-transitory computer-readable medium storing instructions, which, when executed by the at least one hardware processor, causes the at least one hardware processor to perform the following operations:

[0006] Accessing a sequence of one or more capture configurations, each capture configuration defining a plurality of image captures by at least one camera, each image capture having an exposure time, and indicating which individual light sources of the independently controllable light sources are illuminated during the plurality of image captures;

[0007] receiving a plurality of component images captured while executing one of the one or more capture configurations of the sequence;

[0008] performing a photometric stereo operation on the plurality of component images to create a three-dimensional reconstruction of the component;

[0009] Measuring depth variations of component surface features in 3D reconstructions; and

[0010] Whether the component surface feature is a defect in the component is determined based on the depth variation of the component surface feature.

[0011] In another aspect, the present application provides a method, comprising:

[0012] Accessing a sequence of one or more capture configurations, each capture configuration defining a plurality of image captures by at least one camera, each image capture having an exposure time, and indicating which individual light sources of the independently controllable light sources are illuminated during the plurality of image captures;

[0013] receiving a plurality of component images captured while executing one of the one or more capture configurations of the sequence;

[0014] performing a photometric stereo operation on the plurality of component images to create a three-dimensional reconstruction of the component;

[0015] Measuring depth variations of component surface features in 3D reconstructions; and

[0016] Whether the component surface feature is a defect in the component is determined based on the depth variation of the component surface feature.

[0017] In a third aspect, the present application provides a non-transitory machine-readable storage medium, which contains instructions executable by one or more machines, and the instructions perform the following operations:

[0018] Accessing a sequence of one or more capture configurations, each capture configuration defining a plurality of image captures by at least one camera, each image capture having an exposure time, and indicating which individual light sources of the independently controllable light sources are illuminated during the plurality of image captures;

[0019] receiving a plurality of component images captured while executing one of the one or more capture configurations of the sequence;

[0020] performing a photometric stereo operation on the plurality of component images to create a three-dimensional reconstruction of the component;

[0021] Measuring depth variations of component surface features in 3D reconstructions; and

[0022] Whether the component surface feature is a defect in the component is determined based on the depth variation of the component surface feature.

[0023] From the technical solution provided by the present application, it can be seen that by capturing multiple images of a component under different lighting conditions and reconstructing a three-dimensional (3D) model of the component using photometric stereo technology, the reliability of directly detecting component defects can be improved. The 3D model can be used to perform other operations to further improve the reliability of defect detection, such as measuring the height of irregular changes on the surface of the component. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a schematic block diagram of an inspection camera system according to an exemplary embodiment.

[0025] Figure 2 A-2H is a schematic diagram illustrating a photometric stereo technique using a mask according to an exemplary embodiment.

[0026] Figure 3 and Figure 4 is a screenshot of the user interface.

[0027] Figure 5 is a schematic diagram of a deep convolutional neural network (DCNN) according to an exemplary embodiment.

[0028] Figure 6 is a schematic diagram of a method for performing three-dimensional reconstruction according to an exemplary embodiment.

[0029] Figure 7 is a schematic diagram of a mobile device according to an exemplary embodiment.

[0030] Figure 8 is a schematic block diagram of a machine illustrating an example form of a computer system in which instructions may be executed to cause the machine to perform one of the methodologies described herein. DETAILED DESCRIPTION

[0031] By improving the design of lighting equipment, inspection cameras can be improved to enhance the light in various scenarios. More specifically, instead of using a single light source - which cannot provide enough light to capture high-quality images to determine surface defects on all surface materials of various components or products - a lighting device with multiple light sources can be used. In addition, a controller can be provided for independently controlling multiple light sources, so that lighting combinations and sequences can be used to maximize the flexibility of the lighting device to provide sufficient light for different products, components, materials and environments.

[0032] In addition to the light source system directly improving the reliability of defect detection on a component (e.g., by capturing an image of the component under ideal lighting conditions), in an exemplary embodiment, the light source system is also used to capture multiple images of the component under different lighting conditions and use photometric stereo technology to reconstruct a three-dimensional (3D) model of the component, which can be used to perform additional operations to improve the reliability of defect detection, such as measuring the height of changes on the surface of the component.

[0033] In an exemplary embodiment, a controller and computer system is provided for a lighting device for a three-dimensional reconstruction assembly. More specifically, a user can use the computer system to define a sequence of one or more capture configurations. Each capture configuration includes an identification of one or more light sources to be activated during image capture, one or more channels for capturing images, and an exposure time. Each sequence can also define a custom delay between capture configurations.

[0034] Upon receiving a trigger signal from hardware or software, the controller fires the stored sequence, triggering the camera to take images according to the capture configuration (and delay).

[0035] The following description includes illustrative systems, methods, techniques, instruction sequences, and computer program products that are useful for understanding various inventive subject matters. However, it will be apparent to one skilled in the art that the embodiments of the inventive subject matter may be practiced without these specific details. Often, well-known instruction instances, protocols, structures, and techniques are not described in detail.

[0036] Figure 1 1 is a schematic block diagram of an inspection camera system 100 according to an exemplary embodiment. The inspection camera system 100 may include a lighting device 102, a controller 104, and a computing system 106. The lighting device 102 includes a plurality of different lights, such as light emitting diode (LED) lights, which are independently controlled by the controller 104. This means that these lights can be turned on and off independently of each other, so that the lighting device 102 only turns on some but not all lights at any given time. In some exemplary embodiments, the brightness of each light can also be independently controlled, so that the light can have a customized brightness / darkness level in addition to being on or off.

[0037] Furthermore, for photometric stereo, it may be more advantageous to use a single (different) light source when capturing multiple images of a component. Thus, instead of turning on multiple LEDs in each image, only one LED may be lit in each image, at least for images that are intended to be used for photometric stereo for 3D reconstruction (different lighting scenarios may be used for images for direct defect detection).

[0038] The lighting device 102 may also include one or more cameras. Each camera may also be independently controlled to take a picture / capture an image when a signal is received. Variables may also be independently controlled. These variables may include any software definable parameters of the imaging system, including but not limited to: the aperture and / or focal length of the lens, the exposure time of the camera, the phase and / or pattern of the diffuser, the wavelength and / or intensity of the light, etc.

[0039] In an exemplary embodiment, the lighting device 102 is a light shield. The light shield is used to illuminate a target object, such as a metal casting or other product. The light shield includes a housing that accommodates a plurality of light sources, and the specific details will be described in detail below. In some examples, these light sources include a plurality of LEDs or display screens, which are arranged in a flexible manner to illuminate the target object.

[0040] One or more cameras can be mounted on the light mask via a bracket and capture images of the illuminated target object through a hole in the top of the light mask.

[0041] The controller 104 is an electronic component designed to send signals to the lighting device 102 through one or more channels to control the lights on the lighting device 102.

[0042] The computing system 106 includes a variety of different software components running on a computing hardware platform. These components include a sequence generation user interface 108. The sequence generation user interface 108 allows a user to create a sequence as an ordered combination of capture configurations, which can be optionally separated by custom delays. The created sequence can be stored in the controller 104, such as in a memory component of the controller 104. Upon receiving an external trigger, the controller 104 retrieves the sequence and executes the sequence, which means that the capture configurations in the sequence are used to control the lighting device 102 according to the parameters defined in each capture configuration and separated by custom delays.

[0043] The external trigger may be a hardware trigger (e.g., from a programmable logic controller) or a software trigger (e.g., from an industrial personal computer). In some example embodiments, one or more trigger signals may come from the factory computer 109. Once the trigger occurs, the sequence begins to execute, controlling the lighting device 102 to light up the corresponding lights at the appropriate time and triggering the camera to take pictures / capture images at the appropriate time.

[0044] The controller 104 sends the trigger information to the image processing component 110 on the computing system 106. The image processing component 110 also records the time of receipt of the trigger information using the system timestamp when the trigger information is received. The controller 104 also receives a photo from the lighting device 102, adds a timestamp to it, and then sends the image with the timestamp to the image processing component 110. The image processing component 110 then encodes the photo, capture configuration information, timestamp, camera identification, and other information into data packets, which are stored in the first shared memory 112 in the form of a data structure.

[0045] The image analysis component 114 then retrieves this data structure and decodes it. It can perform one or more analysis tasks on the image. In the image used for 3D reconstruction, multiple images with different single light sources will be processed using the photometric stereo technique described later.

[0046] The image analysis component 114 then encodes the results of these image analysis tasks into another data structure. This data structure may include, for example, a 3D reconstruction of the component. The data structure is then stored in the second shared memory 116.

[0047] The central processing component 118 then retrieves the data structure from the second shared memory 116 and performs timestamp sorting on the data in the data structure using information from the programmable logic controller, such as component identification, inspection identification, inspection ready, component start, and component end. Subsequently, the central processing component 118 packages the sorted data into a data packet and stores it in the third shared memory 120. The data packet may include, for example, component identification, inspection identification, camera identification, image, inference mask, other inference post-processing results, and error codes, etc.

[0048] A user interface component 122 is then provided that can access the data packets in the third shared memory 120 and display a portion of them to the user through a graphical user interface. Here, the user can specify the inspection mode (such as manual or automatic) and can also add customer-specific settings, such as image display settings, how and whether to upload images to a cloud environment, etc.

[0049] It should be noted that in some exemplary embodiments, the user interface component 122 and the user interface 108 can be combined into a single component. In other words, the sequence definition functionality of the user interface 108 can be combined with the output and setting related functionality of the user interface component 122.

[0050] Photometric stereo assumes that the surface of an object follows Lambertian reflectance, meaning that the amount of light reflected from a surface is determined by its orientation relative to the light source and the camera. The camera remains in a fixed position while the light source changes. For each pixel in the image, the intensity variation in the different images is analyzed. Due to the change in the lighting angle, the intensity varies based on the surface orientation of the object at that pixel. Pixels that appear brighter in some images and darker in others indicate changes in surface orientation and curvature.

[0051] By analyzing intensity changes, photometric stereo estimates the surface normal at each pixel of the object. A surface normal is a vector pointing outward from the surface, indicating the orientation of the surface. The relationship between surface normals, light direction, and camera detection is described by Lambert's law, which governs how light reflects from a Lambertian surface. In an exemplary embodiment, the normal is calculated using the L2 method, where the L2 norm is calculated by squaring the vector value and then taking the square root, but other techniques may also be used.

[0052] The estimated surface normals are integrated to reconstruct a three-dimensional representation of the object's shape. The integration process involves accumulating the normals to estimate the surface depth. The result of the integration is a height map, which represents the height of the geometry of the object's surface relative to some horizontal plane. Therefore, each surface can be assigned a height map, which is essentially a two-dimensional representation of the height at each point along the surface. In an exemplary embodiment, the height is calculated using a Poisson distribution.

[0053] This height map can then be filtered using the uniform filtering technique described previously to reduce the global error.

[0054] Figure 2 A to Figure 2 H is a schematic diagram illustrating a photometric stereo technique using a photomask 200 according to an exemplary embodiment. Figure 2 A to Figure 2 Each of the figures in H shows the state of the mask 200 at different time points when different light sources in the mask 200 are illuminated. Component 202 is placed in the center of the mask 200. Here, component 202 is a wedge-shaped component. The camera 204 on the top of the mask 200 captures images of the component 202 when illuminated by different light sources. For example, Figure 2 In A, light source 206A is illuminated, while other light sources are not illuminated. Figure 2 In B, light source 206B is illuminated, while other light sources are not illuminated.

[0055] Although in Figure 2 A to Figure 2 H, but component 202 may have one or more defects that need to be detected. Photometric stereo techniques can be used to reconstruct component 202 in 3D based on multiple images of component 202 taken successively under different light sources. This allows other characteristics of component 202 to be measured, such as changes in the height of surface portions, which may or may not indicate a defect. For example, not only can a bump be detected, but the relative height of the bump relative to other portions of the surface can also be measured. Similar techniques can also be used to measure the relative height of dents on the surface of component 202.

[0056] As previously described, user interface 108 provides a mechanism by which a user may specify a capture configuration, including a lighting combination. Figure 3 and Figure 4 is a screenshot showing this mechanism. In particular, first see Figure 3 , user interface 300 displays a screen where a user can name / select a sequence name 302 and name / create various capture configurations 304A-304H. Although not shown here, the user can also specify a custom delay time for one or more of the capture configurations 304A-304H. Each sequence contains multiple capture configurations that are processed sequentially. Therefore, in this example, capture configuration 304A is triggered before capture configuration 304B, and so on.

[0057] See also Figure 4 , user interface 300 also displays a screen where a single capture configuration can be modified. Here, a capture configuration can be selected via drop-down menu 402 and its name is displayed in field 404. The exposure time can be specified via slider 406. Drop-down menu 408 is used to select an editing mode. When annular is selected as the editing mode, clicking on the pattern will toggle the entire ring on and off. When a different editing mode is selected, such as a single light source, only the single light source will be toggled on and off when clicked. Here, the user has selected the "annular" pattern, and the annular pattern is displayed in lighting diagram 410, with LEDs 412A-412H shown as "lit" and other LEDs, such as LED 412I, shown as "unlit."

[0058] The user may also select different individual LEDs or combinations of LEDs to be "lit" in the capture configuration. When in a ring pattern, the system may default to selecting all LEDs in a ring when any LED in that ring is selected. Here, there may be 5 rings, and if the user selects any LED in a ring, then all LEDs in that ring will be selected and lit. For example, here the user may have selected LED 412B as lit, which would select the entire ring, resulting in all LEDs 412A-412H in the ring being selected and lit. The user may choose to light up any combination or number of rings at the same time. However, in the case of photometric stereo, the user may only specify one LED in a single ring and specify a different individual LED for each of multiple different images.

[0059] In an exemplary embodiment, when selecting lights to illuminate for a photometric stereoscopic captured image, lights are selected that maximize even distribution of light and minimize overexposed and underexposed areas.

[0060] In one exemplary embodiment, a human determines which lights are illuminated in the photometric stereo technique. In other exemplary embodiments, partially or fully automated techniques may be used. The more prior knowledge there is about the component being imaged, the more likely it is that an automated technique can be used in the photometric stereo technique. For example, if the component is a part with known dimensions, including measurements of each surface of a theoretically perfect version, then this can be used as a baseline for the automated technique to identify the "best" combination of lighting angles to use in the photometric stereo technique. Continuing with this example, images of a "test" component (preferably a component unit that has been thoroughly inspected and found to be free of defects) can be captured using many or even all available light sources alone. For example, if there are 32 different LEDs on the mask, the training operation may involve capturing 32 different images of the test component, with a different LED illuminated in each image. Different groups of these images can then be tested by performing a 3D reconstruction using the photometric stereo technique on each group, and using these 3D reconstructions to measure the heights of some known protrusions on the component. The group of images that produces a reconstruction with height measurements closest to the known dimensions is selected as the combination to be used in future 3D reconstructions of different units of the component. Thus, for example, if a test component is known to have a protrusion on one of its surfaces with a maximum height of 30 mm, it may be found that a reconstruction generated using a combination of light sources #1, #9, #17, and #30, rather than a combination of light sources #1, #9, #19, and #31, measures the height of the protrusion closest to 30 mm. In this case, for units of the same component, the system may automatically determine that images should be captured using light sources #1, #9, #17, and #30 (independently). If a test component is known to have a protrusion on one of its surfaces with a maximum height of 30 mm, it may be found that a reconstruction generated using a combination of light sources #1, #9, #17, and #30, rather than a combination of light sources #1, #9, #19, and #31, measures the height of the protrusion closest to 30 mm.

[0061] Alternatively, reconstructions could be performed for different light combinations to try to identify which lights have no effect on the height measurement capability.

[0062] In addition, although not shown in the figure, in some exemplary embodiments, the user interface 300 can add a real-time camera view from the corresponding camera. If the user places a component into the lighting device 102, the user can see in real time how the component is illuminated according to the changes they made to the lighting combination in the user interface 300.

[0063] During use or 3D reconstruction, depth measurements may be made by a neural network. The neural network may be trained using images from an image data source, and these images may be enhanced by adding one or more labels to the training images. This may involve converting the images to another format in order to accept these labels. The labels added depend on the training objective of the neural network, and in particular may correspond to the classification that the neural network is expected to perform. For example, if an image shows an instance of a defect on a particular component, it may be labeled as a defect so that the neural network can learn what a defect looks like for that particular component. This is an example of a positive label. Additionally, if an image shows an instance of a component without a defect, it may also be labeled as non-defective so that the neural network can learn what a component without a defect looks like. This is an example of a negative label. While these examples are binary (e.g., positive or negative), in practice, labels can have any number of values ​​depending on the classification task that the neural network is performing.

[0064] In an exemplary embodiment, the neural network is a deep convolutional neural network (DCNN). A DCNN is a machine learning model that is able to efficiently infer nonlinear relationships between homogeneous input domains and desired outputs, which can be either class classifications or scalars. A DCNN is a model that maps inputs to outputs through a series of so-called convolutional layers, which are composed of artificial neurons. A DCNN can be trained by providing it with a large amount of sample data (e.g., more than 10,000) and labels, with the goal of minimizing the difference (or "loss") between the model output and the desired output.

[0065] DCNN is not only designed to learn to classify images or groups of images, but also learn feature hierarchies by defining multiple layers. The inference process consists of applying a series of mathematical functions called layers to a given input and computing these functions on the input data. Each layer extracts features from the output of the previous layer, and all layers are trained jointly. Because of this layer-based architecture, it is called a "deep" convolutional neural network.

[0066] In an exemplary embodiment, five different types of layers are used. The first four layers are convolutional layers, nonlinear layers, pooling layers, and classification layers (although classification is just a special case of convolution followed by "softmax"). These first four layers can be thought of as a stage, and in practice a DCNN may be designed to have any number of these stages. When all stages are completed, a loss layer is used. Figure 5 is a schematic diagram illustrating a DCNN 500 consistent with an example embodiment. Here, two stages 502A and 502B are shown.

[0067] Convolutional layers 504A and 504B are the core of DCNN 500. Their parameters include a set of learnable filters that have small receptive fields but extend across the entire depth of the input data. During the forward pass in convolutional layers 504A and 504B, each filter is convolved on the features, the dot product between the filter entries and the input is calculated, and a two-dimensional activation map of the filter is generated. As a result, DCNN 500 learns filters that activate when a certain type of feature is detected.

[0068] The feature maps of all filters can be stacked together along the depth dimension to form the complete three-dimensional output of convolutional layers 504A and 504B.

[0069] Convolutional layers 504A and 504B apply a mathematical operation called a convolution. For two spatial dimensions and any number of non-spatial dimensions (called "channels"), a convolution is defined using the * operator as follows:

[0070]

[0071] Convolutional layers 504A and 504B typically have some very small support, such as N=1 and M=1, so when |n|>1 or |m|>1, g[n,m,d]=0.

[0072] It should be noted that the filters used in convolutional layers 504A and 504B may be activated in the first iteration of DCNN 500 and optimized before each additional iteration based on the actions taken by other layers in the previous iteration until a certain error term is minimized below a certain threshold. In an example embodiment, this can be achieved through back-propagation, which will be described in detail below.

[0073] The output of convolutional layers 504A and 504B is a set of arrays called feature maps 506A-506C. Each feature map 506A-506C may be generated by a different filter and modified based on various functions in each stage. At the output, each feature map 506A-506C represents a specific and conditioned feature extracted at all locations on the input. Figure 5 The example is a two-stage system, although one of ordinary skill in the art will recognize that more or fewer stages may be used while still consistent with the present disclosure, and in fact, as will be seen in the exemplary embodiments, the number of stages may be dynamically determined at runtime to optimize results.

[0074] Nonlinear layers 508A and 508B make DCNN 500 more expressive in revealing nonlinear relationships between inputs and outputs. Many different nonlinearities can be used in nonlinear layers, including sigmoid, tanh, and rectified linear functions. For brevity, an example of a nonlinearity will be described here: the rectified linear function. This function is defined as follows:

[0075]

[0076] Pooling layers 510A, 510B are applied to reduce the spatial dimension of the input image while retaining some information from the input image. In other words, pooling layers 510A, 510B do not actually participate in any learning, that is, they are fixed, predefined operations that do not change as training progresses. Instead, they are used to deal with the spatial dimension of the problem. In an exemplary embodiment, a decimation method can be used to retain one sample every N samples in the spatial dimension. In another exemplary embodiment, some local statistics can be used for pooling, such as max pooling, which is defined as:

[0077]

[0078] Among them, N=M=2.

[0079] When all stages 502A, 502B are completed, the image is classified using a classification layer 512 based on the output of the final pooling layer 510B. As described above, the classification layer 512 is actually a special convolutional layer containing a filter that is used to generate a score based on the volume output of the final pooling layer 510B. The filter applies a classification function, and its weights can be optimized in the same way as the filter weights of ordinary convolutional layers 504A, 504B.

[0080] Backpropagation involves computing the gradient of a loss function (defined later) in a loss layer 514 with respect to a number of weights in the DCNN 500. The gradient is then passed to a method that updates the weights for the next iteration of training the DCNN 500, attempting to minimize the loss function, which uses different sample data (unless it needs to be repeated, such as running out of sample data). Backpropagation uses a batch of labeled sample data to calculate the gradient of the loss function for the batch of samples through stages 502A, 502B (although, as described later, the loss function can be dynamically modified to exclude certain samples from consideration).

[0081] Back propagation may include two aspects: propagation and weight updates. In the propagation aspect, a forward propagation of the training mode input images is performed to generate propagated output activations through the DCNN 500 (i.e., the images pass through stages 502A, 502B). Then, according to the objectives specified by the training mode, a backward propagation of the propagated output activations is performed through the DCNN 500 to generate increments for all outputs.

[0082] In terms of weight updates, for each weight of each filter, the output delta and input activation are multiplied to get the gradient of the weight, and then a ratio of this gradient is subtracted from the weight. This ratio affects the speed and quality of learning. The higher the ratio, the faster the training, but this comes at the cost of reduced accuracy.

[0083] Thus, these two aspects, including the forward pass and the backward pass through stages 502A, 502B, are performed repeatedly until the error rate is below a certain threshold. An example of a back-propagation algorithm compatible with DCNN 500 includes a gradient descent method.

[0084] The use of back propagation may depend on whether the combined error of image classification in the labeled sample data batch exceeds a preset error threshold. If the combined error is too large, back propagation should be performed to update and hopefully minimize the error for the next iteration, and use subsequent batches of labeled sample data in the next iteration until the combined error does not exceed the threshold.

[0085] As mentioned above, classification can score data. The DCNN 500 outputs a vector that can be compared to the expected output of some loss function, such as the sum of squared error function:

[0086]

[0087] Figure 6 6 is a flow chart according to an exemplary embodiment, illustrating a method 600 for performing 3D reconstruction. In operation 602, a sequence is accessed. The sequence includes one or more capture configurations, each capture configuration defining a plurality of image captures, each image capture having an exposure time and indicating which single light source of independently controllable light sources is illuminated during the corresponding image capture process. In operation 604, the sequence is written to a memory accessible to a controller. In operation 606, a plurality of component images captured during the execution of the one or more capture configurations of the sequence are received from the controller. Each image is captured by at least one camera of the lighting device when the lighting device illuminates a corresponding single light source.

[0088] In operation 608, a photometric stereo operation is performed on the plurality of images to create a three-dimensional (3D) reconstruction of the component. In operation 610, a depth variation of a surface feature of the component in the 3D reconstruction is measured. In operation 612, the depth variation of the feature is used to determine whether the feature is a defect in the component.

[0089] Figure 7 700 is a block diagram showing a schematic diagram of a software architecture 702 that can be installed on any one or more of the above-mentioned devices. Figure 7 This is only one non-limiting example of a software architecture, and it is understood that many other architectures may be implemented to facilitate the functionality described herein. In various embodiments, the software architecture 702 is implemented by hardware, such as Figure 8 800 includes a processor 810, a memory 830, and an input / output (I / O) component 850. In this example architecture, the software architecture 702 can be conceptualized as a stack of layers, where each layer may provide specific functionality. For example, the software architecture 702 includes an operating system 704, a library 706, a framework 708, and an application 710. In operation, the application 710 calls an application programming interface (API) call 712 through the software stack and receives a message 714 in response to the API call 712, consistent with certain embodiments.

[0090] In various embodiments, the operating system 704 manages hardware resources and provides common services. The operating system 704 includes, for example, a kernel 720, services 722, and drivers 724. The kernel 720 acts as an abstraction layer between the hardware and other software layers, consistent with certain embodiments. For example, the kernel 720 provides functions such as memory management, processor management (such as scheduling), component management, network and security settings. Services 722 can provide other common services to other software layers. Drivers 724 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 724 can include display drivers, camera drivers, or Low-power drivers, Flash drivers, serial communication drivers (such as Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0091] In some embodiments, the library 706 provides a low-level common infrastructure used by the application 710. The library 706 may include a system library 730 (such as a C standard library) that provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 706 may also include an API library 732, such as a media library (such as a library for supporting the display and operation of multiple media formats, such as Moving Picture Experts Group Part 4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer 3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) Audio Codec, Joint Photographic Experts Group (JPEG or JPG) or Portable Network Graphics (PNG)), a graphics library (such as an OpenGL framework for rendering in a two-dimensional (2D) and three-dimensional (3D) graphics context on a display), a database library (such as SQLite for providing various relational database functions), a network library (such as WebKit for providing web browsing functions), etc. The library 706 may also include a wide range of other libraries 734 that provide many other APIs to the application 710.

[0092] The framework 708 provides a high-level, general-purpose infrastructure that can be utilized by applications 710. For example, the framework 708 provides various graphical user interface functions, advanced resource management, advanced location services, etc. The framework 708 can provide a wide range of other APIs that applications 710 can utilize, some of which may be specific to a particular operating system 704 or platform.

[0093] In one example embodiment, the applications 710 include a home application 750, a contacts application 752, a browser application 754, a book reader application 756, a location application 758, a media application 760, a messaging application 762, a game application 764, and various other applications, such as a third-party application 766. An application 710 is a program that performs the functions defined in the program. One or more applications 710 can be created using a variety of programming languages ​​and structured in various ways, such as an object-oriented programming language (such as Objective-C, Java, or C++) or a procedural programming language (such as C or assembly language). In one specific example, a third-party application 766 (e.g., via ANDROID TM or IOS TM Software Development Kits (SDKs) are applications developed by entities that are not platform specific vendors and can be used on mobile operating systems such as iOS. TM ANDROID TM , Mobile software running on the phone or other mobile operating system. In this example, the third-party application 766 can call the API call 712 provided by the operating system 704 to implement the functions described herein.

[0094] Figure 8 A schematic diagram of a machine 800 is shown, which is in the form of a computer system that executes a set of instructions to cause the machine 800 to perform any of the methods discussed herein. Specifically, Figure 8 A schematic diagram of a machine 800 is shown, which is an example form of a computer system, in which instructions 816 (e.g., software, a program, an application, an applet, an application, or other executable code) cause the machine 800 to perform any of the methods discussed herein. For example, the instructions 816 may cause the machine 800 to perform Figure 6 Additionally or alternatively, instruction 816 may implement Figure 1-6 etc. The instructions 816 transform a general, unprogrammed machine 800 into a specific machine 800 that performs the functions described and illustrated. In other embodiments, the machine 800 may operate as a standalone device or may be coupled to other machines (e.g., networked). In a networked deployment, the machine 800 may operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer network (or distributed) environment. The machine 800 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a mobile phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a network device, a network router, a network switch, a network bridge, or any machine 800 capable of executing the instructions 816, whether sequentially or otherwise, which specify the operational behavior of the machine 800. Further, while a single machine 800 is illustrated, the term "machine" also includes multiple machines 800 that individually or jointly execute the instructions 816 to perform any of the methodologies discussed herein.

[0095] The machine 800 may include a processor 810, a memory 830, and an I / O component 850, which may be configured to communicate with each other, such as via a bus 802. In an example embodiment, the processor 810 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), other processors, or any suitable combination) may include, for example, a processor 812 and a processor 814, which may execute instructions 816. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions 816 simultaneously. Although Figure 8 Multiple processors 810 are shown, and the machine 800 may include a single processor 812 with a single core, multiple processors 812 or 814 with a single core, a single processor 812 with multiple cores, multiple processors 812 or 814 with multiple cores, or any combination thereof.

[0096] The memory 830 may include a main memory 832, a static memory 834, and a storage unit 836, each of which may be accessed by the processor 810 via the bus 802. The main memory 832, the static memory 834, and the storage unit 836 store instructions 816 that embody any of the methods or functions described herein. During execution by the machine 800, the instructions 816 may also reside in whole or in part in the main memory 832, the static memory 834, the storage unit 836, at least one processor 810 (e.g., a cache memory of a processor), or any suitable combination thereof.

[0097] The I / O components 850 may include various components to receive input, provide output, generate output, transmit information, exchange information, capture measurements, etc. The I / O components 850 included in a particular machine depend on the type of machine. For example, a portable machine (such as a cell phone) may include a touch input device or other similar input mechanism, while a headless server machine may not include such a touch input device. It should be understood that the I / O components 850 may include Figure 8Many other components not shown in the figure. The I / O components 850 are grouped according to their functions only to simplify the following discussion, and the grouping method is not limiting. In various example embodiments, the I / O components 850 may include output components 852 and input components 854. The output components 852 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The input components 854 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric inputs, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing devices), tactile input components (e.g., physical buttons, touch screens or other tactile input components that provide touch locations and / or touch or gesture strength), audio input components (e.g., microphones), etc.

[0098] In further example embodiments, the I / O component 850 may include a biometric component 856, a motion component 858, an environmental component 860, or a position component 862, among a number of other components. For example, the biometric component 856 may include a component for detecting expressions (e.g., gestures, facial expressions, vocal expressions, body gestures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying individuals (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 858 may include an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environment component 860 may include, for example, a light sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects concentrations of hazardous gases to ensure safety or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. The position component 862 may include a position sensor component (e.g., a global positioning system (GPS) receiver component), an altitude sensor component (e.g., an altimeter or barometer that derives altitude by detecting air pressure), a direction sensor component (e.g., a magnetometer), etc.

[0099] Communication can be achieved through various technologies. I / O components 850 may include a communication component 864 for connecting machine 800 to network 880 via coupling 882 or to device 870 via coupling 872. For example, communication component 864 may include a network interface component or other suitable device for interfacing with network 880. In further examples, communication component 864 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, Components (e.g. Low power consumption), Components and other components that provide communication through other means. Device 870 can be another machine or various peripheral devices (for example, connected via USB).

[0100] In addition, the communication component 864 can detect the identifier or include a component capable of detecting the identifier. For example, the communication component 864 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reading component (e.g., an optical sensor for detecting one-dimensional bar codes such as universal product codes (UPC) bar codes, multi-dimensional bar codes such as QR codes, Aztec codes, data matrix, Dataglyph, Maxi codes, PDF417, Supercode, UCC RSS-2D bar codes, and other optical codes) or an acoustic detection component (e.g., a microphone for identifying a tag audio signal). In addition, a variety of information can be derived through the communication component 864, such as obtaining a location through Internet Protocol (IP) geolocation, ... Signal triangulation obtains location, and obtains information that may indicate a specific location by detecting NFC beacon signals.

[0101] Various memories (i.e., 830, 832, 834 and / or the memory of the processor 810) and / or storage unit 836 may store one or more instructions 816 and data structures (e.g., software) that embody or use any of the methods or functions described herein. When these instructions (e.g., instructions 816) are executed by the processor 810, various operations are initiated to implement the disclosed embodiments.

[0102] As used herein, the terms "machine storage medium," "device storage medium," and "computer storage medium" have the same meaning and are used interchangeably. These terms refer to a single or multiple storage devices and / or media (e.g., centralized or distributed databases and / or associated caches and servers) that store executable instructions and / or data. Thus, these terms shall include, but are not limited to, solid-state memory, optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage medium, computer storage medium, and / or device storage medium include nonvolatile memory, such as semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), field programmable gate arrays (FPGA), and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM optical disks. The terms "machine storage medium," "computer storage medium," and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, at least in part of which fall under the term "signal media," which is discussed below.

[0103] In various example embodiments, one or more portions of network 880 may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless local area network (WLAN), a wide area network (WAN), a wireless wide area network (WWAN), a metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of a public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, network, other types of networks, or a combination of two or more such networks. For example, network 880 or a portion thereof may include a wireless or cellular network, and coupling 882 may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other types of cellular or wireless coupling. In this example, coupling 882 may implement a variety of data transmission technologies, such as single carrier radio transmission technology (1xRTT), enhanced data rate optimized (EVDO) technology, general packet radio service (GPRS) technology, GSM enhanced data rate (EDGE) technology, third generation partnership project (3GPP), including 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide microwave interoperability (WiMAX), long term evolution (LTE) standard, other protocols defined by various standards development organizations, other long distance protocols, or other data transmission technologies.

[0104] Instructions 816 may be transmitted or received over a transmission medium via network 880 using a network interface device (e.g., a network interface component included in communication component 864) and utilizing any of the well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 816 may also be transmitted or received to device 870 via a transmission medium via coupling 872 (e.g., a point-to-point coupling). The terms "transmission medium" and "signal medium" have the same meaning in this disclosure and may be used interchangeably. The terms "transmission medium" and "signal medium" shall include any intangible medium capable of storing, encoding, or carrying instructions 816 for execution by machine 800, and include digital or analog communication signals or other intangible media to facilitate the communication of such software. Thus, the terms "transmission medium" and "signal medium" shall include any form of modulated data signals, carrier waves, and the like. The term "modulated data signal" refers to a signal that has one or more of its characteristics set or changed in some manner so as to encode information in the signal.

[0105] The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" have the same meaning and are used interchangeably. These terms are defined to include both machine storage media and transmission media. Therefore, these terms include both storage devices / medium and carrier / modulated data signals.

Claims

1. A system comprising: a lighting device comprising a plurality of independently controllable light sources and at least one camera; A computer system comprising at least one hardware processor and a non-transitory computer-readable medium storing instructions, which, when executed by the at least one hardware processor, causes the at least one hardware processor to perform the following operations: Accessing a sequence of one or more capture configurations, each capture configuration defining a plurality of image captures by at least one camera, each image capture having an exposure time, and indicating which individual light sources of the independently controllable light sources are illuminated during the plurality of image captures; receiving a plurality of component images captured while executing one of the one or more capture configurations of the sequence; performing a photometric stereo operation on the plurality of component images to create a three-dimensional reconstruction of the component; Measuring depth variations of component surface features in 3D reconstructions; as well as Whether the component surface feature is a defect in the component is determined based on the depth variation of the component surface feature.

2. The system according to claim 1, characterized in that The depth variation is measured using a Poisson distribution.

3. The system according to claim 1, characterized in that The depth variation is measured using a convolutional neural network trained to predict classification of surface features.

4. The system according to claim 3, characterized in that The training of the convolutional neural network includes: accessing a plurality of images from a first image data source, each image having n color channels; The convolutional neural network is trained using a plurality of labels and the plurality of images, each label of the plurality of labels corresponding to a classification.

5. The system according to claim 1, characterized in that The plurality of component images are sequentially captured by a camera while the component is held in a specific position.

6. The system according to claim 1, characterized in that At least one of the one or more capture configurations is automatically generated by a technique in which known specifications of the test component are used to select groupings of independently controllable light sources.

7. The system according to claim 6, characterized in that The techniques include: testing a plurality of image groups, each image group comprising a plurality of images, the images being respectively captured by at least one camera of the lighting device when the lighting device illuminates different single light sources on the test component; Testing the plurality of image groups includes performing a photometric stereo operation on each image group to create a three-dimensional reconstruction of the plurality of components, measuring depth variations of surface portions of the tested components in the plurality of three-dimensional reconstructions, and selecting the image group whose depth variation measurements are closest to known specifications.

8. A method comprising: Accessing a sequence of one or more capture configurations, each capture configuration defining a plurality of image captures by at least one camera, each image capture having an exposure time, and indicating which individual light sources of the independently controllable light sources are illuminated during the plurality of image captures; receiving a plurality of component images captured while executing one of the one or more capture configurations of the sequence; performing a photometric stereo operation on the plurality of component images to create a three-dimensional reconstruction of the component; Measuring depth variations of component surface features in 3D reconstructions; as well as Whether the component surface feature is a defect in the component is determined based on the depth variation of the component surface feature.

9. The method according to claim 8, characterized in that The depth variation is measured using a Poisson distribution.

10. The method according to claim 8, characterized in that The depth variation is measured using a convolutional neural network trained to predict classification of surface features.

11. The method according to claim 10, characterized in that The training of the convolutional neural network includes: accessing a plurality of images from a first image data source, each image having n color channels; The convolutional neural network is trained using a plurality of labels and the plurality of images, each label of the plurality of labels corresponding to a classification.

12. The method according to claim 8, characterized in that The plurality of component images are sequentially captured by a camera while the component is held in a specific position.

13. The method according to claim 8, characterized in that At least one of the one or more capture configurations is automatically generated by a technique in which known specifications of the test component are used to select groupings of independently controllable light sources.

14. The method according to claim 13, characterized in that The techniques include: testing a plurality of image groups, each image group comprising a plurality of images, the images being respectively captured by at least one camera of the lighting device when the lighting device illuminates different single light sources on the test component; Testing the plurality of image groups includes performing a photometric stereo operation on each image group to create a three-dimensional reconstruction of the plurality of components, measuring depth variations of surface portions of the tested components in the plurality of three-dimensional reconstructions, and selecting the image group whose depth variation measurements are closest to known specifications.

15. A non-transitory machine-readable storage medium having instructions thereon executable by one or more machines, the instructions performing the following operations: Accessing a sequence of one or more capture configurations, each capture configuration defining a plurality of image captures by at least one camera, each image capture having an exposure time, and indicating which individual light sources of the independently controllable light sources are illuminated during the plurality of image captures; receiving a plurality of component images captured while executing one of the one or more capture configurations of the sequence; performing a photometric stereo operation on the plurality of component images to create a three-dimensional reconstruction of the component; Measuring depth variations of component surface features in 3D reconstructions; as well as Whether the component surface feature is a defect in the component is determined based on the depth variation of the component surface feature.

16. The non-transitory machine-readable storage medium according to claim 15, wherein: The depth variation is measured using a Poisson distribution.

17. The non-transitory machine-readable storage medium according to claim 15, wherein: The depth variation is measured using a convolutional neural network trained to predict classification of surface features.

18. The non-transitory machine-readable storage medium according to claim 17, wherein: The training of the convolutional neural network includes: accessing a plurality of images from a first image data source, each image having n color channels; The convolutional neural network is trained using a plurality of labels and the plurality of images, each label of the plurality of labels corresponding to a classification.

19. The non-transitory machine-readable storage medium according to claim 15, wherein: The plurality of component images are sequentially captured by a camera while the component is held in a specific position.

20. The non-transitory machine-readable storage medium of claim 15, wherein: At least one of the one or more capture configurations is automatically generated by a technique in which known specifications of the test component are used to select groupings of independently controllable light sources.

Images