Learning-based perspective sensing suitable for factory automation

Abstract

A scanner for image reconstruction of a structure of a target object uses a neural network trained to classify each segment of a sequence of segments of modified waves into one or more classes. The sequence of segments corresponds to a sequence of layers of the target object such that a segment of modified waves corresponds to a layer in the sequence of layers having the same index as the segment in the sequence of segments. The scanner executes the neural network for each wave modified by penetrating a layer of the target object to produce a class of the segment of modified waves. Next, the scanner selects the classes of segments of different modified waves corresponding to the same layer to produce an image of the layer of the target object, the pixel values of the image being a function of the labels of the selected classes.

Description

Technical Field

[0001] This invention relates to perspective sensing, and more specifically, to systems and methods for image reconstruction of the structure of a target object. Background Technology

[0002] Perspective sensing is important for many applications such as microwave imaging, biological microscopy, medical imaging, through-wall imaging (TWI), infrastructure monitoring, and seismic imaging. In particular, perspective sensing enables the visualization of the internal structure of materials and the characterization of their physical properties. For example, in microwave imaging, perspective sensing provides information about the structure of an object. In biological microscopy, perspective sensing allows visualization of internal cellular structures in three dimensions. In TWI, perspective sensing allows compensation for the delay in signal propagation through walls.

[0003] For example, over the past two decades, terahertz (THz) sensing, in both reflective and transmittance modes, has garnered increasing attention due to its widespread applications in gas sensing, moisture analysis, nondestructive evaluation, biomedical diagnostics, packaging inspection, and security checks. THz sensing systems can examine not only the upper surface of a sample but also its internal structure (whether defects beneath the top layer or multilayer structures) because they can penetrate a wide variety of nonconductive materials.

[0004] However, perspective sensing, such as THz sensing, presents complexities in the image reconstruction of 3D objects. Furthermore, undesirable artifacts degrade the quality of the reconstructed image due to computational complexity. To address this, various techniques, such as layered image restoration described in U.S. Patent 10,217,252, have been employed to improve the quality of the reconstructed image. Nevertheless, a system and method for image reconstruction of the structure of a target object remains needed. Summary of the Invention

[0005] Some embodiments aim to provide a system and method for image reconstruction of the structure of a target object. Additionally or alternatively, another objective of some embodiments is to provide such an image reconstruction system and method suitable for factory automation.

[0006] Some implementations are based on the understanding that, in many cases, the structure of the target object is sparse, and this sparsity can be used for image reconstruction. However, image reconstruction of sparse but three-dimensional structures remains a computationally challenging problem. Some implementations are based on the understanding that this problem can be solved by treating the structure of the target object as a multi-layered structure and reconstructing images of each layer of the target object individually and / or jointly. This multi-layered representation of the structure allows for consideration not only of the sparsity of the entire structure of the target object but also of the sparsity of each individual layer. Furthermore, this multi-layered representation allows for segmentation, thus simplifying the computation of image reconstruction.

[0007] It is important to note that in many cases, the structure of the target object is indeed layered. For example, the target object can be a man-made object with a layered structure, such as a stack of paper (where each sheet is a layer in the multi-layered structure), a semiconductor device formed from layers of different materials, and infrastructure extending underground at different depths. However, in other cases, the target object can be a natural object with a layered structure, such as human tissue with multiple layers. In still other cases, the target object does not have a clearly observable layered structure, but can still be considered as a combination of layers.

[0008] Some implementations are based on the understanding that images of layers of a target object can be reconstructed individually and / or jointly to improve the quality and / or performance of 3D object image reconstruction. However, after numerous experiments, simulations, and tests, some implementations have identified additional problems with layer-by-layer reconstruction. Indeed, why, despite improvements in computational performance, is the image quality of the reconstructed layer images affected? This quality degradation is more pronounced in deeper layers, leading to the understanding that while treating the target object as a layered structure has advantages, this treatment introduces additional problems in multi-level image reconstruction. Specifically, multi-level image reconstruction suffers from shadowing effects due to the non-uniform penetration of waves from earlier layers to deeper layers. This problem can be conceptualized as the structure of one layer casting a shadow on subsequent layers, and this shadow can be considered as the structure of subsequent layers. In this way, shadowing effects contribute to artifacts in the reconstruction of deeper layers.

[0009] Some implementations are based on the understanding that the shadowing effect prevents the layer of the target object from becoming increasingly sparsity in the direction of wave propagation through the layer penetrating the object. In practice, if each subsequent layer has the shadowing effect of the previous layer, then the subsequent layers are at least as sparse as the previous layer, but typically less sparse. In this way, the shadowing effect is detrimental to image reconstruction in the layer's "horizontal" direction, which is perpendicular to and parallel to the plane of the layer and the direction of wave propagation.

[0010] However, some implementations are based on the understanding that when such classification is performed in a "vertical" direction perpendicular to the layer plane, the shadowing effect, considered a disadvantage in image reconstruction using signal processing, can become an advantage in classification-based image reconstruction. Specifically, due to the complexity caused by the shadowing effect, waves penetrating the layered structure are uniquely modified by the material structure. Furthermore, this unique modification is stable.

[0011] As used in this paper, the only modification means that if the target object has a first structure on the first propagation path of the first wave penetrating the target object and a second structure on the second propagation path of the second wave penetrating the target object, and the first structure is different from the second structure, then even if the first wave and the second wave are the same before the modification, the first wave modified by penetration will be different from the second wave modified by penetration.

[0012] As used in this paper, a unique and stable modification means that if the target object has a first structure on the first propagation path of the first wave penetrating the target object and a second structure on the second propagation path of the second wave penetrating the target object, and the first structure is the same as the second structure, then when the first wave and the second wave are the same before the modification, the first wave modified by penetration will be the same as the second wave modified by penetration. However, if the first structure is different from the second structure, then the first wave modified by penetration will be different from the second wave modified by penetration.

[0013] In this way, a wave modified by penetration from a target object leaves a unique and stable feature indicating the object's structure along the penetration path. Therefore, instead of attempting to identify structure during various model-based signal processing methods, some implementations use neural networks trained to classify the modified wave. Furthermore, in various implementations, the object's structure is considered to be layered along the propagation path, and different layers can have different structures. To this end, in some implementations, the neural network is trained to classify each segment in a sequence of modified wave fragments into one or more categories. The fragment sequence corresponds to a layer sequence of the target object, such that a fragment of the modified wave corresponds to a layer in the layer sequence with the same index as the fragment in the fragment sequence. Therefore, by classifying the modified wave, the category of each layer of the target object along the wave's penetration path can be estimated simultaneously, thereby estimating the structure.

[0014] Some implementations are based on the understanding that these classification-based principles can be extended to the reconstruction of images of each layer of a target object. To this end, some implementations emit a set of waves in parallel propagation directions to penetrate the layer sequence of the target object and measure the intensity of the set of waves modified by penetrating the layers of the target object. Due to parallel penetration, the categories of segments corresponding to the same layer for different modified waves can be used to generate images of the layers of the target object, where the pixel values ​​are functions of the label of the selected category. For example, in some implementations, each modified wave is associated with the location of a pixel in the image of each layer, such that a segment of the layer is associated with the location of a pixel associated with its modified wave, and the pixel value at that location in the image of the layer is a function of the label of the category of the segment associated with the pixel location in the image of that layer. Thus, the index of the segment and the location of the wave of the segment specify the layer and the location within the layer, thereby allowing the reconstruction of a two-dimensional (2D) image of the layer and a 3D image of the volume of the target object.

[0015] Therefore, one embodiment discloses a scanner for image reconstruction of a target object structure. The scanner includes: a transmitter configured to emit a set of waves in a parallel propagation direction to penetrate a layer sequence forming the structure of the target object; a receiver configured to measure the intensity of the set of waves modified by penetrating the layers of the target object; a memory configured to store a neural network trained to classify each segment in a segment sequence of modified waves into one or more categories, wherein the segment sequence corresponds to a layer sequence of the target object such that a segment of the modified wave corresponds to a layer in the layer sequence having the same index as the segment in the segment sequence; a hardware processor configured to (1) execute the neural network for each modified wave to generate a category of a segment of the modified wave, and (2) select categories of segments of different modified waves corresponding to the same layer to generate an image of a layer of the target object, the pixel values ​​of the image being a function of the label of the selected category; and an output interface for rendering one or more images of one or more layers of the target object.

[0016] In different implementations, modified waves are classified individually or jointly. Modified waves are classified individually when their category is estimated solely by processing that modified wave. Modified waves are classified jointly when their category is estimated by processing not only that modified wave but also other adjacent modified waves. Individual classification simplifies training and processing. Joint classification can improve the accuracy of annotation.

[0017] In one implementation with separate classification, each modified wave is uniquely associated with the location of a pixel in the image of each layer, and the pixel value is a label for the category of the segment associated with the pixel's location. This implementation is computationally efficient and particularly advantageous when the size of the emitted wave corresponds to the size of the pixel. However, when the wave size is larger than the pixel size and / or the emitted waves partially overlap, adjacent waves may carry some information about neighboring pixels, and this information may be lost if classification is performed considering only a single wave.

[0018] To address this issue, some implementations employ pre-training or combined partial techniques. For example, in one implementation, multiple modified waves are associated with the location of pixels in the image of each layer, and the pixel value is a combination of labels for the categories of the segments associated with that pixel's location. In this implementation, the combined partial technique allows for the use of weighted combinations of multiple categories to account for information from neighboring waves or pixels.

[0019] Additionally or alternatively, some embodiments use a neural network trained to classify segments of modification waves associated with a pixel's location by processing the classified modification wave by combining adjacent modification waves associated with pixels adjacent to the classified modification wave. In this embodiment, the size of the classified segment sequence remains unchanged, but the amount of information used for classification is increased. For example, in one embodiment, the neural network is trained to classify segments of modification waves associated with a pixel's location by processing a cluster of modification waves corresponding to a cluster of pixels centered at the location of the pixel of the classified modification wave. This embodiment ensures the accuracy and uniqueness of the classification.

[0020] Different implementations use different types of emitters selected based on the expected structure of the target object and the desired type of image reconstruction. Examples of emitters include optical, ultrasonic, and X-ray emitters. Some implementations use terahertz (THz) emitters that emit within the terahertz frequency range (e.g., from 0.3 to 3 terahertz, corresponding to wavelengths from 1 mm to 0.1 mm (or 100 μm)). Because THz scanners are sensitive to non-uniform penetrating illumination from the front layer to the depth layer, combined layer-by-layer image restoration is advantageous for these types of emitters.

[0021] Additionally or alternatively, some implementations use scanners operating in a compressed scanning mode. Due to the principle of compressed reconstruction, this type of scanner can further reduce reliance on non-uniform penetration. For example, in one implementation, the transmitter includes: a collimator that collimates the wave into a wide beam; and a spatial encoder that spatially encodes the wide beam using a random mask. Additionally, the receiver includes: a focusing lens for focusing the reflected wave; and a single-pixel photoconductivity detector that receives the focused wave from the focusing lens to provide one measurement of the focused wave at a time, thereby allowing the recovery of images of multi-layered structures using sparse reconstruction.

[0022] Some implementations employ scanners, according to various embodiments, as part of an automation system. Automation is the technology of performing processes or procedures with minimal human assistance. Automation or automatic control uses various control systems to operate equipment with minimal or reduced human intervention, such as machines, plant processes, boilers and heat treatment furnaces, telephone network switching, ships, aircraft and other applications, and vehicle steering and stability. Some implementations are based on the understanding that classification-based image reconstruction of different layers of manufactured objects is advantageous for factory automation, including real-time diagnostic processes. For example, the layered structure and dimensions of modern manufacturing allow for the use of THz-TDS imaging in some embodiments during the manufacturing process. Attached Figure Description

[0023] [ Figure 1 ]

[0024] Figure 1 A block diagram of a scanner suitable for use as an image of the dielectric constant distribution of a material, according to some embodiments, is shown.

[0025] [ Figure 2 ]

[0026] Figure 2 A schematic diagram of an exemplary structure of a target object according to one embodiment is shown.

[0027] [ Figure 3 ]

[0028] Figure 3 A schematic diagram illustrating the classification principles used in some implementations of perspective image reconstruction is shown.

[0029] [ Figure 4 ]

[0030] Figure 4 A schematic diagram of a scanner 400 according to some embodiments is shown.

[0031] [ Figure 5 ]

[0032] Figure 5 A cross-section of the layered structure of a target object along the wave propagation path is shown according to some embodiments.

[0033] [ Figure 6 ]

[0034] Figure 6 A cross-section of a layered structure of a target object perpendicular to the wave propagation path is shown according to some embodiments.

[0035] [ Figure 7 ]

[0036] Figure 7 A schematic diagram of redundant associations according to some implementation methods is shown.

[0037] [ Figure 8 ]

[0038] Figure 8 A schematic diagram of overlapping waves used in some implementations is shown.

[0039] [ Figure 9 ]

[0040] Figure 9 A schematic diagram of image reconstruction using deconvolution response according to one implementation is shown.

[0041] [ Figure 10 ]

[0042] Figure 10 A block diagram of a computer-based information system according to some implementations is shown.

[0043] [ Figure 11A ]

[0044] Figure 11A A schematic diagram of a training classification neural network according to some implementations is shown.

[0045] [ Figure 11B ]

[0046] Figure 11B A schematic diagram of a deep residual network (ResNet) used in some implementations is shown.

[0047] [ Figure 11C ]

[0048] Figure 11C A schematic diagram of an exemplary DNN architecture for a classification neural network used by one implementation is shown.

[0049] [ Figure 12A ]

[0050] Figure 12AA schematic diagram of a THz scanner according to different implementations is shown.

[0051] [ Figure 12B ]

[0052] Figure 12B A schematic diagram of a THz scanner according to different implementations is shown.

[0053] [ Figure 12C ]

[0054] Figure 12C A schematic diagram of a THz scanner according to different implementations is shown.

[0055] [ Figure 12D ]

[0056] Figure 12D A schematic diagram of a THz scanner according to different implementations is shown.

[0057] [ Figure 13 ]

[0058] Figure 13 A schematic diagram of a multilayer structure with different layer patterns according to one embodiment is shown.

[0059] [ Figure 14 ]

[0060] Figure 14 A schematic diagram of an automated system including a scanner, according to some embodiments, is shown.

[0061] [ Figure 15 ]

[0062] Figure 15 A schematic diagram of the layered structure of a photonic integrated circuit (PIC) manufactured by an automated system according to some embodiments is shown. Detailed Implementation

[0063] Figure 1 A block diagram of a scanner 100, according to some embodiments, suitable for use as a dielectric constant sensor for determining the dielectric constant distribution of an image 110 of material 120 is shown. The dielectric constant sensor includes at least one transceiver 130 for propagating waves 135 through the material of the target object 120 and receiving a set of echoes 137 generated by pulses scattered from different parts of the material.

[0064] For example, a transceiver may include at least one transmitter that transmits pulses through the material, such that the pulses scattered by the material produce a set of echoes 137. The pulses can be any type of electromagnetic or light wave, such as one or a combination of microwave pulses, radar pulses, laser pulses, ultrasonic pulses, and acoustic pulses. The transceiver may also include at least one receiver arranged at a predetermined position relative to the transmitter for receiving the set of echoes 137. For example, in one embodiment, the receiver and transmitter form transceiver 130. In different embodiments, the receiver and transmitter are located on the same side of object 120, such that the echoes 137 include the propagation and reflection of the emitted wave. In different embodiments, the receiver and transmitter are located on different sides of object 120, such that the emitted wave is modified once by propagation through the object material. According to different embodiments, the dielectric constant sensor can generate a two-dimensional or three-dimensional image of the material, wherein each location in the image provides a dielectric constant value of the material corresponding to that location.

[0065] The dielectric constant sensor also includes a processor 140 operatively connected to transceiver 130 to determine an image 110 based on a set of echoes 137. To account for multiple scattering, the processor uses a neural network 150 to generate an image 110 of the dielectric constant distribution of the material. Image reconstruction of a three-dimensional structure is computationally challenging due to the complexity of multiple scattering of propagating waves by the material structure. To address this complexity, instead of attempting to reconstruct the physical properties of the scattering using a neural network, neural network 150 is a classification network trained to classify waves modified by propagation into one or more categories. Therefore, neural network 150 is referred to herein as a classification neural network.

[0066] Specifically, some implementations are based on the understanding that the complexity of image reproduction of the complex physical process of wave scattering can be addressed by treating the structure of the target object as a multi-layered structure and reconstructing images of each layer of the target object individually and / or jointly. This multi-layered representation of the structure allows for consideration not only of the sparsity of the entire structure of the target object but also of the sparsity of each individual layer. Furthermore, this multi-layered representation allows for segmentation, thus simplifying the computation of image reconstruction.

[0067] It is important to note that in many cases, the structure of the target object is indeed layered. For example, the target object can be a man-made object with a layered structure, such as a stack of paper where each sheet is a layer in a multi-layered structure, a semiconductor device formed by layers of different materials, and infrastructure extending underground at different depths. However, in other cases, the target object can be a natural object with a layered structure, such as human tissue with multiple layers. In still other cases, the target object does not have a clearly observable layered structure, but can still be considered as a combination of layers.

[0068] Figure 2 A schematic diagram of an exemplary structure of a target object according to one embodiment is shown. In this example, the target structure is an artificial object with a layered structure, i.e., a stack of three sheets of paper. This example illustrates an image reconstruction problem created by treating the target object as a layered structure.

[0069] Specifically, some implementations are based on the understanding that images of layers of a target object can be reconstructed individually and / or jointly. Some implementations reconstruct images using sparse reconstruction by fitting the reconstructed image to intensity measurements of reflected waves. This sparse reconstruction is performed through regularization to prevent overfitting. Different implementations perform individual and / or joint reconstruction of layers by selecting appropriate regularization. For example, for individual reconstruction, the regularization is individual for each layer. In contrast, for joint reconstruction, the regularization is a joint regularization, a weighted combination of the individual regularizations determined for each layer.

[0070] However, after numerous experiments, simulations, and tests, some implementation methods have revealed additional problems with layered reconstruction. Indeed, despite improvements in computational performance, the image quality of the reconstructed images from different layers is compromised. This quality degradation is more pronounced in deeper layers. Joint reconstruction partially addresses this issue, but in many experiments, deeper images reconstructed using individual and / or joint reconstruction exhibit artifacts.

[0071] Some implementations are based on the understanding that while treating the target object as a hierarchical structure has advantages, this approach introduces additional problems in multi-level image reconstruction. Specifically, multi-level image reconstruction suffers from shadowing effects due to the non-uniform penetration of waves from earlier to deeper layers. This problem can be conceptualized as the structure of one layer casting a shadow on subsequent layers, and this shadow may be perceived as part of the structure of those subsequent layers. In this way, shadowing effects contribute to artifacts in the reconstruction of deeper layers.

[0072] Figure 2 A schematic diagram illustrating the shading effect of the pattern from the front layer to the deeper layers is shown. The pattern 215 of the letter "M" appears in the first layer 210 of the target object structure, and the pattern 225 of the letter "E" appears in the second layer 220 of structure 130. Due to uneven penetration, the shading of the letter "M" appears 240 on the second layer 220 and also 241 on the third layer 230. Similarly, the shading of the letter "E" appears 250 on the third layer 230. The aim of some embodiments is to recover the pattern / letter "M" and "E" even in the presence of shading effects.

[0073] Some implementations are based on the understanding that the shadowing effect prevents the target object's layers from becoming increasingly sparsity in the direction of wave propagation through the layers penetrating the object. During layered reconstruction 260, each subsequent layer has a shadowing effect from the previous layer, and subsequent layers are at least as sparse as the previous layer, but typically less sparse. To this end, some implementations are based on the understanding that if such a constraint is imposed on the joint image reconstruction of layers, this constraint forces the reconstruction to account for the shadowing effect to reduce unwanted artifacts. This makes model-based image reconstruction 260 in the "horizontal" direction within each layer plane computationally challenging.

[0074] Figure 3 A schematic diagram illustrating the classification principles used in some implementations of perspective image reconstruction is shown. The implementations are based on the understanding that shadowing effects, considered a disadvantage in signal processing-based image reconstruction, can become an advantage in classification-based image reconstruction. Specifically, due to the complexity caused by shadowing effects, waves penetrating layered structures are uniquely modified by the material structure. Furthermore, this unique modification is stable.

[0075] As used in this paper, the only modification means that if the target object has a first structure on the first propagation path of the first wave penetrating the target object and a second structure on the second propagation path of the second wave penetrating the target object, and the first structure is different from the second structure, then even if the first wave and the second wave are the same before the modification, the first wave modified by penetration will be different from the second wave modified by penetration.

[0076] As used in this paper, a stable modification means that if the target object has a first structure on the first propagation path of the first wave penetrating the target object and a second structure on the second propagation path of the second wave penetrating the target object, and the first structure is the same as the second structure, then when the first wave and the second wave are the same before the modification, the first wave modified by penetration will also be the same as the second wave modified by penetration.

[0077] In this way, the wave modified by penetration from the target object leaves a unique and stable feature indicating the object's structure along the penetration path. Therefore, instead of attempting to identify structure during various model-based signal processing methods, some implementations use neural networks trained to classify the modified signal.

[0078] exist Figure 3 In the example, the target object comprises three layers: 310, 312, and 314. Instead of viewing as per [reference], see [reference]. Figure 2While describing the "horizontal" direction, some implementations examine the "vertical" direction (i.e., along the wave propagation path) to analyze / classify the waves modified by propagation 340. For example, voxels 320 and 330 extending into the wave propagation direction include waves 325 and 335 modified by scattering through the material of the layer. This scattering is sophisticated to ensure that if the materials of voxels 320 and 320 are different from each other, the modified waves 325 and 335 will also be different from each other. However, the scattering works in a stable manner, so if multiple waves repeatedly propagate through the material of voxel 320, the propagating waves 325 will have similar characteristics suitable for classification 340.

[0079] In this way, a classification-based neural network 340 can be trained to classify the modified waves 325 and 335 into vectors 327 and 337 labeled with categories, for example, those that can be used to reconstruct images of the material of the target object. Furthermore, in various embodiments, the structure of the object is considered to be layered along the propagation path, and different layers may have different structures. Therefore, in some embodiments, the neural network is trained to classify each segment in the fragment sequences 327 and 337 of the modified waves 325 and 335 into one or more categories. The fragment sequence corresponds to a layer sequence of the target object, such that a segment of the modified wave corresponds to a layer in the layer sequence with the same index as the segment in the fragment sequence. Thus, by classifying the modified waves, the category of each layer of the target object along the wave's penetration path can be estimated simultaneously, thereby estimating the structure.

[0080] Some implementations are based on the understanding that these classification-based principles can be extended to the reconstruction of images of each layer of a target object. To this end, some implementations emit a set of waves in parallel propagation directions to penetrate the layer sequence of the target object and measure the intensity of the set of waves modified by penetrating the layers of the target object. Due to parallel penetration, the categories of segments corresponding to the same layer for different modified waves can be used to generate images of the layers of the target object, where the pixel values ​​are functions of the label of the selected category. For example, in some implementations, each modified wave is associated with the location of a pixel in the image of each layer, such that a segment of the layer is associated with the location of a pixel associated with its modified wave, and the pixel value at that location in the image of the layer is a function of the label of the category of the segment associated with the pixel location in the image of that layer. Thus, the index of the segment and the location of the wave of the segment specify the layer and the location within the layer, thereby allowing the reconstruction of a two-dimensional (2D) image of the layer and a 3D image of the volume of the target object.

[0081] Figure 4A schematic diagram of a scanner 400 according to some embodiments is shown. The scanner includes a sensing system 420, which includes a transmitter 421 and a receiver 424. The transmitter is configured to emit a set of waves 427 from an antenna 422 along a propagation direction to penetrate layers 431-432 of the structure of a target object 430. The receiver 424 is configured to measure the intensity of waves 429 that have passed through layer modifications of the target object 430 and are received by the antenna 423.

[0082] Scanner 400 includes a computer-based information system 410, which includes a hardware processor configured to execute a neural network 402 for each modified wave 429 to generate segmented categories of the modified wave. The neural network is trained to classify each segment in a sequence of modified wave segments into one or more categories. The processor selects 408 categories of segments corresponding to the same layer from different modified waves to generate an image of the layer of the target object, the pixel values ​​of which are a function of the label of the selected category. Scanner 400 also includes an output interface 441 for rendering a reconstructed image of the layer of the target object.

[0083] In some implementations, the sequence of segments corresponds to a sequence of layers of the target object, such that a segment of the modified wave corresponds to a layer in the layer sequence having the same index as the segment in the segment sequence. This correspondence between the indexes of the labels generated by the neural network and the indices of the layers of the target object allows for layer-based image reconstruction. Additionally or alternatively, each modified wave is associated with the location of a pixel in the image of each layer, such that a segment of the layer is associated with the location of the pixel associated with its modified wave, and wherein the pixel value at that location in the image of the layer is a function of the label of the category of the segment associated with that pixel location in the image of the layer.

[0084] Figure 5 A cross-section of the layered structure of a target object along the propagation path of a wave, according to some embodiments, is shown to illustrate the principle of modified wave classification 550. In this example, the target object 500 includes 10 layers 510. In some embodiments, these layers are physical layers of the target object's structure. In other embodiments, layers can be considered as abstract concepts captured during the training of a classification neural network 550 according to some embodiments. This is because the number of layers corresponds to the number of outputs of the neural network. For example, if the target object has 10 layers, the output of the neural network classifying modified waves (such as wave 520 penetrating the target object 500) includes 10 labels. Similarly, if the output of the classification neural network 550 includes 10 labels, the target object 500 is considered to have 10 layers.

[0085] In this way, the index of layer 530 is the index of the segment of wave 520 and the index of label 535 in the output of the neural network. This index allows processor 402 to select 408 a label 525 in the output of the neural network that has the same index as the label 535 of the image forming the same layer 530.

[0086] Similarly, each column of the output of neural network 550 corresponds to the classification of a specific wave. For example, if seven waves 540 penetrate the layers of an object in a single cross-section, the classification output comprises seven vectors 545. For example, vector 525 corresponds to the classification result of wave 320. Wave 520, in turn, corresponds to a specific location across all layers that allows the association of classification values ​​(such as labels 555) with the layer and its location within the layer.

[0087] Some implementations use a 2L×1 binary content vector (e.g., [0; 0; 0; 0; 0; 0]). T The content on layer L is represented by a ), where 1 indicates the presence of pencil ink, and 0 indicates the absence of pencil ink in that pixel. These implementations treat each layer as having two sides. Additionally or alternatively, in some implementations, the neural network is a binary classifier to estimate the black or white value at the pixel location of the layer's image. Additionally or alternatively, in some implementations, the neural network is a multi-class classifier to estimate the grayscale value at the pixel location of the layer's image. Additionally or alternatively, in some implementations, the neural network is a multi-class classifier to estimate the dielectric constant value of the target object's material at the pixel location of the layer's image.

[0088] Figure 6 A cross-section 610 of a layered structure of a target object perpendicular to the wave propagation path is shown according to some embodiments to illustrate the principle of the location of the classification output in the layer image 620 of the target object. In various embodiments, each modified wave is associated 605 with the location of a pixel in the image of each layer. For example, wave 630 is associated 635 with the location of pixel 637 in image 620, while wave 640 is associated 645 with the location of pixel 647. In this way, the pixel value is a label of the category of the segment associated with the pixel's location.

[0089] In some implementations, each modified wave is uniquely associated with the location of a pixel in the image of each layer, and the pixel value is a label of the category of the segment associated with that pixel location. For example, the value of pixel 637 in layer 630 is a label of the category of a segment of wave 630, which has an index of that layer in the segment sequence of wave 630. In different implementations, multiple modified waves are associated with the location of pixels in the image of each layer, and the pixel value is a combination of labels of the category of the segments associated with that pixel location.

[0090] Figure 7 A schematic diagram of a redundant association according to some implementations is shown. In this example, multiple waves in box 730 are associated with the position of pixel 720 such that the pixel value is a combination of the labels of the waves in box 730. Different waves can have different weights in the combination. For example, as Figure 7 In this implementation, the center wave 710 corresponding to the position of pixel 720 can have the highest weight.

[0091] This implementation is based on the understanding that modified waves can overlap with the positions of multiple pixels, and that multiple waves can reuse the full energy of waves adjacent to the positions of pixels. In this implementation, this energy is collected in post-processing after classification. Additionally or alternatively, in some implementations, the distribution of energy is considered during the training of the classification neural network.

[0092] Figure 8 A schematic diagram of overlapping waves used in some implementations is shown. In this example, because the energies of adjacent waves 810 overlap, the neural network is trained to classify segments of modified waves associated with the pixel location by processing the classified modified wave by combining adjacent modified waves associated with pixels adjacent to the pixel of the classified modified wave. For example, the neural network is trained to classify segments of modified waves associated with the location of pixel 820 by processing a cluster of modified waves corresponding to a cluster of pixels 822, 824, 826, and 828 centered at the location of pixel 820 of the classified modified wave. In this way, the accuracy and flexibility of classification-based image reconstruction can be increased.

[0093] Figure 9 A schematic diagram of image reconstruction using deconvolution responses according to one embodiment is shown. In this embodiment, a transmitter is configured to emit a reference wave 900 that propagates at the same distance as a modified wave 940 without penetrating the layer of the target object. The embodiment is configured to deconvolve 910 on each modified wave 930 (e.g., wave 920) using the reference wave 900 to produce a set of deconvolution responses. A neural network is trained against the deconvolution responses of the modified waves to classify 940 each segment of the deconvolution response into one or more categories, such that the embodiment determines the image of the layer by classifying a set of deconvolution responses using the neural network. Deconvolution simplifies the training of the neural network without degrading the quality of image reconstruction.

[0094] Figure 10A block diagram of a computer-based information system 1010 according to some embodiments is shown. The information system 1010 may include a processor 1002 configured to execute stored instructions and a memory 1008 capable of storing instructions executable by the processor. The processor 1002 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. The memory 1008 may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable non-transitory computer-readable storage medium. The processor 1002 is connected via a bus 1006 to one or more input interfaces / devices 1065 and output interfaces / devices 1041.

[0095] These instructions 1004, stored in memory 1008, can realize image restoration of the structure of the target object. For example, the instructions may include preprocessing 1052 of the measurement results 1095 of the reflected waves, such as filtering, segmentation, time gating, peak finding, and denoising. These instructions also provide implementations of image reconstruction 1053 according to different embodiments. Optionally, the instructions may include post-processing to further improve the quality of the reconstructed image and / or combine the reconstructed images of the target object layers to produce an image of the structure of the target object.

[0096] Information system 1010 may include an output interface / device 1041 for rendering estimated information. In some embodiments, output interface 1041 may include a printer interface (not shown) adapted to connect an encoder to a printing device (not shown). In some embodiments, display interface 1047 may be adapted to connect processor 1002 to display device 1042. Display device 1042 may include a camera, computer, scanner, mobile device, webcam, or any combination thereof. In some embodiments, network interface 1043 is adapted to connect processor 1002 and potentially additionally connect to one or more third-party devices 1044 on network 1090. In some embodiments, by way of a non-limiting example, application interface 1045 may be used to submit estimated information to application device 1046, such as a controller, for controlling the movement of a moving object.

[0097] Information system 1010 may also include input interface 1065 to receive amplitude measurement results 1095 of the modified wave amplitude. For example, network interface controller (NIC) 1060 may be adapted to connect information system 1010 to network 1090 via bus 1006. Network 1090 may be implemented as a wired or wireless network. Through network 1090 and / or other implementations of input interface 1065, the measurement results 1095 of the reflected signal amplitude can be downloaded and stored for storage and / or further processing.

[0098] Figure 11AA schematic diagram of a training classification neural network according to some implementations is shown. A neural network is a series of models inspired by biological neural networks and used to estimate or approximate functions that can depend on a large number of inputs and are often unknown. A neural network typically behaves as a system of interconnected nodes or “neurons” exchanging messages with each other. Each node (such as nodes 1111, 1112, 1113) is associated with a function used to transform a wave 1135 modified by a layer penetrating a target object. This function is typically non-linear, forming the non-linear part of the message transformation. Each connection between nodes is associated with numerical weights (such as weights 1115, 1116, 1117) used to scale the message, forming the linear part of the message transformation. Typically, the function is fixed and predetermined for all nodes, for example, chosen by the designer of the neural network. Examples of functions typically chosen for nodes include sigmoid functions and rectifier functions. In contrast, the numerical weights are different and adjusted empirically, enabling the neural network to adapt to the input and learn.

[0099] In addition to the hidden layers, the classification neural network may have an input layer 1135 for receiving the modified wave and an output layer 1110 for outputting the classification result. The classification result is compared 1107 with real-world information 1137 about the structure of the material corresponding to the wave propagation, and the error is propagated 1106 back to update the weights and / or functions of the nodes in the classification neural network. The real-world information can be obtained from simulations of the material structure and / or various alternative analyses.

[0100] Figure 11B A schematic diagram of the Deep Residual Network (ResNet) 1150 used in some implementations is shown. The ResNet 1150 uses so-called residual blocks to skip one or more layers in a deep learning architecture, achieving robust performance even in very deep network models for learning classification vectors.

[0101] Figure 11CA schematic diagram of an exemplary DNN architecture for a classification neural network used in one implementation is shown. The DNN feeds (truncates) a THz-TDS time-domain waveform 1151 and outputs an estimated binary content vector. This implementation can also extend the input waveform by including the time-domain waveform 1151 from nearby pixels, depending on the THz-TDS aperture size. Specifically, this implementation truncates the full-length time-domain waveform into a 1024×1 real vector and feeds this waveform vector into the input layer 1153 of the DNN, where the input layer 1153 is first transformed to a 200-node dimension through a fully connected linear layer. The DNN then employs two hidden layers, each with 200 nodes, including a batch normalization layer 1155, a rectified linear unit (ReLU) activation layer 1157 with 10% dropout, and a fully connected linear layer 1159. Dropout is a technique to prevent overfitting and improve generalization. Additionally, some implementations consider skip connections 1161 that jump from the input to the output of a hidden layer to learn residual gradients to improve training stability; i.e., ResNet. A fully connected linear layer 1169 following a dropped activation layer 1167 produces the output of the DNN. The DNN is trained to minimize a softmax cross-entropy loss 1175 to predict content vectors from labeled training pixels.

[0102] It should be noted that in many cases, the structure of the target object is indeed layered. For example, the target object can be a man-made object with a layered structure, such as a stack of paper (where each sheet is a layer in the multi-layered structure), a semiconductor device formed from layers of different materials, and infrastructure extending underground at different depths. Alternatively, the target object can be a natural object with a layered structure, such as human tissue with multiple layers. In these cases, the layers of the multi-layered structure can be identified using a pick-and-seek method that searches for the inhomogeneities of waves reflected from the layers of the target object.

[0103] However, in other cases, the target object does not have a clearly observable hierarchical structure, and segmentation can be performed using different techniques (such as time gating with uniform and / or non-uniform gating periods). In any case, in some implementations, the target object is considered to have a multi-layered structure with hierarchical layers defined by the segmentation of reflected waves.

[0104] For example, by stacking the columns of the two-dimensional reflection matrix in the l-th layer of the sample, x can be made... l =[x l (1),x l (2),...,x l (N)] Tdenotes the reflection vector, where N is the total number of pixels at each layer of the sample. For example, a transmitter (such as a THz source) irradiates a target object from a spatially encoded mask. Then, the received measurement results can be described as

[0105] y l =A l x l +v l , (1)

[0106] where A l =[a l,1 ,…,a l,M T is the measurement matrix corresponding to the l-th layer, and v l =[v l (1),...,v l (M)] T is Gaussian distributed noise with zero mean and variance unknown, that is,[[]] y l =[y l (1),...,y l (M)] T , and M is the number of measurements.[[]]

[0107] In practice, the THz-band SLM may remain unchanged during the electro-optic sampling process, which results in the same measurement matrix A for all layers. However, some embodiments assume that the measurement matrix A l is a function of the layer index, because the measurement matrix can absorb layer-related mutual reflections and surface irregularities.[[]]

[0108] Note that the signal model of (1) can actually describe both grating and compressive scan acquisitions. For example, in the case of raster scanning (i.e., each pixel is individually irradiated and measured), M = N and A simplifies to a diagonal matrix, where the diagonal elements are responsible for depth variations. In the case of compressive scanning (e.g., single-pixel THz camera), M < N and each row of the measurement matrix A corresponds to a random mask pattern used to form a measurement result y m . In the case of a layered structure, due to non-uniform penetration irradiation from the front layer to deeper layers, there is a shadow effect.[[]]

[0109] Different implementations use different types of emitters selected based on the expected structure of the target object and the desired type of image reconstruction. Examples of emitters include optical, ultrasonic, and X-ray emitters. Some implementations use terahertz (THz) emitters that emit within the terahertz frequency range (e.g., from 0.3 to 3 terahertz, corresponding to wavelengths from 1 mm to 0.1 mm (or 100 μm)). Because THz scanners are sensitive to non-uniform penetrating illumination from the front layer to the depth layer, combined layer-by-layer image restoration is advantageous for these types of emitters.

[0110] Figure 12A A schematic diagram is shown illustrating the use of a single THz transceiver 1206 to mechanically scan multiple layers 1202, multiple tracks 1201, and multiple scales (i.e., across tracks and along tracks) to receive reflected waveforms in an encoded pattern for location identification. Alternatively, the single THz transceiver may be a transmitter, wherein the transmitted THz waveform passes through multiple scales and is received by a receiver 1209. Specifically, the transmitted THz waveform passes through multiple / multi-scales, and the THz waveform continues to enter the receiver 1209 in an encoded pattern for location identification.

[0111] Figure 12B A schematic diagram is shown using multiple THz transceivers 1207 (or an array of THz transceivers), wherein each THz transceiver in the array can be aligned with a single track 1201. Each THz transceiver is also capable of receiving reflected waveforms to identify the encoded pattern of the corresponding track at that location. The THz transceiver array 1207 can move simultaneously along the track direction D for absolute positioning.

[0112] Figure 12C A schematic diagram is shown using a single THz transceiver 1206 along with a collimating / focusing lens 1211 and a spatial light modulator 1212 in the THz band. The single transceiver 1206 transmits a THz waveform to the collimating lens 1211. The waveform is collimated by the collimating lens 1211 and then modulated with a random pattern by the spatial light modulator 1212. The reflected THz waveform passes through the focusing lens 1211 and is detected by the single THz transceiver 1206.

[0113] Figure 12D A schematic diagram is shown illustrating the use of a THz transmitter 1220 and a THz receiver 1225 separated on both sides of a multi-layer non-overlapping sample 130 in the THz band, along with a collimating / focusing lens 1211 and a spatial light modulator 1212. This acquisition is similar to... Figure 12C However, it is in transmission mode.

[0114] Figure 13A schematic diagram of a multilayer structure with different layer patterns according to one embodiment is shown. The layered structure includes multiple tracks 1301 of coded patterns and multiple layers 1302 of coded patterns. In each track, a digit 1303 specifies the reflectivity level of the coded pattern in the THz band. For example, 0 indicates complete absorption of the THz transmitted signal, while 1 indicates strong reflection of the THz transmitted signal. The digit 1303 can be binary or multi-level with a predefined mapping between reflectivity and numbers. THz scanning on multilayer non-overlapping patterns of this embodiment can be used for positioning in elevator / train systems.

[0115] Some implementations employ scanners, according to various embodiments, as part of an automation system. Automation is the technology of performing processes or procedures with minimal human assistance. Automation or automatic control is the operation of equipment, such as machines, plant processes, boilers and heat treatment furnaces, telephone network switching, ships, aircraft and other applications, and vehicle steering and stability, using various control systems with minimal or reduced human intervention.

[0116] Figure 14 A schematic diagram of an automated system including a scanner, according to some embodiments, is shown. In these embodiments, classification-based image reconstruction is used for anomaly detection to control the process of manufacturing a target object. The automated system includes one or a combination of: a manufacturing controller 1430 configured to control a device 1401 for manufacturing the target object; an anomaly detector 1440 configured to inspect layers of images 1410 of the target object after and / or during the manufacturing process; and a recovery controller 1450 configured to cause modifications to the control of the device based on negative results of the inspection.

[0117] For example, the anomaly detector can compare the reconstructed image 1410 with the test image, and if the comparison error is greater than a threshold, the recovery controller stops control 1435. Additionally or alternatively, the recovery controller can change control 1435 without stopping the manufacturing process. For example, in one embodiment, device 1401 paints the surface of a vehicle body. The reconstructed image includes density information for each layer of the coating, and the recovery controller can request the manufacturing controller to add another layer of coating to areas where the density is insufficient.

[0118] Figure 14The illustrated device 1401 is a robotic assembly that performs operations including inserting components along insertion lines to assemble a target object. The robotic assembly includes a robotic arm that inserts a first component 1403 into a second component 1404. In some embodiments, the robotic arm includes a wrist 1402 for ensuring multiple degrees of freedom of the movable component 1403. In some embodiments, the wrist 1402 has a gripper 1406 for holding the movable component 1403. Examples of the target object include semiconductors, transistors, photonic integrated circuits (PICs), etc.

[0119] Figure 15 A schematic diagram of the layered structure of a photonic integrated circuit (PIC) manufactured by an automated system according to some embodiments is shown. The photonic integrated circuit includes optical components integrated on a substrate 10 having a contact layer 30. The photonic integrated circuit 100 is attached to a base 40 via the bottom of the substrate 10 and the contact layer 30. The base 40 is attached to a temperature-controlled thermoelectric cooler 50, such as a Peltier device. The temperature-controlled thermoelectric cooler 50 may be referred to as a cooling device 50. The substrate 10 may be a semiconductor substrate such as indium phosphide (InP), gallium arsenide (GaAs), gallium nitride (GaN), and silicon (Si). Furthermore, the base 40 may be formed of silicon, aluminum nitride (AlN), or aluminum oxide. The layered structure and size of the PIC allow for THz-TDS imaging during the manufacturing process according to some embodiments.

[0120] The embodiments of the present invention described above can be implemented in any of a variety of ways. For example, the embodiments can be implemented using hardware, software, or a combination thereof. When implemented in software, the software code can execute on any suitable processor or set of processors, whether it is located in a single computer or distributed across multiple computers. Such a processor can be implemented as an integrated circuit, with one or more processors in the integrated circuit assembly. However, the processor can be implemented using any suitable form of circuitry.

[0121] Furthermore, embodiments of the present invention can be embodied in the methods already provided as examples. The actions performed as part of this method can be ordered in any suitable manner. Therefore, embodiments can be constructed that perform actions in a different order than those illustrated, which may include performing some actions simultaneously, even if they are shown as sequential actions in the illustrative embodiments.

[0122] The use of ordinal terms such as “first” or “second” in claims to modify claim elements does not imply any priority, precedence, or order of one claim element relative to another, or the chronological order in which the actions of the method are performed, but is merely used as a label to distinguish one claim element with a specific name from another element with the same name (in addition to using ordinal terms to distinguish claim elements).

[0123] Although the invention has been described by way of example of preferred embodiments, it should be understood that various other adjustments and modifications may be made within the spirit and scope of the invention.

[0124] Therefore, the purpose of the appended claims is to cover all variations and modifications that fall within the true spirit and scope of the invention.

Claims

1. A scanner for image reconstruction of the structure of a target object, the scanner comprising: A transmitter configured to emit a set of waves in a parallel propagation direction to penetrate the layer sequence of the structure forming the target object; A receiver configured to measure the intensity of the set of waves modified by penetrating a layer of the target object; A memory configured to store a neural network trained to classify each segment in a sequence of segments of a modified wave into one or more categories, wherein the sequence of segments corresponds to the sequence of layers of the target object, such that the segments of the modified wave correspond to layers in the sequence of layers having the same index as the segments in the sequence of segments; A hardware processor configured to execute the neural network for each modified wave to generate a category of a segment of the modified wave, and to select categories of segments corresponding to the same layer of different modified waves to generate an image of a layer of the target object, wherein the pixel values ​​of the image are a function of the label of the selected category; as well as An output interface for rendering one or more images from one or more layers of the target object.

2. The scanner according to claim 1, wherein, Each modified wave is associated with the position of a pixel in the image of each layer, such that a fragment of the layer is associated with the position of a pixel associated with the modified wave of that fragment, and wherein the pixel value at that position in the image of the layer is a function of the label of the category of the fragment associated with that pixel position in the image of the layer.

3. The scanner according to claim 2, wherein, Each modified wave is uniquely associated with the location of a pixel in the image of each layer, and the pixel value is a label of the category of the segment associated with the location of that pixel.

4. The scanner according to claim 2, wherein, Multiple modified waves are associated with the location of pixels in the image of each layer, and the pixel value is a combination of labels of the category of the segment associated with the location of the pixel.

5. The scanner according to claim 2, wherein, The neural network is trained to classify segments of the modified wave associated with the position of a pixel by processing the classified modified wave by combining adjacent modified waves associated with pixels adjacent to the classified modified wave.

6. The scanner according to claim 2, wherein, The neural network is trained to classify segments of a modified wave associated with the position of a pixel by processing a cluster of modified waves corresponding to a cluster of pixels centered on the position of the pixel of the classified modified wave.

7. The scanner according to claim 1, wherein, The transmitter is configured to emit a reference wave that propagates at the same distance as the modified wave without penetrating the layer of the target object, wherein the processor is configured to deconvolve each modified wave with the reference wave to generate a set of deconvolution responses, and wherein the neural network is trained on the deconvolution responses of the modified waves to classify each segment of the deconvolution responses into one or more categories, such that the processor determines the image of the layer by classifying the set of deconvolution responses with the neural network.

8. The scanner according to claim 1, wherein, The target object comprises a stack of paper, such that at least one layer of the target object is a sheet of paper with text printed on one or both sides thereon, wherein the image of the layer comprises the text printed on one or both sides of the paper.

9. The scanner according to claim 1, wherein, The transmitter emits waves in the terahertz frequency range.

10. The scanner according to claim 9, wherein, The terahertz frequency range is from 0.3 terahertz to 3 terahertz.

11. The scanner according to claim 1, wherein, The scanner operates in a compressed scanning mode, wherein the transmitter includes a collimator for collimating the wave into a wide beam and a spatial encoder for spatially encoding the wide beam using a random mask, and wherein the receiver includes a focusing lens for focusing the reflected wave and a single-pixel photoconductivity detector, the single-pixel photoconductivity detector receiving the focused wave from the focusing lens to provide one measurement of the focused wave at a time. The hardware processor uses sparse reconstruction to recover images with multi-layered structures.

12. The scanner according to claim 1, wherein, The neural network is a binary classifier that estimates the black or white value of a pixel in the image.

13. The scanner according to claim 1, wherein, The neural network is a multi-class classifier that estimates the grayscale values ​​of pixels in the image.

14. The scanner according to claim 1, wherein, The neural network is a multi-class classifier that estimates the dielectric constant of the material of the target object at the pixel location in the image of the layer.

15. The scanner according to claim 1, wherein, The processor is configured to: Combine the reconstructed images of the layers of the target object to produce an image of the structure of the target object.

16. An automated system comprising the scanner of claim 1, the automated system comprising: A manufacturing controller configured to control a device for manufacturing a target object; An anomaly detector, configured to inspect images of the layer; as well as A recovery controller, configured to modify the control of the device based on the results of the check.

17. The automation system according to claim 16, wherein, The target object is one or a combination of semiconductors, photonic integrated circuits (PICs), and coatings on the surface of a vehicle body.

18. A method for image reconstruction of the structure of a target object, wherein, The method uses a processor associated with stored instructions that implement the method, wherein the instructions, when executed by the processor, perform the steps of the method, the method including the following steps: A set of waves is propagated in parallel directions to penetrate the layer sequence that forms the structure of the target object; Receive the intensity of the set of waves modified by penetrating the layers of the target object; A neural network is executed for each modified wave to generate a category of the segments of the modified wave, wherein the neural network is trained to classify each segment in the segment sequence of the modified wave into one or more categories, wherein the segment sequence corresponds to the layer sequence of the target object, such that the segments of the modified wave correspond to layers in the layer sequence having the same index as the segments in the segment sequence; Different modified wave categories corresponding to segments of the same layer are selected to generate an image of the layer of the target object, wherein the pixel values ​​of the image are a function of the label of the selected category; and Render one or more images from one or more layers of the target object.

19. The method according to claim 18, wherein, The wave is in the terahertz frequency range.

20. A non-transitory computer-readable storage medium having a program executable by a processor for performing a method, the method comprising the steps of: A set of waves is propagated in parallel directions to penetrate the layer sequence that forms the structure of the target object; Receive the intensity of the set of waves modified by penetrating the layers of the target object; A neural network is executed for each modified wave to generate a category of the segments of the modified wave, wherein the neural network is trained to classify each segment in the segment sequence of the modified wave into one or more categories, wherein the segment sequence corresponds to the layer sequence of the target object, such that the segments of the modified wave correspond to layers in the layer sequence having the same index as the segments in the segment sequence; Different modified wave categories corresponding to segments of the same layer are selected to generate an image of the layer of the target object, wherein the pixel values ​​of the image are a function of the label of the selected category; and Render one or more images from one or more layers of the target object.

Images