System and method for reconstructing 3D objects using neural networks

Abstract

A method is provided for determining the correspondence between a projection pattern and an image of the projection pattern projected onto the surface of an object. [Solution] The method includes acquiring an image of an object while a projection pattern is illuminating the surface of the object. The method further includes outputting the correspondence between each corresponding pixel in the image and the coordinates of the projection pattern by using a neural network. The method further includes reconstructing the shape of the surface of the object by using the correspondence between each corresponding pixel in the image and the coordinates of the projection pattern.

Description

Technical Field

[0001] This disclosure generally relates to three-dimensional scanning technology, and more particularly to three-dimensional scanning technology using neural networks.

Background Art

[0002] Three-dimensional (3D) scanning technology can construct a 3D model of the surface of a physical object. Three-dimensional scanning is applied in many fields, including industrial design and manufacturing, computerized animation, science, education, medicine, art, design, and others.

Summary of the Invention

[0003] This disclosure relates to 3D scanning technology. One approach to 3D scanning is the use of so-called "structured light" where a projector projects a known light pattern (hereinafter, "projection pattern") onto the surface of an object. For example, the light from the projector may be guided through a slide on which the projection pattern is printed. From the distortion of the pattern of the light captured by a camera, the shape of the surface of the object is estimated. By using one or more cameras, an image of the pattern reflected on the object may be acquired. By measuring the position of the pattern within the image (e.g., measuring the distortion of the pattern), a computer system may determine the position on the surface of the object using simple geometric calculations such as, for example, a triangulation algorithm.

[0004] In order to determine the position on the surface of the object, the computer system needs to know which points within the projection pattern correspond to which points within the image. According to some embodiments, the correspondence between the image pixels and the coordinates of the projection pattern can be inferred using a trained neural network.

[0005] According to several embodiments, a method is provided for clarifying imaged elements in a structured optical approach to 3D scanning. The method includes acquiring an image of an object. The image includes multiple imaged elements of an imaged pattern. The imaged pattern corresponds to a projection pattern projected onto the surface of the object, and the projection pattern includes multiple projection elements. The method also includes outputting a correspondence between the multiple imaged elements and the multiple projection elements by using a neural network. The method further includes reconstructing the shape of the surface of the object by using the correspondence between the multiple imaged elements and the multiple projection elements.

[0006] According to several embodiments, a method is provided for determining a correspondence between a projection pattern and an image of the projection pattern projected onto the surface of an object. The method includes acquiring an image of the object while the projection pattern is projected onto its surface. The method further includes outputting a correspondence between each corresponding pixel in the image and the coordinates of the projection pattern by using a neural network. The method further includes reconstructing the shape of the object's surface by using the correspondence between each corresponding pixel in the image and the coordinates of the projection pattern.

[0007] According to some embodiments, a method for training a neural network is provided. The neural network is trained using simulated data which includes multiple simulated images relating to projection patterns projected onto the surface of a simulated object. Each simulated image contains a simulated pattern with multiple simulated elements. Each of the simulated elements corresponds to a corresponding projection element in the projection pattern. The simulated data also includes data indicating the shape of each corresponding simulated object and data indicating the correspondence between the simulated elements and their corresponding projection elements. Using the simulated data, a neural network is trained to determine the correspondence between multiple projection elements in the projection pattern and multiple simulated elements in the simulated pattern. The trained neural network is stored for use in reconstructing the image at a later point in time.

[0008] According to some embodiments, an alternative method for training a neural network is provided. The method includes generating simulated data, which includes a plurality of simulated images relating to projection patterns projected onto the surface of corresponding simulated objects, data indicating the shape of each corresponding simulated object, and data indicating the correspondence between the coordinates of the projection patterns and the corresponding pixels in the simulated images. The method further includes training a neural network using the simulated data to determine the correspondence between images and projection patterns. The method further includes storing the trained neural network for use in reconstructing images at a later point in time.

[0009] According to several embodiments, a computer system is provided. The computer system includes one or more processors and a memory storing instructions for performing any of the methods described herein.

[0010] According to some embodiments, a non-transient computer-readable storage medium storing instructions is provided. The non-transient computer-readable storage medium includes instructions that, when executed by a computer system, cause the computer system to perform any of the methods described herein.

[0011] For a better understanding of the various embodiments described, please refer to the following drawings in conjunction with the embodiments for carrying out the invention, where similar reference numerals throughout the drawings refer to corresponding parts. [Brief explanation of the drawing]

[0012] [Figure 1A] Figure 1A shows imaging systems according to several embodiments. [Figure 1B] Figure 1B shows imaging systems according to several embodiments. [Figure 1C] Figure 1C shows projection patterns according to several embodiments. [Figure 1D] Figure 1D shows imaged patterns according to several embodiments. [Figure 2] Figure 2 is a block diagram of an imaging system according to several embodiments. [Figure 3] Figure 3 is a block diagram of a remote device according to several embodiments. [Figure 4] Figure 4 shows the inputs and outputs of neural networks in several embodiments. [Figure 5A] Figure 5A shows a flowchart illustrating a method for 3D reconstruction according to several embodiments. [Figure 5B] Figure 5B shows flowcharts of methods for 3D reconstruction according to several embodiments. [Figure 5C] Figure 5C shows flowcharts of methods for 3D reconstruction according to several embodiments. [Figure 6A] Figure 6A shows flowcharts of methods for training neural networks according to several embodiments. [Figure 6B] Figure 6B shows flowcharts of methods for training neural networks according to several embodiments. [Figure 7]Figure 7 shows a flowchart illustrating another method for 3D reconstruction according to several embodiments. [Modes for carrying out the invention]

[0013] Here, we refer to embodiments illustrated in the accompanying drawings. The following description includes numerous specific details to provide a thorough understanding of the various embodiments described. However, it will be apparent to those skilled in the art that the various embodiments described can be carried out without the need for these specific details. In other examples, well-known methods, procedures, components, circuits, and networks are not described in detail so as not to unnecessarily obscure the inventive aspects of the embodiments.

[0014] Figures 1A and 1B show a three-dimensional ("3D") imaging environment 100 comprising a projector 110 and one or more cameras 112 (e.g., sensors) according to one embodiment of the present invention. Note that in various embodiments, two or more projectors and / or two or more cameras may be used. As shown in Figure 1A, the projector 110 is configured to project a projection pattern (sometimes also referred to as "structured illumination") onto an object 120 to be imaged. To do this, in some embodiments, light from the projector is projected through a slide on which the projection pattern is printed. The projection pattern comprises multiple projection elements. Non-limiting examples of projection patterns include a series of multiple contrast lines (e.g., black and white lines), a series of multiple contrast zigzag lines, and a grid pattern consisting of multiple dots. Further examples of projection patterns are described in U.S. Patent Application No. 11 / 846,494, which is incorporated herein by reference in its entirety.

[0015] Each of the light rays 190-1 to 190-4 corresponds to a respective projection element of the projection pattern (e.g., different lines within the projection pattern). For example, light ray 190-1 represents the first projection element within the projection pattern projected from projector 110 onto the surface 121 of object 120, and light ray 190-2 represents another projection element projected from projector 110 onto the surface 121 of object 120. The light rays 190 are reflected by the surface 121 of object 120 (each being reflected as respective reflected light rays 192-1 to 192-4 corresponding to light rays 190-1 to 190-4). At least a portion of the light is captured by one or more cameras 112.

[0016] In some embodiments, the (one or more) cameras 112 capture a plurality of images of object 120 while the surface 121 of object 120 is illuminated by the projection pattern. In some embodiments, the projection pattern is stroboscopically illuminated onto the surface of object 120, and each time the projection pattern is illuminated onto the surface of object 120, one of the plurality of images is captured. As used herein, the term "stroboscopically" means being performed repeatedly at a fixed rate (e.g., 15 frames per second).

[0017] In FIGS. 1A-1B, although the projector 110 and the camera 112 are shown separately, it should be noted that in some embodiments, the projector 110 and the camera 112 are integrated within a single housing as a 3D scanner 200 (FIG. 2). The user of the 3D scanner 200 may scan the object by moving the 3D scanner 200 relative to the object 120 while collecting data. Thus, in some embodiments, images of object 120 are captured by the (one or more) cameras 112 from different angles or different positions.

[0018] Each image forming the plurality of images has a projection pattern that appears distorted due to the surface of object 120 It shows the imaged pattern corresponding to the [object]. Therefore, the imaged pattern includes a plurality of imaged elements, and each of the imaged elements corresponds to the corresponding projection element in the projection pattern.

[0019] FIG. 1B shows another operation example related to the 3D imaging environment 100. As shown in FIG. 1B, the projector 110 projects the same projection pattern as in FIG. 1A towards the object 120. In this example, another object 122 is arranged within the path of the light ray 190-3, and thus, the light ray 190-3 is reflected on the surface of the object 122. Note that the object 122 may be an integral part of the object 120 (e.g., the handle of a drinking mug) or a separate object. The (one or more) cameras 112 capture a plurality of images regarding the objects 120 and 122. As shown, due to the presence of the object 122, the light ray 192-3 (corresponding to the light ray 190-3) is assumed to enter the (one or more) cameras 112 at a position different from the position shown in FIG. 1A. Therefore, the plurality of images captured by the (one or more) cameras 112 will show the imaged elements corresponding to the light rays 190-3 and 192-3 at different positions (and in some cases, in a different order) within the imaged pattern with respect to the other imaged elements of the imaged pattern.

[0020] FIG. 1C shows an example of the projection pattern 130 emitted from the projector 110, and FIG. 1D shows an example of the imaged pattern 132 captured by the (one or more) cameras 112.

[0021] In the example shown in Figure 1C, the projection pattern 130 includes a plurality of projection elements 140. For example, the projection pattern 130 shown in Figure 1C can be projected onto objects 120 and 122 by the projector 110, as shown in Figure 1B. In this case, the projection elements 140 are a plurality of repeated lines. In some embodiments, as described in U.S. Patent Application No. 11 / 846,494, the lines have alternating thick and thin regions. In some embodiments, the alternating thick and thin regions differ from line to line, but the lines are still considered non-encoded elements.

[0022] Figure 1D is an example of an imaged pattern 132 captured by (one or more) cameras 112 when the projector 110 projects the projection pattern 130 onto objects 120 and 122, as shown in Figure 1B. The image contains multiple imaged elements 142, in this case, multiple distorted lines. Each imaged element 142 (e.g., a distorted line) corresponds to the corresponding projection element 140 (e.g., the corresponding line) in the projection pattern 130. In this example, imaged element 142-1 in imaged pattern 132 corresponds to projection element 140-1 in projection pattern 130, imaged element 142-2 in imaged pattern 132 corresponds to projection element 140-2 in projection pattern 130, imaged element 142-3 in imaged pattern 132 corresponds to projection element 140-3 in projection pattern 130, and furthermore, imaged element 142-4 in imaged pattern 132 corresponds to projection element 140-4 in projection pattern 130. Note that due to the geometry of the object being scanned (as shown in Figure 1B), the imaged elements may be transposed relative to the positions of their respective projection elements 140 in projection pattern 130. For example, the relative order of the imaged elements 142-3 and 142-4 is reversed compared to the projected elements 140-3 and 140-4.

[0023] To construct a model of the surface of an object using a structured light approach, a computer system must establish a correspondence between an image and a projection pattern (for example, each pixel in the image). It is necessary to know the coordinates of the corresponding projection pattern, and / or the correspondence between the imaged elements and the projection elements. To resolve this ambiguity, there are two common approaches: one that utilizes patterns with encoded elements, and the other that relies on patterns with unencoded elements. In patterns with encoded elements, the elements within the pattern have some unique identifying property that allows the computer system to identify each imaged element. In patterns with unencoded elements, the elements within the pattern (e.g., multiple lines, or multiple repeating elements) do not have any individual unique property that makes a particular element of the pattern identifiable in the captured image. In the case of unencoded elements (e.g., multiple lines), some other method is needed to determine the correspondence between the image and the projection pattern.

[0024] In some embodiments, the projection pattern is an unencoded pattern of light, such that the projection elements of the projection pattern are unencoded elements. In some embodiments, as will be described in detail below, the correspondence between the projection pattern and the image of the object onto which the projection pattern is projected is determined using a neural network. In some embodiments, the unencoded pattern of light includes a structured pattern of light, such as a plurality of lines or a plurality of repeating elements.

[0025] Although Figures 1C and 1D show an example in which multiple projection elements 140 within the projection pattern 130 are multiple lines, it will be understood that the projection elements 140 and the imaged elements 142 can take on any shape. For example, the projection elements 140 may be multiple bars, multiple zigzag elements, multiple dots, multiple squares, a series of three small bars, and so on. Non-limiting examples of images showing the imaged pattern formed by the object onto which the projection pattern is irradiated are provided in Figures A1 and A2 of Appendix A of U.S. Provisional Application No. 63 / 070,066, which is incorporated herein by reference in its entirety.

[0026] Figure 2 is a block diagram of a 3D scanner 200 according to several embodiments. The 3D scanner 200, or the computer system of the 3D scanner 200, typically includes memory 204, one or more processors 202, a power supply 206, a user input / output (I / O) subsystem 208, one or more sensors 203 (including, for example, one or more cameras 112 in Figures 1A to 1B), a projector 110, and a communication bus 210 for interconnecting these components. The (one or more) processors 202 execute modules, programs, and / or instructions stored in memory 204, thereby performing processing operations.

[0027] In some embodiments, the (one or more) processor 202 includes at least one central processing unit. In some embodiments, the (one or more) processor 202 includes at least one image processing unit. In some embodiments, the (one or more) processor 202 includes at least one neural processing unit (NPU) for running the neural networks described herein. In some embodiments, the (one or more) processor 202 includes at least one field-programmable gate array.

[0028] In some embodiments, memory 204 stores one or more programs (e.g., multiple sets of instructions) and / or data structures. In some embodiments, memory 204, or the non-transient computer-readable storage medium of memory 204, stores, or subsets or supersets thereof, the following programs, modules, and data structures: • Procedures for handling various basic system services and performing hardware-dependent tasks. Operating System 212, including the procedure • One or more network communication modules 218 for connecting the 3D scanner to other computer systems (e.g., remote devices 236) via one or more communication networks 250. A user interface module 220 receives commands and / or inputs from the user via the user input / output (I / O) subsystem 208 and provides outputs for presentation and / or display on the user input / output (I / O) subsystem 208. A data processing module 224 for processing or preprocessing data from sensor 203, including optionally performing any or all of the operations described with respect to Method 500 (Figures 5A-5C) and / or Method 700 (Figure 7). Alternatively, in various embodiments, any or all of the data processing may be performed by a remote device 236 to which the 3D scanner 200 is connected via a network 250. • Data acquisition module 226 for controlling camera, projector, and sensor readouts. Storage 230, including one or more buffers, RAM, ROM, and / or other memory, for storing data used or generated by the 3D scanner 200.

[0029] The modules identified above (e.g., data structures and / or programs including sets of instructions) do not need to be implemented as separate software programs, procedures, or modules; therefore, various subsets of these modules may be combined or rearranged in various embodiments. In some embodiments, memory 204 stores a subset of the modules identified above. Furthermore, memory 204 may store additional modules not described above. In some embodiments, modules stored in memory 204, or modules stored in the non-transient computer-readable storage medium of memory 204, provide instructions for implementing the corresponding operations in the manner described below. In some embodiments, some or all of these modules may be implemented by special hardware circuits (e.g., FPGAs) that encompass some or all of the module functions. One or more of the elements identified above may be executed by one or more of the (one or more) processors 202.

[0030] In some embodiments, the user input / output (I / O) subsystem 208 connects the 3D scanner 200 communicably to one or more devices, such as one or more remote devices 236, via a communication network 250 and / or via wired or wireless connections. In some embodiments, the communication network 250 is the Internet. In some embodiments, the user input / output (I / O) subsystem 208 connects the 3D scanner 200 communicably to one or more integrated devices or peripheral devices, such as touch-sensitive displays.

[0031] In some embodiments, the projector 110 includes one or more lasers. In some embodiments, one or more lasers include vertical-cavity surface-emitting lasers (VCSELs). In some embodiments, the projector 110 also includes an array of multiple light-emitting diodes (LEDs) that produce visible light. In some embodiments, instead of lasers, the projector 110 includes a flash bulb or several other light sources.

[0032] The communication bus 210 optionally includes circuits (sometimes referred to as chipsets) that interconnect and control communication between system components.

[0033] Figure 3 shows, in several embodiments, a 3D scanner 200 via network 250. This is a block diagram of the connected remote device 236. The remote device 236 typically includes memory 304, one or more processors 302, a power supply 306, a user input / output (I / O) subsystem 308, and a communication bus 310 for interconnecting these components. The (one or more) processors 302 execute modules, programs, and / or instructions stored in memory 304, thereby performing processing operations.

[0034] In some embodiments, the (one or more) processor 302 includes at least one central processing unit. In some embodiments, the (one or more) processor 302 includes at least one image processing unit. In some embodiments, the (one or more) processor 302 includes at least one neural processing unit (NPU) for running the neural networks described herein. In some embodiments, the (one or more) processor 302 includes at least one field-programmable gate array.

[0035] In some embodiments, memory 304 stores one or more programs (e.g., multiple sets of instructions) and / or data structures. In some embodiments, memory 304, or the non-transient computer-readable storage medium of memory 304, stores, or subsets or supersets thereof, the following programs, modules, and data structures: • Operating system 312, which includes procedures for handling various basic system services and procedures for performing hardware-dependent tasks. • One or more network communication modules 318 for connecting the remote device 236 to other computer systems (e.g., a 3D scanner 200) via one or more communication networks 250. A user interface module 320 that receives commands and / or input from the user via the user input / output (I / O) subsystem 308 and provides outputs for presentation and / or display on the user input / output (I / O) subsystem 308. A data processing module 324 for processing data from the 3D scanner 200, including optionally performing any or all of the operations described with respect to Method 500 (Figures 5A to 5C) and / or Method 700 (Figure 7). In some embodiments, the data processing module 324 includes a neural network module 340 (for determining the correspondence between images and projection patterns) and a triangulation module 344 for executing a triangulation algorithm to determine the spatial coordinates of an object using the correspondence determined by the neural network module 340. In some embodiments, the neural network module 340 includes instructions for executing neural networks 340-a and 340-b, which are described below with reference to Figure 4. Neural network training module 328 for training a neural network to determine the correspondence between imaged elements in an imaged pattern and projected elements in a projection pattern, including optionally performing any or all of the operations described with respect to method 600 (Figures 6A-6B) or the operations described with respect to alternative methods referred to herein. Storage 330, including one or more buffers, RAM, ROM, and / or other memory, for storing data used or generated by the remote device 236.

[0036] The multiple modules identified above (for example, a program including data structures and / or sets of instructions) do not need to be implemented as separate software programs, procedures, or modules; therefore, various parts of these modules The sets may be combined or rearranged in various embodiments. In some embodiments, memory 304 stores a subset of the modules identified above. Furthermore, memory 304 may store additional modules not described above. In some embodiments, modules stored in memory 304, or modules stored in the non-transient computer-readable storage medium of memory 304, provide instructions for implementing the corresponding operations in the manner described below. In some embodiments, some or all of these modules may be implemented by special hardware circuitry (e.g., FPGA) that encompasses some or all of the module functions. One or more of the elements identified above may be executed by one or more of the (one or more) processors 302.

[0037] In some embodiments, the user input / output (I / O) subsystem 308 connects the remote device 236 to one or more devices, such as one or more 3D scanners 200 or external displays, via a communication network 250 and / or via wired or wireless connections. In some embodiments, the communication network 250 is the Internet. In some embodiments, the user input / output (I / O) subsystem 308 connects the remote device 236 to one or more integrated devices or peripheral devices, such as touch-sensitive displays.

[0038] The communication bus 310 optionally includes circuits (sometimes referred to as chipsets) that interconnect and control communication between system components.

[0039] Figure 4 shows the inputs and outputs of the neural network 340-a in several embodiments. As mentioned above, one problem that arises in the context of structured optical approaches for 3D scanning is the ambiguity of elements imaged on the surface of an object (for example, it is necessary to know which elements in the image correspond to which elements on the slide). According to some embodiments, this problem is solved by using a neural network trained to determine the correspondence between projected elements and elements imaged on the surface of an object (or, more generally, the correspondence between image pixels and their corresponding coordinates in the projection pattern). For this purpose, an imaged pattern 132 (described with reference to Figure 1C) is provided as input to the neural network 340 (the imaged pattern 132 is an image of the surface of an object when the projection pattern is projected onto the surface of the object). In some embodiments, the neural network 340 outputs a correspondence 402. In some embodiments, the correspondence 402 is an "image" having the same number of pixels as the imaged pattern 132. For example, in some embodiments, the input to the neural network 340 is a 9-megapixel photograph of the surface of an object when the projection pattern is projected onto the surface, and the output of the neural network is a 9-megapixel output "image" in which each pixel corresponds one-to-one with the input image. That is, each pixel in the "image" that forms the correspondence relationship 402 is a value that represents the correspondence between what is imaged in the imaged pattern 132 and the projection pattern. For example, the value "4" in the correspondence relationship 402 indicates that those pixels in the imaged pattern 132 correspond to the coordinate (or line) that has the value "4" in the projection pattern (note that the range of values ​​in the coordinate system of the projection pattern is arbitrary and may be in the range of 0 to 1, 0 to 10, or other ranges).

[0040] In some embodiments, the neural network 340-a receives additional input. For example, the neural network receives information about projection patterns.

[0041] In some embodiments, neural network 340-a outputs a rough value regarding the correspondence, and neural network 340-b outputs a fine value regarding the correspondence. In some embodiments, neural network 340-b operates in a similar manner to neural network 340-a, except that neural network 340-b receives an image of the surface of the object as input, as well as the output of neural network 340-a (for example, neural networks 340-a and 340-b are connected in series). Note that any number of neural networks can be used in any configuration. For example, in some embodiments, three or four neural networks are used, some of which are arranged in a series configuration, and some are arranged to operate independently.

[0042] Figures 5A to 5C show flowcharts relating to a method 500 for providing 3D reconstruction from a 3D imaging environment 100, according to several embodiments. The method 500 is performed on a computing device having (one or more) processors and memory storing (one or more) programs configured for execution by (one or more) processors. In some embodiments, the method 500 is performed on a computing device communicating with a 3D scanner 200. In some embodiments, certain operations of the method 500 are performed by a computing device different from the 3D scanner 200 (e.g., a computer system that receives and / or transmits to the 3D scanner 200). In some embodiments, certain operations of the method 500 are performed by a computing device storing (one or more) neural networks 340, such that the trained (one or more) neural networks 340 can be used as part of a 3D reconstruction based on images captured by the 3D scanner 200. Some operations in method 500 are combined at will, and / or the order of some operations is changed at will.

[0043] In various embodiments, Method 500 may include any of the features or operations of Method 700, provided that the features or operations of Method 700 described below do not conflict with Method 500 described below. For brevity, some of those details described with reference to Method 700 will not be repeated here.

[0044] Method 500 includes (510) acquiring an image of an object (e.g., object 120 shown in Figures 1A and 1B). The image includes a plurality of imaged elements 142 of an imaged pattern 132. The imaged pattern 132 corresponds to a projection pattern 130 projected onto the surface of object 120. The projection pattern 130 includes a plurality of projection elements 140. Method 500 also includes (520) outputting the correspondence between the plurality of imaged elements 142 and the plurality of projection elements 140 by using a neural network (e.g., neural network 340-a). In some embodiments, the neural network outputs the correspondence directly (e.g., at least a plurality of nodes in the output layer of the neural network have a one-to-one correspondence with at least a plurality of nodes in the input layer). In some embodiments, the neural network is a classification neural network (for example, classifying each pixel of an input image according to its correspondence to a projection pattern).

[0045] It should be noted that conventional neural networks are trained to recognize different instances of the same thing. For example, a neural network can be trained to recognize human-written characters using examples of human-written characters. In contrast, according to the embodiments described herein, even if the training data does not contain other instances of the object, between the projected element and the imaged element on the surface of the object, It has been found that neural networks can be trained to determine correspondences. For example, by training a neural network with data from objects that have a wide variety of features, even if the training data does not include skulls of that species, the neural network can be used to determine the correspondences of elements when scanning skulls of previously undiscovered extinct whale species.

[0046] The complex geometric shapes of objects (e.g., narrow features, sharp edges, deep grooves, etc.) exacerbate the difficulty in determining elemental correspondences. Herein, we have additionally found that by using a trained neural network, image resolution and completeness are improved, particularly in the presence of "sharp" features within the object.

[0047] In some embodiments, method 500 includes inputting values ​​for each corresponding pixel of the image of object 120 onto the corresponding nodes in the input layer of a neural network (e.g., neural network 304-a) (522).

[0048] In some embodiments, each corresponding pixel in the image of object 120 corresponds to each corresponding node in the output layer of the neural network (524). The values ​​for each corresponding node in the output layer of the neural network represent the correspondence between the corresponding pixel and the multiple projection elements 140 of the projection pattern 130 (for example, the values ​​represent coordinates on the projection pattern).

[0049] In some embodiments, the output layer of the neural network has the same size as the image of the object 120 (526) (for example, the neural network outputs an “image” having the same number of pixels as the input image, as described with reference to Figure 4).

[0050] In some embodiments, the output layer of the neural network is smaller than the image size (528). In some embodiments, the output layer of the neural network is larger than the image size.

[0051] In some embodiments, method 500 includes inputting information about the projection pattern 130 into the input layer of the neural network (530).

[0052] In some embodiments, the multiple projection elements 140 of the projection pattern 130 projected onto the surface of the object 120 include uncoded elements (e.g., any of the projection patterns having uncoded elements described herein) (532). In some embodiments, the multiple projection elements 140 of the projection pattern 130 projected onto the surface of the object 120 include multiple lines.

[0053] In some embodiments, the neural network is trained using simulated data (535). The simulated data includes a plurality of simulated images, each of which includes a simulated pattern containing a plurality of simulated elements. Each of the plurality of simulated elements corresponds to a corresponding projection element of a plurality of projection elements projected onto the surface of a corresponding simulated object. Each of the plurality of simulated images also includes correspondence data showing the correspondence between the plurality of simulated elements of the simulated image and the plurality of projection elements of the projection pattern.

[0054] In some embodiments, each of the multiple simulated images includes texture information relating to the corresponding simulated object (536).

[0055] In some embodiments, the texture information for each corresponding simulated object is texture information other than the natural texture of the corresponding simulated object (538).

[0056] In some embodiments, the texture information for each corresponding simulated object includes features similar to the multiple projection elements 140 of the projection pattern (540).

[0057] In some embodiments, the texture information for the corresponding simulated object includes text (542).

[0058] In some embodiments, the texture information for each corresponding simulated object includes multiple lines (544).

[0059] Operations 534-544 will be described in more detail below with respect to Method 600 (Figures 6A-6B). In other words, in some embodiments, the neural network used in Method 500 is trained using Method 600.

[0060] In some embodiments, multiple neural networks are used. The multiple neural networks may be connected in series or may operate independently of each other. As a non-limiting example of multiple networks connected in series, in some embodiments, the neural network is a first neural network (e.g., neural network 340-a), and method 500 further includes using a second neural network (e.g., neural network 340-b) to output an offset (e.g., a refinement thereof) from the correspondence determined by the first neural network between the multiple imaged elements 142 and the multiple projection elements 140 (550). Thus, in some embodiments, the resolution of the 3D reconstruction is increased by using two neural networks: (i) a first neural network that identifies the correspondence between the projection elements and the elements imaged on the surface of the object, and (ii) a second neural network that identifies the offset to the identified correspondence. In some embodiments, the second neural network directly outputs the offset (for example, at least several nodes in the output layer of the second neural network have a one-to-one correspondence with each pixel in the input image). The inventors have found that this two-step approach significantly and outstandingly improves the resolution of the resulting image.

[0061] Figures 6A and 6B show flowcharts relating to a method 600 for training (one or more) neural networks 340 according to several embodiments. The method 600 is performed on a computing device having (one or more) processors and memory storing (one or more) programs configured for execution by (one or more) processors (601). In some embodiments, the method 600 is performed on a computing device communicating with a 3D scanner 200. In some embodiments, certain operations of the method 600 are performed by a computing device different from the 3D scanner 200 (e.g., a computer system that receives and / or transmits to the 3D scanner 200). In some embodiments, certain operations of the method 600 are performed by a computing device that trains and stores (one or more) neural networks 340 so that the trained (one or more) neural networks 340 can be used as part of a 3D reconstruction based on images captured by the 3D scanner 200. Some operations in method 600 are combined at will, and / or the order of some operations is changed at will.

[0062] According to some embodiments, Method 600 uses simulated (also called "synthesized") data in which the spatial relationships between the projector, camera, and object are known with respect to each training image. One difficulty in training neural networks is obtaining the necessary "ground truth" for training. In many cases, hundreds of thousands of elements are projected onto the surface of an object. Existing algorithms for determining line correspondences suffer from the very same problem that the neural networks of this disclosure solve. Therefore, existing algorithms cannot be used to provide the ground truth for training such neural networks. Moreover, unlike image analysis, character recognition, and similar applications, tagging humans is not practical for 3D scanning / reconstruction applications and is just as error-prone as existing algorithms. These problems are solved by training the neural network using simulated data in which the precise correspondences and image acquisition geometry are known. In this way, training data can be generated for countless different object shapes and countless different geometry of the camera and projector relative to the object.

[0063] Method 600 includes generating simulated data (610). The simulated data includes i) a plurality of simulated images (for example, as shown in Figures A1 to A2 of Appendix A of U.S. Provisional Application No. 63 / 070,066), ii) object data (for example, as shown in Figures A3 to A6 of Appendix A of U.S. Provisional Application No. 63 / 070,066), and iii) correspondence data. The plurality of simulated images are images relating to a known projection pattern projected onto the surface of a simulated object. The projection pattern comprises a plurality of elements, and each image contains an imaged element of the simulated pattern. Each imaged element corresponds to the corresponding element in the plurality of elements of the known projection pattern. The object data includes data indicating the shape of each corresponding simulated object, and the correspondence data includes data indicating the correspondence between the imaged elements in the simulated images and the corresponding elements in the plurality of elements of the known projection pattern. Method 600 further includes training a neural network (620) to determine the correspondence between each corresponding element in a known projection pattern and an imaged element of the known projection pattern projected onto the surface of an actual object (e.g., imaged element 142 of imaged pattern 132) using simulated data. Method 600 also includes storing the trained neural network 340 (630) for use in reconstructing the image at a later point (e.g., for use in Method 500 as shown in Figures 5A-5C).

[0064] In some embodiments, the simulated data also includes texture (e.g., color) information relating to the simulated object (611). In some embodiments, multiple simulated images also include texture information relating to the corresponding simulated object. One difficulty in training a neural network to determine the correspondence between projected elements and elements imaged on the surface of an object is that the object itself has color, and that color is prone to changing across images of the object (e.g., because the color of the object itself changes, or due to lighting, shadows, etc.). This makes it difficult to identify patterns from the texture of the object itself. This problem is solved by using simulated training data with various textures and various reflectivity (in practice, increasing the difficulty of the task during the training phase makes the trained neural network more effective). In particular, the inventors have found that adding texture to simulated objects to include text, patterns, or other abrupt (high-contrast) texture features is especially effective in training a neural network to distinguish between the object's texture and its projected elements (for example, text, like projected patterns, involves large contrast fluctuations between light and dark).

[0065] In some embodiments, the texture information for each corresponding simulated object is texture information other than the natural texture of the corresponding simulated object (612).

[0066] In some embodiments, the texture information for each corresponding simulated object includes features similar to those of multiple elements in known projection patterns (613).

[0067] In some embodiments, the texture information for each corresponding simulated object includes text (614).

[0068] In some embodiments, the texture information for each corresponding simulated object includes multiple lines (615).

[0069] In some embodiments, each corresponding simulated object includes one or more sharp features (616).

[0070] In some embodiments, alternative methods are provided for training a neural network, which include generating simulated data having a plurality of simulated images relating to projection patterns projected onto the surface of a corresponding simulated object, data indicating the shape of the corresponding simulated object, and data indicating the correspondence between the corresponding pixels in the simulated images and the coordinates on the projection pattern. The alternative method further includes training a neural network to determine the correspondence between images and projection patterns using the simulated data. The alternative method further includes storing the trained neural network for use in reconstructing the images at a later point in time. Note that in some embodiments, the alternative method may share any of the feature points or operations of method 600 described above, as long as the feature points or operations of method 600 do not conflict with the alternative method for training a neural network.

[0071] Figure 7 shows a flowchart relating to a method 700 for providing 3D reconstruction from a 3D imaging environment 100, according to several embodiments. The method 700 is performed on a computing device having (one or more) processors and memory storing (one or more) programs configured for execution by (one or more) processors. In some embodiments, the method 700 is performed on a computing device communicating with a 3D scanner 200. In some embodiments, certain operations of the method 700 are performed by a computing device different from the 3D scanner 200 (e.g., a computer system that receives and / or transmits to the 3D scanner 200). In some embodiments, certain operations of the method 700 are performed by a computing device storing (one or more) neural networks 340, such that the trained (one or more) neural networks 340 can be used as part of a 3D reconstruction based on images captured by the 3D scanner 200. Some operations in the method 700 are optionally combined, and / or the order of some operations is optionally changed.

[0072] In various embodiments, Method 700 may include any of the features or operations of Method 500, provided that the features or operations of Method 500 described above do not conflict with Method 700. For the sake of brevity, some of those details described with reference to Method 500 will not be repeated here.

[0073] Method 700 includes acquiring an image of the object (702) while the projection pattern is illuminating the surface of the object. In some embodiments, the projection pattern is generated by transmitting light through a slide. In some embodiments, a coordinate system is associated with the projection pattern. The coordinate system describes the arrangement of each position of the projection pattern on the slide.

[0074] Method 700 further includes outputting (for example, with respect to a coordinate system) a correspondence between each corresponding pixel in an image and the coordinates of a projection pattern by using a neural network (704). To do so, in some embodiments, an image is provided to the input layer of a neural network (e.g., neural network 340-a). In some embodiments, the output layer of the neural network directly generates the corresponding coordinates (one or more) of each pixel in the projection pattern. For example, the neural network outputs an output image having the same number of pixels as the input image, in which case each pixel in the output image has a one-to-one correspondence with a pixel in the input image and holds a value for the coordinates (one or more) of that input pixel on the projection pattern. In this way, the output image is spatially correlated with the input image.

[0075] In some embodiments, the neural network is trained using method 600 as described above, or using an alternative method.

[0076] It should be noted that in some embodiments, the neural network outputs two coordinates (e.g., x and y coordinates on a slide pattern) for each pixel of the input image. Alternatively, in some embodiments, the neural network outputs only a single coordinate for each pixel of the input image. In such embodiments, other coordinates are known or can be inferred from the epipolar geometry of the scanner 200.

[0077] In some embodiments, projection pattern coordinates for each pixel of an input image may be determined by using multiple neural networks. For example, in some embodiments, a first neural network determines rough coordinates, while a second neural network determines finer coordinates (e.g., refinement of the coordinates from the first neural network). In various embodiments, the first and second neural networks may be arranged in series (e.g., the output of the first neural network is input into the second neural network), or the two neural networks may operate independently and their outputs may be combined. In various embodiments, three or more neural networks (e.g., four neural networks) may be used.

[0078] In some embodiments, the input image is a multi-channel image. As a non-limiting example, the input image may contain 240 × 320 pixels, but each pixel may store two or more values ​​(for example, three values ​​in the case of an RGB image). In some embodiments, additional channels are provided to input additional information into the neural network. Continuing with the non-limiting example, the input image would then have a size of 240 × 320 × n, where n is the number of channels. For example, in some embodiments, information about the projection pattern is input into the neural network as an additional "channel" for each image. In some embodiments, one or more of the multiple channels include information acquired when the projection pattern is not illuminating the surface of the object. For example, a grayscale image of a projection pattern projected onto the surface of an object may be superimposed on an RGB image acquired in close proximity to the grayscale image (e.g., within 200 milliseconds), where the RGB image was acquired when the projection pattern was not projected onto the surface of the object (recall that in some embodiments, the projection pattern is projected onto the surface of the object in a stroboscope-like manner).

[0079] In some embodiments, the output image is a multi-channel image. In some embodiments, one channel of the multi-channel output image provides the correspondences described above. Continuing with the non-limiting examples described above, each channel of the output image may contain 240 × 320 pixels. In this case, the output would have a size of 240 × 320 × m, where m is the number of channels. One of the multiple channels stores a value relating to the correspondence (e.g., a value relating to one or more coordinates on the projection pattern). In some embodiments, another channel in the output image stores a confidence value for each correspondence value for each pixel. The confidence value for each correspondence value for each pixel may be used in reconstruction (e.g., by weighting the data differently or by discarding data with too low a confidence value). In some embodiments, the output image may also include a channel describing the curvature of an object, a channel describing the texture of an object, or any other information that is spatially correlated with the input image.

[0080] Those skilled in the art will understand that the input and output images can be of any size. For example, in some embodiments, a 9-megapixel image (or any other size image) may be used rather than a 240 x 320 pixel image as described in the non-limiting example above.

[0081] It should be noted that conventional neural networks are trained to recognize different instances of the same thing. For example, a neural network can be trained to recognize human-written characters using examples of human-written characters. In contrast, according to the embodiments described herein, it has been found that a neural network can be trained to determine the correspondence between each corresponding pixel in an image and the coordinates of a projection pattern, even if the training data does not contain other instances of the object. For example, by training a neural network with data from objects having a wide variety of features, it is possible to determine the correspondence when scanning for skulls of previously undiscovered extinct whale species, even if the training data does not contain skulls of that species.

[0082] The complex geometric shapes of objects (e.g., narrow features, sharp edges, deep grooves, etc.) exacerbate the difficulty in determining correspondences. Herein, the inventors have further found that by using a trained neural network, the resolution and completeness of the images are improved, particularly in the presence of "sharp" features within the object. Examples of 3D reconstructed images using conventional methods and, further, the neural network according to the present invention, are provided in Figures A7–A18 of Appendix A of U.S. Provisional Application No. 63 / 070,066 (note that Figures A7–A9 are single-image reconstructions, while Figures A10–A18 are multi-image reconstructions). These reconstructed images exhibit significantly superior quality, including better resolution and better completeness with respect to images reconstructed according to the present invention.

[0083] Method 700 further includes reconstructing the shape of the surface of an object (706) (for example, using a triangulation algorithm) by using the correspondence between each corresponding pixel in the image and the coordinates of the projection pattern.

[0084] It should be understood that the specific order of operations described in Figures 5A-5B and 6A-6B is merely illustrative and not intended to indicate that the described order is the only possible order in which those operations can be performed. Those skilled in the art will recognize various ways of changing the order of operations described herein.

[0085] In the above description, specific embodiments were used for illustrative purposes. However, the above exemplary discussion is not intended to be exhaustive or to limit the invention to any specific form disclosed. Many modifications and variations are possible in view of the above teachings. The embodiments have been selected and described to best illustrate the principles of the invention and its practical applications, so that those skilled in the art may best use the invention and the various embodiments described, with various modifications to suit the specific intended use.

[0086] Furthermore, although terms such as "first," "second," etc., are used herein to describe various components in several examples, it will be understood that these components are not limited by these terms. These terms are used merely to distinguish one component from another. For example, without departing from the scope of the various embodiments described, a first neural network may be called a second neural network, and similarly, a second neural network may be called a first neural network. Although both the first and second neural networks are neural networks, they are not the same neural network unless the context clearly indicates otherwise.

[0087] With respect to the various embodiments described herein, the terms used in the description are solely for the purpose of describing a particular embodiment and are not intended to limit it. When used in the description of the various embodiments described herein and in the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural form unless the context clearly indicates otherwise. It will also be understood that the terms “and / or” as used herein refer to, and encompass, any combination and all possible combinations of one or more of the items listed in relation. It will also be understood that, when used herein, the terms “include,” “contain,” “have,” and / or “have,” while specifying the presence of the described feature points, completes, steps, operations, components, and / or ingredients, do not exclude the presence or addition of one or more other feature points, completes, steps, operations, components, ingredients, and / or groups thereof.

[0088] As used herein, the term "if" is optionally interpreted, depending on the context, to mean "when," "upon," "in response to a determination," or "in response to a detection." Similarly, the phrases "if a determination is made" or "[the described condition or event] is detected" are optionally interpreted, depending on the context, to mean "at the time a determination is made," "in response to a determination," "at the time "[the described condition or event] is detected," or "[the described condition or event] is detected."

Claims

1. It is a method, The method involves obtaining an image of an object, wherein the image includes multiple imaged elements of an imaged pattern. The imaged pattern corresponds to the projection pattern projected onto the surface of the object, The aforementioned projection pattern includes multiple projection elements, and the acquisition of this pattern is... By using a neural network, the correspondence between the multiple imaged elements and the multiple projection elements is output, A method comprising: reconstructing the shape of the surface of the object by using the correspondence between the plurality of imaged elements and the plurality of projection elements.

2. The aforementioned neural network is the first neural network, The aforementioned method, The method according to claim 1, further comprising using a second neural network to output an offset from the correspondence relationship determined by the first neural network between the plurality of imaged elements and the plurality of projection elements.

3. The method according to claim 1, wherein outputting the correspondence between the plurality of imaged elements and the plurality of projection elements by using the neural network includes inputting the values ​​for each corresponding pixel in the image of the object onto the corresponding nodes in the input layer of the neural network.

4. Each corresponding pixel in the image of the object corresponds to the corresponding node in the output layer of the neural network. The method according to claim 1, wherein the values ​​for each corresponding node within the output layer of the neural network represent the correspondence between each corresponding pixel and the plurality of projection elements of the projection pattern.

5. The method according to claim 4, wherein the output layer of the neural network has the same size as the image of the object.

6. The method according to claim 5, wherein the output layer of the neural network is smaller than the size of the image.

7. The method according to claim 6, wherein outputting the correspondence between the plurality of imaged elements and the plurality of projection elements by using the neural network includes inputting information about the projection pattern into the input layer of the neural network.

8. The method according to claim 1, wherein the plurality of projection elements of the projection pattern projected onto the surface of the object include non-encoded elements.

9. The method according to claim 8, wherein the plurality of projection elements of the projection pattern projected onto the surface of the object include lines.

10. The neural network is trained using simulated data, the simulated data includes a plurality of simulated images, each of the plurality of simulated images is A simulated image pattern containing multiple simulated elements, wherein the multiple simulated elements Each of the multiple projection elements projected onto the surface of the corresponding simulated object corresponds to a simulated imaged pattern, The method according to claim 1, further comprising data indicating the correspondence between a plurality of simulated elements of the simulated imaged pattern and the plurality of projection elements of the projection pattern.

11. The method according to claim 10, wherein, with respect to each of the plurality of simulated images, the corresponding simulated object includes texture information.

12. The method according to claim 11, wherein the texture information relating to each of the corresponding simulated objects is texture information other than the natural texture of each of the corresponding simulated objects.

13. The method according to claim 12, wherein the texture information relating to each of the corresponding simulated objects includes features similar to those of the plurality of projection elements of the projection pattern.

14. The method according to claim 11, wherein the texture information relating to each of the corresponding simulated objects includes text.

15. The method according to claim 11, wherein the texture information relating to each of the corresponding simulated objects includes lines.

16. A computer system comprising one or more processors and a memory storing instructions for performing the method according to any one of claims 1 to 15.

17. A non-transient computer-readable storage medium that stores instructions causing a computer system to perform the method described in any one of claims 1 to 15 when executed by the computer system.

18. It is a method, A computing device having one or more processors and memory storing one or more programs configured for execution by the one or more processors, When the projection pattern is irradiated onto the surface of the object, an image of the object is acquired, By using a neural network, the correspondence between each corresponding pixel in the image and the coordinates of the projection pattern is output, A method comprising: reconstructing the shape of the surface of the object by using the correspondence between each corresponding pixel in the image and the coordinates of the projection pattern.

19. A computer system comprising one or more processors and a memory storing instructions for performing the method according to claim 18.

20. A non-transient computer-readable storage medium that stores instructions causing a computer system to perform the method described in claim 18 when executed by the computer system.

21. It is a method, One or more processors, and execution by said one or more processors A computing device having memory that stores one or more programs configured for this purpose, This involves generating simulated data. Multiple simulated images relating to projection patterns projected onto the surface of the corresponding simulated object, The projection pattern includes a plurality of projection elements, each of the simulated images includes a simulated pattern having a plurality of simulated elements, and each of the plurality of simulated elements corresponds to a plurality of simulated images for each corresponding projection element of the projection pattern. The data showing the shape of each corresponding simulated object, To generate simulated data that includes data showing the correspondence between the simulated elements and their respective projection elements, The process involves training a neural network to determine the correspondence between the multiple projection elements of the projection pattern and the multiple simulation elements of the simulation image using the aforementioned simulated data. A method comprising storing the trained neural network for use in reconstructing the image at a later point in time.

22. The method according to claim 21, wherein each of the plurality of simulated images further includes texture information relating to the respective simulated object.

23. The method according to claim 22, wherein the texture information relating to each of the corresponding simulated objects is texture information other than the natural texture of each of the corresponding simulated objects.

24. The method according to claim 22, wherein the texture information relating to each of the corresponding simulated objects includes features similar to those of the plurality of projection elements of the projection pattern.

25. The method according to claim 22, wherein the texture information relating to each of the corresponding simulated objects includes text.

26. The method according to claim 22, wherein the texture information relating to each of the corresponding simulated objects includes lines.

27. The method according to claim 21, wherein each of the corresponding simulated objects includes one or more sharp features.

28. A computer system comprising one or more processors and a memory storing instructions for performing the method according to any one of claims 21 to 27.

29. A non-transient computer-readable storage medium that stores instructions causing a computer system to perform the method described in any one of claims 21 to 27 when executed by the computer system.

30. It is a method, A computing device having one or more processors and memory storing one or more programs configured for execution by the one or more processors, This involves generating simulated data. Multiple models relating to the projection patterns projected onto the surface of the corresponding simulated object. Pseudoimages and, The data showing the shape of each corresponding simulated object, To generate simulated data that includes data showing the correspondence between each corresponding pixel in the simulated image and the coordinates of the projection pattern, The process involves training a neural network to determine the correspondence between images and projection patterns using the aforementioned simulated data, A method comprising storing the trained neural network for use in reconstructing the image at a later point in time.

31. A computer system comprising one or more processors and a memory storing instructions for performing the method according to claim 30.

32. A non-transient computer-readable storage medium that stores instructions causing a computer system to perform the method described in claim 30 when executed by the computer system.

Images