Three-dimensional asset reconstruction

Abstract

A three-dimensional asset (3D) reconstruction technique for generating a 3D asset representing an object from an image of the object. Images are captured from different viewpoints in a darkroom using one or more light sources having known positions. The system estimates a camera pose for each of the captured images, and then constructs a 3D surface mesh consisting of surfaces using the captured images and their respective estimated camera poses. The texture attributes of each surface of the 3D surface mesh are then refined to generate a 3D asset.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. application serial number 17 / 942,796, filed on September 12, 2022, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] Examples set forth in this disclosure relate to generating three-dimensional (3D) assets. More specifically, but not by way of limitation, this disclosure describes an asset reconstruction technique that generates a virtual representation (3D asset) from a physical object. Background Art

[0004] Augmented reality (AR) shopping and try-on allows brands to enhance the user experience by bringing assets directly to users and allowing them to interact with the assets seamlessly. According to consumer testing, interacting with 3D assets results in more conversions and engagement than conventional catalog-based shopping. There are currently 3D asset generation engines on the market that allow users to quickly render and try on assets even on mobile and edge devices (e.g., LensCore by Snap, Inc., Filament by Google, etc.). One of the remaining problems is that brands typically do not produce 3D assets when launching new products. This is because creating such 3D assets requires time-consuming manual work by 3D artists, and each 3D asset reconstruction may take forty (40) hours or more, and multiple iterations with brand owners. In addition, conventional automated solutions cannot achieve production quality without the need for time-consuming post-processing and improvements. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The features of the various embodiments disclosed will be readily understood from the following detailed description with reference to the accompanying drawings. In the description and in the several views throughout the accompanying drawings, each element is referenced by a reference numeral. When there are multiple similar elements, a single reference numeral may be assigned to the similar element, with the additional lower case letters referring to the specific element. When referring to non-specific one or more elements, the lower case letters may be omitted.

[0006] Unless otherwise indicated, the various elements shown in the figures are not drawn to scale. For clarity, the size of the various elements may be enlarged or reduced. These figures depict one or more embodiments and are presented only as examples and should not be construed as limiting. The drawings include the following figures:

[0007] Figure 1 is a block diagram of a 3D asset reconstruction system;

[0008] Figure 2A It is a block diagram of the 3D asset reconstruction pipeline;

[0009] Figure 2B yes Figure 2A Illustration of the 3D asset reconstruction pipeline;

[0010] Figure 3A is a flow chart of the 3D asset reconstruction and rendering method;

[0011] Figure 3B is used for Figure 3A A flowchart of a texture determination method of an asset reconstruction and rendering method;

[0012] Figure 3C is used for Figure 3B A flowchart of a texture refinement method of a texture determination method;

[0013] Figure 4A is an image of a regular 3D asset;

[0014] Figure 4B is an image of a 3D asset reconstructed according to a 3D asset reconstruction according to the systems and methods described herein; and

[0015] Figure 5 is a block diagram of an example configuration of a machine suitable for implementing a method of generating a 3D asset in accordance with the systems and methods described herein. DETAILED DESCRIPTION

[0016] A 3D asset reconstruction technique for generating a 3D asset representing a physical object from an image of the object. Images are captured from different viewpoints in a dark room using one or more light sources with known positions. The technique estimates a camera pose for each of the captured images and then constructs a 3D surface mesh consisting of surfaces using the captured images and their corresponding estimated camera poses. Texture properties (e.g., one or more of reflectivity, color, or roughness) for each surface of the 3D surface mesh are then refined to generate a 3D asset.

[0017] The following detailed description includes systems, methods, techniques, instruction sequences, and computer program products that illustrate the examples set forth in this disclosure. Many details and examples are included for the purpose of providing a thorough understanding of the disclosed subject matter and its related teachings. However, those skilled in the relevant art can understand how to apply the related teachings without these details. The various aspects of the disclosed subject matter are not limited to the specific devices, systems, and methods described, because the related teachings can be applied or practiced in various ways. The terms and nomenclature used herein are only for the purpose of describing specific aspects and are not intended to be restrictive. In general, well-known instruction instances, protocols, structures, and techniques are not necessarily shown in detail.

[0018] As used herein, the terms "connect," "connected," "coupled," and "coupled" refer to any logical, optical, physical, or electrical connection, including a link or the like, through which an electrical or magnetic signal generated or supplied by one system element is transferred to another coupled or connected system element. Unless otherwise specified, coupled or connected elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements, or communication media, one or more of which can modify, manipulate, or carry electrical signals. The term "on" means directly supported by an element, or indirectly supported by an element through another element that is integrated into or supported by the element.

[0019] Additional objects, advantages and novel features of the examples will be set forth in part in the following description, and in part will become apparent to those skilled in the art upon examination of the following and the accompanying drawings, or may be learned by production or operation of the examples. The objects and advantages of the subject matter may be realized or attained by the methods, tools and combinations particularly pointed out in the appended claims.

[0020] Reference will now be made in detail to the examples illustrated in the accompanying drawings and discussed below. Figure 1 , 2A , 2B, 3, 4B and 5 describe example 3D asset reconstruction systems and methods.

[0021] Figure 1 An example 3D asset reconstruction system 100 is depicted. System 100 includes a computing system 102 configured to obtain an image of an object 104 in a darkroom 106 using a camera system 108. Camera system 108 includes one or more cameras that capture images (raw images) of object 104 from different viewpoints (e.g., hundreds of viewpoints). Camera system 108 may include one or more cameras that can be moved to different locations within darkroom 106 to capture different viewpoints, multiple cameras that are statically placed within darkroom 106 to capture different viewpoints, or a combination thereof. Each camera may be a conventional camera capable of capturing high-resolution digital images. Computing system 102 may include machine 500 ( Figure 5 ) such as one or more processors.

[0022] The light source 110 illuminates the object and has a known position relative to each camera (or camera position) of the camera system 108. In one example, a corresponding light source 110 is coupled to each camera of the camera system 108, which helps determine the position of the light source 110 relative to the captured image because the light source will have a known relationship to the camera. As used herein, the term darkroom refers to a room where almost all of the light present in the room comes from one or more known source locations.

[0023] Figure 2A and2B An example pipeline 200 for 3D asset reconstruction is depicted, wherein Figure 2A Includes blocks of text corresponding to the functions performed during the rebuild, Figure 2B Images corresponding to those blocks of text that visually depict the functionality are included. Although pipeline 200 is described with reference to components of 3D asset reconstruction system 100, it will be apparent to one skilled in the art how to implement the pipeline using other systems described herein.

[0024] The camera system 108 captures an original photograph 202 of the object 104. The camera system captures multiple original photographs 202 (e.g., hundreds) from different viewpoints around the object 104. A camera pose 204 is determined for each original photograph 202. A conventional structure from motion (SfM) algorithm may be used to determine the camera pose. The computing system 102 then generates a surface mesh 206 for the object 104 using the original photographs 202 and the determined camera pose 204. The surface mesh 206 includes multiple surfaces (typically triangles) and may be generated using an SfM algorithm (e.g., an SfM for determining camera poses or another SfM). After generating the initial surface mesh 206, the computing system 102 applies a differential renderer 210 to the surface mesh 206, the camera pose 204, and the original photographs 202. The difference renderer 210 may be initialized with, for example, predefined texture values ​​(e.g., uniform texture values ​​208, where all texture attributes are set to the same value, such as "0" or "1", or non-uniform texture values ​​based on possible materials), pseudo-random texture values, texture values ​​including noise, or a combination thereof, and then refined by the difference renderer 210. In one example, the difference renderer 210 applies a conventional gradient descent algorithm to reduce the difference between the texture of the original photo 202 and the interim texture attributes associated with the surfaces constituting the surface mesh (starting from the initialized texture attributes) to generate learned texture attributes 212 for the corresponding surface of the surface mesh 206. In one example, the learned texture attributes 212 are stored as metadata associated with the corresponding surface of the surface mesh 206 to generate a 3D asset.

[0025] Figures 3A-3C Flowchart 300 / 320 / 340 depicts example steps for generating a 3D reconstruction asset of an object. These steps are described with reference to system 100 and pipeline 200, however, embodiments using other systems / pipelines will be apparent to those skilled in the art from the description herein. In addition, it should be understood that one or more of the steps described in the flowchart may be performed sequentially as shown, out of order, at least substantially simultaneously, or may be omitted depending on the implementation.

[0026] Flowchart 300 depicts steps for reconstructing a three-dimensional (3D) asset representing an object. At block 302, camera system 108 captures images 202 of object 104 (e.g., a shoe) from different viewpoints (e.g., 400 viewpoints). Camera system 108 may store images 202 (and metadata) in a memory (e.g., memory 504; Figure 5 ). The computing system 102 may obtain the image 202 directly from the camera system or via a memory. In an example, the image 202 is captured in the dark room 106. Capturing the image 202 in the dark room 106, where almost all of the light used to illuminate the object 104 comes from one or more known source locations, enhances the ability of the difference renderer 210 to determine surface texture properties of the surfaces in the surface mesh 206.

[0027] At block 304, the computing system 102 estimates the camera pose 204 for each of the captured images 202. In an example, the computing system 102 estimates the camera pose 204 by applying a SfM algorithm to the images 202. The SfM algorithm matches each image with neighboring images in an iterative manner to estimate the camera pose 204 for each captured image 202. For example, the SfM algorithm may extract features in a particular image, match the extracted features with extracted features in at least one other image to obtain image correspondences (e.g., a set of points in one image that can be identified as the same point in another image), and estimate the camera pose based on the matches and the associated geometric relationships of the feature combinations of one image with at least one other image that includes the features. Images that include the same (matching) set of features captured from different viewpoints will have different geometric relationships between the feature sets in each image due to their respective viewpoints. The SfM algorithm is able to use these geometric relationships to accurately determine the camera pose for each image by comparing the geometric relationships of the feature sets in the image with the geometric relationships of the feature sets in at least one other image. The camera pose 204 for each image may be stored in metadata for that image.

[0028] At block 306, the computing system 102 constructs the 3D surface mesh 206. The 3D surface mesh 206 includes a surface (e.g., a triangular surface). The computing system 102 constructs the 3D mesh 206 using the captured image 202 and its corresponding estimated camera pose 204. In an example, the computing system 102 constructs the 3D surface mesh 206 by applying a second SfM algorithm to the image 202 and its corresponding estimated camera pose 204. The SfM algorithm matches each image with adjacent images in an iterative manner to construct the 3D surface mesh 206 using the captured image 202 and its corresponding camera pose 204.

[0029] In an example, computing system 102 may additionally receive manual input from a user during or after block 306. For example, if upon visual inspection, it is apparent to the user that surface mesh 206 does not correctly model object 104, computing system 102 may receive manual input provided by the user to adjust vertices of one or more surfaces (e.g., triangles) to more accurately model object 104. The computing system may then correct 3D surface mesh 206 in response to the manual input.

[0030] At block 308 , computing system 102 refines texture properties of multiple surfaces of 3D surface mesh 206 to generate a 3D asset that includes learned texture properties 212 for surfaces of 3D surface mesh 206 . In an example, each texture property is stored as a separate parameter for each surface of 3D surface mesh 206 .

[0031] In an example, a difference renderer 210 is used to reconstruct texture properties from the captured image 202 to refine the texture properties. The difference renderer 210 enhances scene parameters (e.g., mesh vertices or textures) by minimizing the difference between an artificially rendered image and a real-world image. In an example, the difference is modeled by the mean square error (MSE) between the real image and the generated image, which is minimized via gradient descent. This is an iterative process in which the system (1) generates an image / scene from a known camera position, (2) calculates the MSE between the generated image / scene and the corresponding captured image, (3) calculates the gradient of the MSE from the scene parameters, and (4) updates the scene parameters (param) in a manner that can be roughly described as: param=param-alpha*gradient. Iterations are performed until the loss is minimized and the rendered image begins to look the same as the captured image.

[0032] Flowchart 320 depicts a method for Figure 3A Example steps of a texture determination method of an asset reconstruction and rendering method of the invention. At block 322, the computing system 102 models a scene based on the captured image 202. In the example, the computing system 102 models the scene including the object 104 using adjacent captured images 202 to generate a 3D surface mesh 206. At block 324, the computing system 102 initializes texture parameters for each surface of the 3D surface mesh 206. In the example, the computing system 102 initializes the texture parameters by setting one or more parameters of each surface to a predefined value, a pseudo-random value, a texture value including noise, or a combination thereof. At block 326, the computing system 102 refines the texture parameters, for example, using the differential renderer 210.

[0033] Flowchart 340 depicts a method for Figure 3BExample steps of a texture refinement method of a texture determination method of the texture determination method of the invention. At block 342, the computing system 102 generates a scene including an object (e.g., an image of one or more surfaces) using the captured image 202. At block 344, the computing system 102 generates an error value for a texture parameter. In the example, the computing system 102 generates the texture parameter error value by: (1) comparing each parameter of each pixel of the scene / one or more surfaces with the same parameter in a portion of the area corresponding to the pixel / surface in the captured image 202, and (2) calculating the difference therebetween (e.g., the difference for each parameter or the difference between the averages of multiple parameters). For example, the computing system 102 may calculate the mean square error (MSE) of the parameter between the pixel of the generated image and the corresponding area of ​​the one or more captured images. At block 346, the computing system 102 calculates a gradient value (i.e., a gradient of the MSE value) representing the generated error value using the scene parameters. At block 348 , the computing system 102 updates the texture properties of the surface (ie, to reduce the gradient values), and at block 350 , determines the difference between the error / gradient values ​​using the updated parameters before updating the error / gradient values.

[0034] At block 352, the computing system 102 checks whether the current iteration of refinement has improved the surface mesh 206 using the learned texture attributes 212. If the texture attributes have not resulted in an improvement (i.e., the determined difference between the texture attribute / value (or the average of multiple texture attributes) in the region of the captured image 202 and the corresponding surface of the 3D surface mesh 206 is greater than in the previous iteration; or if the determined difference is equal to or below a threshold), refinement ends at block 354. Alternatively, if the texture refinement has resulted in an improvement (i.e., the determined difference is less than in the previous iteration; or if the determined difference is above a threshold), another iteration is performed starting at block 342.

[0035] Return to reference Figure 3A At block 310, the computing system 102 renders the 3D asset including the learned texture attributes 212. The 3D asset may be stored in a memory of the computing system. In an example, the 3D asset is rendered by retrieving the 3D asset from the memory, thereby providing the 3D asset and parameters affecting the appearance of the texture (e.g., light intensity, light direction, color / hue of the light) to a 3D rendering algorithm. According to the example, the computing system 102 executing instructions for implementing the 3D rendering algorithm applies the parameters to the 3D asset and generates instructions for presenting an image of the 3D asset (as modified based on the parameters) to a user, for example, on a display visible to the user.

[0036] Figure 4A depicts a conventional 3D reconstruction asset 400 generated from a shoe, Figure 4BA 3D reconstructed asset 410 generated from the same shoe according to the new techniques described herein is depicted. The 3D reconstructed asset 410 includes greater and more realistic details relative to features of the shoe, for example, the laces 412 of the 3D reconstructed asset 410 have greater details than the laces 402 of the conventional 3D reconstructed asset 400. In addition, the 3D reconstructed asset 410 includes details (such as stitching 414) that are not present in conventional 3D reconstructed assets.

[0037] The techniques described herein may be used with one or more computing systems described herein, or with one or more other systems. For example, the various processes described herein may be implemented using hardware or software, or a combination of the two. For example, at least one of the processors, memories, storage devices, (one or more) output devices, (one or more) input devices, or communication connections discussed below may be at least a portion of one or more hardware components. A dedicated hardware logic component may be constructed to implement at least a portion of one or more techniques described herein. For example, but not limited to, such hardware logic components may include field programmable gate arrays (FPGAs), program application specific integrated circuits (ASICs), program application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like. The applications of the devices and systems that may include various aspects may include a wide range of electronic systems and computing systems. The techniques may be implemented using two or more specific interconnected hardware modules or devices with associated control and data signals that may communicate between and through the modules, or as part of an application specific integrated circuit. In addition, the techniques described herein may be implemented by a software program executable by a computing system. As an example, implementations may include distributed processing, component / object distributed processing, and parallel processing. Furthermore, the virtual computing system processing may be configured to implement one or more techniques or functions as described herein.

[0038] Figure 5 An example configuration of a machine 500 is shown that includes components that may be incorporated into a computing system 102 suitable for managing 3D asset reconstruction.

[0039] In particular, Figure 5A block diagram of an example of a machine 500 on which one or more configurations can be implemented is shown. In an alternative configuration, the machine 500 can operate as a standalone device or can be connected (e.g., networked) to other machines. In a networked deployment, the machine 500 can operate in a server-client network environment with the capabilities of a server machine, a client machine, or both. In an example, the machine 500 can act as a peer machine in a peer-to-peer (P2P) (or other distributed) network environment. In a sample configuration, the machine 500 can be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a smart phone, a network device, a server, a network router, a switch or a bridge, or any machine capable of executing instructions (sequentially or otherwise) specifying the actions to be taken by the machine. For example, the machine 500 can be used as a workstation, a front-end server, or a back-end server of a communication system. The machine 500 can implement the method described herein by running software for implementing features as described herein. Further, while a single machine 500 is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

[0040] As described herein, examples may include, or may operate on, a processor, logic or multiple components, modules or mechanisms (herein "modules"). A module is a tangible entity (e.g., hardware) that is capable of performing a specified operation and may be configured or arranged in some manner. In an example, a circuit may be arranged as a module in a specified manner (e.g., internally or relative to an external entity such as other circuits). In an example, all or part of one or more computing systems (e.g., independent, client or server computer systems) or one or more hardware processors may be configured by firmware or software (e.g., instructions, application parts or applications) as a module that operates to perform a specified operation. In an example, the software may reside on a machine-readable medium. When executed by the underlying hardware of the module, the software causes the hardware to perform the specified operation.

[0041] Thus, the term "module" is understood to include at least one of a tangible hardware or software entity, which is a physically constructed, specially configured (e.g., hardwired) or temporarily (e.g., temporarily) configured (e.g., programmed) entity to operate in a specified manner or perform part or all of any operations described herein. Considering examples where modules are temporarily configured, each module need not be instantiated at any one time. For example, where a module includes a general-purpose hardware processor configured using software, the general-purpose hardware processor can be configured as corresponding different modules at different times. The software can configure the hardware processor accordingly, for example, to constitute a particular module at one time instance and to constitute a different module at a different time instance.

[0042] The machine (e.g., computing system) 500 may include a hardware processor 502 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory 504, and a static memory 506, some or all of which may communicate with each other via an interconnect (e.g., a bus) 508. The machine 500 may also include a display unit 510 (shown as a video display), an alphanumeric input device 512 (e.g., a keyboard), and a user interface (UI) navigation device 514 (e.g., a mouse). In an example, the display unit 510, the input device 512, and the UI navigation device 514 may be a touch screen display. The machine 500 may additionally include a mass storage device (e.g., a drive unit) 516, a signal generating device 518 (e.g., a speaker), a network interface device 520, and one or more sensors 522. Example sensors 522 include one or more of a global positioning system (GPS) sensor, a compass, an accelerometer, a temperature sensor, a light sensor, a camera, a video camera, a sensor of a physical state or position, a pressure sensor, a fingerprint sensor, a retinal scanner, or other sensors. The machine 500 may include an output controller 524, such as a serial (e.g., Universal Serial Bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate with or control one or more peripheral devices (e.g., a printer, a card reader, etc.).

[0043] The mass storage device 516 may include a machine-readable medium 526 having stored thereon one or more data structure sets or instruction sets 528 (e.g., software) embodying or used by any one or more of the techniques or functionality described herein. The instructions 528 may also reside, completely or at least partially, within the main memory 504, within the static memory 506, or within the hardware processor 502 during execution thereof by the machine 500. In an example, one or any combination of the hardware processor 502, the main memory 504, the static memory 506, or the mass storage device 516 may constitute a machine-readable medium.

[0044] Although the machine-readable medium 526 is shown as a single medium, the term "machine-readable medium" may include a single medium or multiple media (e.g., a centralized or distributed database, or at least one of an associated cache and server) configured to store one or more instructions 528. The term "machine-readable medium" may include any medium capable of storing, encoding, or carrying instructions executed by the machine 500 and causing the machine 500 to perform any one or more of the techniques of the present disclosure, or any medium capable of storing, encoding, or carrying data structures used by or associated with these instructions. Non-limiting machine-readable medium examples may include solid-state memory and optical and magnetic media. Specific examples of machine-readable media may include: non-volatile memory, such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; random access memory (RAM); solid-state hard disks (SSD); and CD-ROM and DVD-ROM disks. In some examples, the machine-readable medium may include non-transitory machine-readable media. In some examples, machine-readable media may include machine-readable media that is not a transitory propagating signal.

[0045] The instructions 528 may also be sent or received over the communication network 532 using a transmission medium via the network interface device 520. The machine 500 may communicate with one or more other machines using any of a number of transmission protocols, such as frame relay, Internet Protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc. Example communication networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), a mobile telephone network (e.g., a cellular network), a plain old telephone (POTS) network, and a wireless data network (e.g., a wireless network known as a Wi-Fi network). The network interface device 520 may include one or more physical jacks (e.g., Ethernet, coaxial, or telephone jacks) or one or more antennas 530 to connect to a communication network 532. In an example, the network interface device 520 may include multiple antennas 530 to wirelessly communicate using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) technology. In some examples, the network interface device 520 may use multi-user MIMO technology for wireless communication.

[0046] The features and flowcharts described herein may be embodied in one or more methods as method steps, or in one or more applications as described above. According to some configurations, "one or more applications" are (one or more) programs that execute the functions defined in the program. Various programming languages ​​may be employed to generate one or more applications structured in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In a specific example, a third-party application (e.g., an application created by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Applications developed with a software development kit (SDK) can be run on mobile operating systems such as IOS TM ANDROID TM , The present invention relates to mobile software on an Android phone or other mobile operating system). In this example, the third-party application can call API calls provided by the operating system to facilitate the functions described in this article. The application can be stored in any type of computer-readable medium or computer storage device and executed by one or more general-purpose computers. In addition, the methods and processes disclosed herein can alternatively be embodied in dedicated computer hardware or an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a complex programmable logic device (CPLD).

[0047] The program aspects of the technology can be considered as "products" or "articles of manufacture", typically carried on or embodied in a type of machine-readable medium in the form of at least one of executable code or associated data. For example, the programming code may include code for a touch sensor or other functions described herein. "Storage" type media include any or all tangible memories of a computer, processor, or the like or its associated modules, such as various semiconductor memories, tape drives, disk drives, and the like, which can provide non-transitory storage for software programming at any time. All or part of the software can sometimes communicate over the Internet or various other telecommunications networks. For example, such communication can enable software to be loaded from one computer or processor to another, such as from a server system or a service provider's host computer to a computer platform of a smart watch or other portable electronic device. Therefore, another type of medium that can carry programming, media content, or metadata files includes optical waves, radio waves, and electromagnetic waves, such as across physical interfaces between local devices, through wired and optical landline networks, and through various air links. Physical elements that carry such waves, such as wired or wireless links, optical links, or the like, can also be considered as media that carry software. As used herein, unless limited to "non-transitory," "tangible," or "storage" media, terms such as computer or machine "readable media" refer to any medium that participates in providing instructions or data to a processor for execution.

[0048] Therefore, machine-readable media can take the form of tangible storage media in a variety of forms. Non-volatile storage media include, for example, optical or magnetic disks, such as any storage device in any (one or more) computers or the like, such as client devices, media gateways, transcoders, etc. shown in the implementation drawings can be used. Volatile storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media include coaxial cables; copper wires and optical fibers, including wires that contain buses within a computing system. Carrier transmission media can take the form of electrical or electromagnetic signals, or the form of sound or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Therefore, common forms of computer-readable media include, for example: floppy disks, floppy disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punched card tapes, any other physical storage media with hole patterns, RAMs, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cassettes, carriers that transmit data or instructions, cables or links that transmit such carriers, or any other media from which a computer can read at least one programming code or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

[0049] The scope of protection is limited only by the claims that now follow. When interpreted in light of this specification and subsequent prosecution history, the scope is intended and should be interpreted to be consistent with the ordinary meaning of the language used in the claims and to include all structural and functional equivalents. Notwithstanding the foregoing, no claim is intended to include subject matter that does not satisfy the requirements of Sections 101, 102, or 103 of the Patent Act, nor should they be interpreted in such a manner. Any unintended inclusion of such subject matter is hereby disclaimed.

[0050] Except as stated immediately above, nothing that has been stated or shown is intended or should be construed as conferring any component, step, feature, object, benefit, advantage, or equivalent to the public, whether or not stated in the claims.

[0051] It will be understood that the terms and expressions used in this article have the common meanings given by such terms and expressions relative to their corresponding respective explorations and research fields, unless the specific meanings thereof have been set forth in addition herein. Relational terms (such as first and second and the like) can be used only to distinguish an entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "include", "comprise", "contain", "cover" or any other variation thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including or including a series of elements or steps not only include those elements or steps, but also may include other elements or steps that are not explicitly listed or inherent to such processes, methods, articles or devices. In the absence of further constraints, an element starting with "one" or "an" does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0052] In addition, in the foregoing detailed description, it can be seen that, for the purpose of streamlining the present disclosure, individual features are grouped together in various examples. This method of disclosure should not be interpreted as reflecting an intention that the claimed examples require more features than those explicitly stated in each claim. On the contrary, as reflected in the following claims, the subject matter to be protected lies in less than all the features of any single disclosed example. Therefore, the following claims are hereby incorporated into the detailed description, with each claim standing alone as a separate claimed subject matter.

[0053] Although what is considered to be the best mode and other examples have been described above, it is to be understood that various modifications may be made therein and that the subject matter disclosed herein may be implemented in various forms and examples and that they may be applied to numerous applications, only some of which have been described herein. The following claims are intended to claim any and all modifications and variations that fall within the true scope of the present concept.

Claims

1. An asset reconstruction system, comprising: at least one camera located within the darkroom, the at least one camera configured to capture images of an object positioned using the darkroom from a plurality of viewpoints; at least one light source configured to illuminate the object, each light source having a corresponding position; one or more computer processors; and a memory storing instructions that, when executed by the one or more computer processors, cause the asset reconstruction system to perform operations for generating a three-dimensional (3D) asset representing the object, the operations comprising: capturing, with the at least one camera, a set of images of the object from the plurality of viewpoints; estimating a camera pose for each corresponding image in the set of images; constructing a 3D surface mesh comprising a plurality of surfaces using the set of images and the camera pose estimated for each respective image; and Texture properties of the plurality of surfaces of the 3D surface mesh are optimized to generate the 3D asset.

2. The asset reconstruction system of claim 1, wherein: Estimating a camera pose for each corresponding image in the set of images includes: Extracting features from the corresponding images; matching features extracted from the corresponding image with features extracted from at least one other image of the set of images to obtain a set of matching features providing image correspondence; and A camera pose of the corresponding image is estimated based on the set of matching features and a geometric relationship of the set of matching features between the corresponding image and the at least one other image.

3. The asset reconstruction system of claim 2, wherein constructing the 3D surface mesh comprises: A structure from motion (SfM) algorithm is applied based on the set of images and the estimated camera pose for each respective image in the set of images.

4. The asset reconstruction system of claim 1, wherein the operations further comprise: Receiving a manual input instruction of the 3D surface mesh; and The 3D surface mesh is corrected in response to the manual input instruction. 5 . The asset reconstruction system of claim 1 , wherein the texture attributes include one or more of reflectivity, color, or roughness.

6. The asset reconstruction system of claim 5, wherein the operations further comprise: Each of the texture attributes is stored as a separate texture parameter.

7. The asset reconstruction system of claim 1 , wherein optimizing texture properties of a plurality of surfaces of the 3D surface mesh comprises: modeling a scene using the 3D surface mesh, the camera pose estimated for each corresponding image in the set of images, and the at least one light source; Initializing the texture attributes; and The texture properties are optimized using inverse rendering and parameters of the at least one light source until convergence.

8. The asset reconstruction system of claim 7, wherein initializing the texture attributes comprises: All pixels of the corresponding surface of the 3D surface mesh are set equal to a predefined value or a pseudo-random value.

9. The asset reconstruction system of claim 7, wherein: Optimizing the texture properties using inverse rendering and the parameters of the at least one light source until convergence includes: generating a portion of the scene from a known camera position; calculating a mean square error (MSE) between the portion of the scene and a corresponding portion of an image from the set of images; and A determination is made that a difference between the portion of the scene and the corresponding portion of the image is less than a threshold difference.

10. An asset reconstruction method for generating a three-dimensional (3D) asset representing an object, the method comprising: capturing a set of images of the object from a plurality of viewpoints in a dark room having at least one light source; estimating a camera pose for each corresponding image in the set of images; constructing a 3D surface mesh comprising a plurality of surfaces using the set of images and the camera pose estimated for each respective image; and Texture properties of a plurality of surfaces of the 3D surface mesh are optimized to generate the 3D asset.

11. The asset reconstruction method according to claim 10, wherein: Estimating a camera pose for each corresponding image in the set of images includes: Extracting features from the corresponding images; matching features extracted from the corresponding image with features extracted from at least one other image in the set of images to obtain a set of matching features providing image correspondence; and A camera pose of the corresponding image is estimated based on the set of matching features and a geometric relationship of the set of matching features between the corresponding image and the at least one other image.

12. The asset reconstruction method according to claim 11, wherein constructing the 3D surface mesh comprises: A structure from motion (SfM) algorithm is applied based on the set of images and the estimated camera pose for each respective image in the set of images.

13. The asset reconstruction method of claim 10, further comprising: receiving manual input instructions for the constructed 3D surface mesh; and The 3D surface mesh is corrected in response to the manual input instruction.

14. The asset reconstruction method of claim 10, wherein the texture attribute comprises one or more of reflectivity, color, or roughness.

15. The asset reconstruction method of claim 14, further comprising: Each of the texture attributes is stored as a separate texture parameter.

16. The asset reconstruction method of claim 10, wherein optimizing texture properties of a plurality of surfaces of the 3D surface mesh comprises: modeling a scene using the 3D surface mesh, the camera pose estimated for each corresponding image in the set of images, and the at least one light source; Initializing the texture attributes; and The texture properties are optimized using inverse rendering and parameters of the at least one light source until convergence.

17. The asset reconstruction method according to claim 16, wherein initializing the texture attribute comprises: All pixels of the corresponding surface of the 3D surface mesh are set equal to a predefined value or a pseudo-random value.

18. The asset reconstruction method of claim 16, wherein optimizing the texture properties using inverse rendering and parameters of the at least one light source until convergence comprises: generating a portion of the scene from a known camera position; calculating a mean square error (MSE) between the portion of the scene and a corresponding portion of an image from the set of images; and A determination is made that a difference between the portion of the scene and the corresponding portion of the image is less than a threshold.

19. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to generate a three-dimensional (3D) asset representing an object by performing operations comprising: capturing a set of images of the object from a plurality of viewpoints in a dark room having at least one light source; estimating a camera pose for each corresponding image in the set of images; constructing a 3D surface mesh comprising a plurality of surfaces using the set of images and the camera pose estimated for each respective image; and Texture properties of the plurality of surfaces of the 3D surface mesh are optimized to generate the 3D asset.

20. The non-transitory computer-readable storage medium of claim 19, wherein optimizing texture properties of the plurality of surfaces of the 3D mesh comprises: modeling a scene using the 3D surface mesh, the camera pose estimated for each corresponding image in the set of images, and the at least one light source; Initializing the texture attributes; and The texture properties are optimized using inverse rendering and parameters of the at least one light source until convergence.

Images