Monocular pose estimation and correction

Abstract

Various embodiments are directed to a pose correction engine ("engine"). The engine generates a reference image of an object of interest. The reference image depicts the object of interest in a first pose. The engine receives a source image of an instance of the object. The source image depicts the instance of the object in a pose that has changed from the first pose. The engine determines a disparity between the first pose of the reference image and the pose that has changed from the first pose of the source image. The engine identifies, based on the determined disparity, one or more portions of a three-dimensional (3D) map of the object's shape that are obscured by the pose that has changed from the first pose depicted in the source image. The engine generates a pose-corrected image of the instance of the object depicting at least a portion of the source image and at least one identified portion of the 3D map of the object's shape.

Description

[Background technology]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of Indian Patent Application No. 202121039730 filed on September 2, 2021, which is incorporated herein by reference in its entirety. background Various industries are experiencing an increase in popularity due to the rise in brands and products whose value has increased in the resale industry. The limited release of some types of special models of items makes such limited release items one of the most coveted items in the market. Due to the exclusivity of limited release items, counterfeit items are rampant in the market. Unlucky customers who purchase limited release items from one of the many markets or stores do not know whether the item they purchased is genuine or not. Although the market guarantees that the items are genuine, consumers are fooled many times. Even market and store owners are sometimes unsure about the authenticity of some items. This can cause a lack of trust in the trade and ultimately stifle the growth of various industries. Summary of the Invention

[0002] overview Various embodiments of the pose correction engine ("Engine") provide significant improvements and advantages over conventional systems by providing pre-processing of user source images for use in determining whether objects depicted in the user source images are fraudulent or counterfeit. For example, the Engine provides a user with the flexibility to acquire source images of objects in various predefined poses without requiring the user to acquire source images depicting the objects exactly in those predefined poses. A user may upload source images to the Engine, and the Engine outputs a pose corrected image based on the user's source images. The Engine outputs a pose corrected image that represents the source images as if the user had successfully captured the appearance of the object exactly in the predefined pose.

[0003] Various embodiments of the apparatus, methods, systems, and computer program products described herein are directed to a pose correction engine ("engine"). The engine generates a reference image of an object of interest ("object"), the reference image depicting the object of interest ("object") in a first pose. The engine receives a source image of an instance of the object, the source image depicting the instance of the object in a pose that has changed from the first pose. The engine determines disparities between the first pose of the reference image and the pose that has changed from the first pose of the source image based on the determined disparities. The engine identifies one or more portions of a three-dimensional (3D) map of the object's shape that are obscured by the pose that has changed from the first pose depicted in the source image. The engine generates a pose-corrected image of the object instance depicting at least a portion of the source image and at least the identified portion of the 3D map of the object's shape. It will be appreciated that in various embodiments, the object of interest may be any type of object.

[0004] In one embodiment, a user may wish to determine whether a particular shoe is authentic. The user obtains one or more source images of the shoe, each image depicting a particular perspective appearance of the shoe. The user uploads the one or more source images to an engine. The engine pre-processes the one or more source images in preparation for authenticating the particular shoe.

[0005] The engine has access to a reference image of a particular shoe, which may depict the particular shoe in a particular pose (i.e., position and orientation). The source image may depict the particular shoe in a pose that is generally similar to the particular pose of the reference image. In other words, the pose of the source image may not perfectly match the particular pose of the reference image. The engine pre-processes the source and reference images through segmentation, depth estimation, scaling, and registration steps to generate a pose-corrected image. The pose-corrected image output by the engine represents a version of the source image depicting the particular shoe with the particular pose of the reference image.

[0006] In various embodiments, the engine generates and stores multiple reference images for one or more types of objects of interest. For example, for a particular type of shoe, the engine generates multiple reference images of that particular type of shoe, each image depicting that particular type of shoe in a different pose (i.e., in a different position and orientation).

[0007] In one or more embodiments, the engine trains the machine learning network on the reference images during a training phase, which in various embodiments includes a feedback propagation loop.

[0008] In one embodiment, the scaling step performed by the engine includes applying one or more scale factors to the depth map image based on the reference image.

[0009] In various embodiments, the registration step performed by the engine generates one or more translation parameters and one or more rotation parameters. The engine applies the translation parameters in the rotation parameters to the source image to generate an attitude-corrected image.

[0010] In one or more embodiments, a user may place an object in a light box that places the object in a particular predefined pose. The light box may include one or more apertures for predefined fixed camera lens positions. The user may acquire one or more source images with a camera associated with the light box. The engine may preprocess a particular source image acquired with the light box to output a corresponding pose-corrected image. The engine may further utilize data from the alignment step of the particular source image to generate pose-corrected images of each of the other source images acquired with the light box.

[0011] Various embodiments include one or more modules and / or functions that redact confidential information / data, encrypt information / data, and anonymize data to ensure confidentiality and security of user and platform information / data as well as compliance with data privacy laws in the United States and / or international jurisdictions.

[0012] Further areas of applicability of the present disclosure will become apparent from the detailed description, claims and drawings. The detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. [Brief description of the drawings]

[0013] BRIEF DESCRIPTION OF THE DRAWINGS The present disclosure will become better understood from the detailed description and drawings.

[0014] [Figure 1A] FIG. 1A illustrates an exemplary environment in which an embodiment may operate.

[0015] [Figure 1B] FIG. 1B illustrates an exemplary environment in which an embodiment may operate.

[0016] [Figure 2A-2B] 2A and 2B each illustrate an exemplary environment in which an embodiment may operate.

[0017] [Figure 2C] FIG. 2C illustrates an exemplary method that may be implemented in an embodiment.

[0018] [Figure 2D] FIG. 2D illustrates an exemplary environment in which an embodiment may operate.

[0019] [Figure 2E] FIG. 2E illustrates an exemplary method that may be implemented in an embodiment.

[0020] [Diagram 3] FIG. 3 illustrates an exemplary environment in which an embodiment may operate.

[0021] [Figure 4] FIG. 4 illustrates an exemplary environment in which an embodiment may operate.

[0022] [Diagram 5] FIG. 5 illustrates an exemplary environment in which an embodiment may operate.

[0023] [Figure 6] FIG. 6 illustrates an exemplary environment in which an embodiment may operate.

[0024] [Figure 7A-7C] 7A, 7B, and 7C each illustrate an exemplary environment in which an embodiment may operate.

[0025] [Figure 8] FIG. 8 illustrates an exemplary environment in which an embodiment may operate.

[0026] [Figure 9] FIG. 9 illustrates an exemplary environment in which an embodiment may operate. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] Detailed Description DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Reference will now be made in detail to certain embodiments of the invention, some of which or aspects thereof are illustrated in the drawings.

[0028] Although the present invention has been described with reference to specific embodiments for clarity of explanation, it should be understood that the present invention is not limited to the described embodiments. On the contrary, the present invention is intended to cover alternatives, modifications, and equivalents as may be included within the scope defined by any claims. The following embodiments of the present invention are described without loss of generality and without imposing limitations on the claimed invention. In the following description, specific details are set forth in order to provide a thorough understanding of the present invention. The present invention may be practiced without some or all of these specific details. Additionally, well-known features may not be described in detail in order to avoid unnecessarily obscuring the present invention.

[0029] Additionally, it should be understood that the steps of the exemplary method described in this exemplary patent may be performed in a different order than presented herein, some steps of the exemplary method may be performed in parallel rather than sequentially, and the steps of the exemplary method may be performed in a networked environment where some steps are performed by different computers in the networked environment.

[0030] Some embodiments are implemented by a computer system, which may include a processor, a memory, and a non-transitory computer-readable medium. The memory and the non-transitory medium may store instructions for performing the methods and steps described herein.

[0031] A diagram of an exemplary network environment in which embodiments may operate is shown in FIG. 1A. The computer system 101 may include, for example, a smartphone, a smart device, a smart watch, a tablet, a desktop computer, a laptop computer, a notebook computer, a server, or other processing system. In some embodiments, the computer system 101 is mobile to accommodate a form factor that a user may have. In other embodiments, the computer system 101 is stationary. The computer system 101 may include a CPU 102 and a memory 103. The computer system 101 may include internal or external peripherals, such as a microphone 104 and a speaker 105. The computer system may also include an authentication application 110, which may include an image capture system 111 and a user interface (UI) system 112. For example, the image capture system may support functionality for displaying one or more graphic overlays via the UI 112.

[0032] The computer system 101 may be connected to a network 150. The network 150 may include, for example, a local network, an intranet, a wide area network, the Internet, a wireless network, a wired network, Wi-Fi, Bluetooth, one of a number of networks, or other networks. The network 150 may connect numerous computer systems and enable communication between the devices. The server 120 may be connected to the computer system 101 over the network 150. The server 115 may include a posture correction engine 120.

[0033] The environment 100 may be a cloud computing environment that includes remote servers or remote storage systems. Cloud computing refers to pooled network resources that can be quickly stockpiled to allow for easy scalability. Cloud computing can be used to provide software as a service, platform as a service, infrastructure as a service, and similar features. In a cloud computing environment, users can store files in the "cloud." "Cloud" means that the actual hardware storing the files is not visible to the user, but the files are stored on remote network resources.

[0034] 1B illustrates a block diagram of an example system 120 for a pose correction engine that includes a dissimilarity determination module 121, an identification module 122, a pose correction module 123, a machine learning network module 124, and a synthetic image generation module 125. The system 120 may be in communication with a user device 140 that transmits one or more source images.

[0035] Dissimilarity determination module 121 of system 120 may perform functions such as those shown in Figures 2A, 2B, 2C, 2D, 2E, 3, 4, 5, 6, 7A, 7B, 7C, and / or 8. In one embodiment, dissimilarity determination module 121 may perform functions related to determining dissimilarity between a reference image and a source image, where the reference image depicts an object in a first pose and the source image depicts the object in a second pose that is a changed pose from the first pose.

[0036] The identification module 122 of the system 120 may perform functions illustrated in Figures 2A, 2B, 2C, 2D, 2E, 3, 4, 5, 6, 7A, 7B, 7C, and / or 8. In one embodiment, the identification module 122 may perform functions related to identifying, based on the determined disparities, one or more portions of a three-dimensional (3D) map of the object's shape that are obscured by a pose that has changed from a first pose depicted in the source image.

[0037] Pose correction module 123 of system 120 may perform functions such as those shown in Figures 2A, 2B, 2C, 2D, 2E, 3, 4, 5, 6, 7A, 7B, 7C, and / or 8. In one embodiment, pose correction module 123 may perform functions related to generating a pose corrected image of the object that depicts at least a portion of the source image and at least one identified obscured portion of the object's shape.

[0038] The machine learning network module 124 of the system 120 may perform functions such as those shown in Figures 2A, 2B, 2C, 2D, 2E, 3, 4, 5, 6, 7A, 7B, 7C, and / or 8. In some embodiments, the machine learning network module 124 may perform functions related to training, updating, and running one or more types of machine learning networks.

[0039] Synthetic image generation module 125 of system 120 may perform functions such as those shown in Figures 2A, 2B, 2C, 2D, 2E, 3, 4, 5, 6, 7A, 7B, 7C, and / or 8. In some embodiments, synthetic image generation module 125 may perform functions related to synthetically generating multiple reference images of a particular type of object.

[0040] One or more software modules of the engine may be implemented on a computer system associated with a particular end user ("user"). As shown in flow chart 200 of FIG. 2A, the engine may present one or more graphic overlays to the user in a user interface (operation 202). One set of graphic overlays may correspond to predefined poses for a particular type of object, such as shoes. It will be appreciated that there may be multiple sets of graphic overlays for a particular type of object, with each separate graphic overlay representing a single particular predefined pose for the same type of object.

[0041] It will be further understood that the engine may have access to reference images depicting objects of the same type in each predefined pose. For example, a first type of shoe may have a first set of graphic overlays for a first predefined pose and a second set of graphic overlays for a second predefined pose. The first reference image depicts the same type of shoe in the first predefined pose and the second reference image depicts the same type of shoe in the second predefined pose.

[0042] The user attempts to match the camera viewpoint of the object to the displayed graphic overlay (operation 204). For example, one or more graphic overlays may be presented as visual guides on the user interface. The graphic overlays represent target display positions of various parts of a physical instance of a particular type of shoe as viewed through a camera function. The displayed graphic overlays serve as visual guides to help the user generate source images that depict the shoe instance in a pose that matches the predefined pose that corresponds to the displayed graphic overlay.

[0043] A user acquires one or more source images depicting the object at various object pose changes via a camera associated with the computer system (operation 206). Because the user acquires the source images in real time, the source images may conceivably depict the shoe in a pose that does not exactly match the predefined pose corresponding to the displayed graphic overlay. Thus, the pose depicted in the source images will necessarily be a pose that has changed from the predefined pose. The user submits the source images to the engine for preprocessing to generate a pose-corrected image. The pose-corrected image depicts content from the source images as if the user had perfectly aligned the graphic overlay to a portion of the shoe to generate a source image that perfectly matches the predefined pose.

[0044] According to various embodiments, the engine receives one or more source images for pre-processing. The source images may each depict the same particular shoe, but each source image may depict the same particular shoe from a different perspective appearance. It will be appreciated that the perspective appearance may be based on a particular position and orientation of the shoe, each particular position and orientation constituting a pose.

[0045] The identified reference images correspond to predefined poses similar to those depicted in the received source images. The engine identifies various reference images of the same particular shoe, each depicting the same particular shoe in a different predefined pose. For example, the source image may include metadata identifying a set of graphic overlays that were displayed to the user when the source image was captured. The engine may identify a reference image associated with the identified set of graphic overlays. The engine preprocesses one or more of the source images in various stages to output a pose-corrected image for each of the preprocessed source images.

[0046] As shown in diagram 208 of Figure 2B, the engine receives one or more source images 210, each depicting an object in a pose that varies from a predefined pose associated with a reference image 212. The engine identifies each reference image 212, each depicting the same type of object in a predefined pose. For each particular combination of source image 210 and reference image 212, the engine determines the differences between the paired images 210, 212 and generates a pose-corrected image 214.

[0047] As shown in diagram 216 of Figure 2C, pre-processing of a source image 210 by the engine may include one or more pre-processing stages. Upon receiving a source image 210, the engine inputs the source image 210 and a corresponding reference image 212 to a segmentation stage 28. The segmentation stage 218 outputs a segmented mask source image and a segmented mask reference image. The engine inputs the segmented mask source image and the segmented mask reference image to a depth estimation stage 220. The depth estimation stage 220 outputs a depth map source image and a depth map reference image.

[0048] The engine applies 222 one or more scale factors to the depth-mapped reference image to generate a scaled depth-mapped reference image. The engine inputs the scaled depth-mapped reference image and the depth-mapped source image to an alignment stage 224. The alignment stage 224 returns a perspective-corrected image as an output 226.

[0049] As shown in flow chart 230 of FIG. 2D, the engine receives a source image 232 depicting a shoe in a pose that has changed from a first pose. The engine identifies a reference image 234 depicting the same type of shoe (i.e., model, brand) in the first pose. The engine sends the images 232, 234 through pre-processing stages 218, 220, 222, 224, 226. The engine returns a pose-corrected image 236. The pose-corrected image 236 includes content from the received source image 232 and further includes content depicting various portions of the shoe that were obscured by the pose change in the source image 232.

[0050] As shown in flowchart 240 of FIG. 2E, the engine generates a reference image of the object of interest (operation 242). The reference image depicts the object in a first pose. In various embodiments, the engine generates multiple reference images of the object, each reference image depicting the object in a different pose. The engine generates the multiple reference images synthetically and / or based on a 3D scan of a physical instance of the object.

[0051] The engine receives a source image of an instance of the object (operation 244), the source image depicting the instance of the object in a pose that varies from a first pose. The engine identifies a reference image that corresponds to the received source image. For example, if the source image depicts a particular shoe model manufactured by a particular shoe manufacturer, the engine identifies a set of reference images depicting the same particular shoe model. The engine further accesses the identified set of reference images and selects a reference image depicting the shoe in a predefined pose. The source image was acquired during the display of a graphic overlay intended to guide a user to physically align a camera to generate a perspective view of the physical shoe in the predefined pose.

[0052] The engine determines disparities between the first pose of the reference image and the changed pose from the first pose of the source image (operation 246). The engine identifies one or more portions of a three-dimensional (3D) map of the shape of the object depicted by the source image that are obscured by the changed pose from the first pose based on the determined disparities (operation 248). The engine generates segmented mask images for both the source image and the identified corresponding reference image. The engine further generates a depth map based on the segmented mask images.

[0053] The engine generates a pose-corrected image of the object instance depicting at least a portion of the source image and at least the identified portion of the 3D map of the object's shape (operation 250). The engine performs global and local registration steps as described herein to generate a pose-corrected image for each received source image.

[0054] As shown in diagram 300 of FIG. 3, various embodiments described herein include collecting and generating various portions of training data. The engine collects a plurality of 3D models for various types of objects (operation 302). For example, the engine receives respective 3D models for various types of shoes. It will be appreciated that the respective 3D models may include 3D models for a plurality of different types (i.e., models) of shoes provided by a plurality of shoe manufacturers. The models may be, for example, shoe types referenced by unique SKU identifiers. Additionally, the engine receives a scan of an object (operation 304). For example, a scan of a physical instance of a particular type of shoe. In various embodiments, the engine receives a plurality of scans, each of which may represent a different type of shoe. Further, each scan may be converted into a corresponding 3D shoe model.

[0055] The engine receives input data (i.e., 3D model, multiple scans) and processes this input data to generate various images of poses of the object (operation 306). For example, given one or more 3D models and / or scans of a particular model of shoe, the engine synthetically generates one or more additional reference images of the same shoe model in different poses. It will be appreciated that a pose of an object depicts the object with a perspective view defined by a particular orientation of the object on the x-axis, y-axis, and z-axis in 3D space. For example, the engine synthetically generates multiple reference images of a particular type of athletic shoe, each reference image depicting a view of the same type of athletic shoe in a different pose. Thus, the engine may synthetically generate multiple reference images for a particular type of shoe, and further generate multiple reference images for other particular types of shoes. The engine may perform this process for any number of shoes. It will be appreciated that the various embodiments described herein are not limited to objects that are only different types of shoes. Instead, the object may be any type of physical item.

[0056] In various embodiments, the engine synthetically generates various reference images from the perspective of an emulated camera that serves as a reference point in 3D space for each pose. The engine may generate a first reference image of a first type of shoe. The engine may modify and / or manipulate the pose of the first reference image of the first type of shoe to generate additional reference images of the first type of shoe in different poses.

[0057] The engine sends one or more reference images to the machine learning network 308 as training data (operation 308). For each reference image used as training data, the reference image further includes a depth value for each pixel of the corresponding reference image. The pixel depth value represents the distance from a particular pixel in the reference image to the position of the emulated camera that is the basis of the perspective appearance of the object's pose in the reference image. For example, the pixel depth value represents the distance between a particular pixel in the reference image and the location of the emulated camera in 3D space with respect to the object's orientation (i.e., pose) depicted in the reference image and defined by the x-axis, y-axis, and z-axis. During the training phase, the machine learning network 308 outputs various predicted segmentation mask images 310 and various predicted depth mask images 314 as described in various embodiments herein. The training phase further includes an implementation of a backpropagation algorithm 312 that includes feedback of a loss function.

[0058] For a predicted segmented mask image output during the training phase resulting from an input training reference image, the feedback loss function provides an indication of a measure of classification error between portions of the predicted segmented mask image depicting the object and portions of the predicted mask image depicting the background surrounding the object. The measure of classification error may be determined by comparing the predicted segmented mask image with a classification ground truth provided by the input training reference image. For a predicted depth map image generated during the training phase resulting from an input training reference image, the feedback loss function compares the predicted depth with the ground truth depth of the input training reference image.

[0059] As shown in diagram 400 of Figure 4, the segmentation stage 218 performed by the engine receives an input source image 402 and sends the input source image 402 to an encoder / decoder network. The segmentation stage 218 outputs a segmented mask image 404 based on the source image 402. For example, the segmented mask image 404 is based on the content of the input source image 402, where the encoder / decoder network removes one or more portions of background content in the input source image 402 that do not contain content corresponding to an object of interest, such as a shoe.

[0060] Additionally, the engine, upon receipt of the input source image 402, identifies a corresponding reference image 406. For example, the corresponding reference image includes content depicting the same type of object at a predefined pose from which variations in pose are depicted in the input source image 402. The segmentation stage 218 further generates a segmented mask image 408 based on the reference image 406.

[0061] As shown in diagram 500 of FIG. 5, the depth estimation stage 220 performed by the engine receives the segmented mask source image 502 output from the segmentation stage 218 and inputs the segmented mask source image 502 to a depth estimation network. The depth estimation stage 2020 outputs a depth map source image 504 based on the input segmented mask source image 502. According to various embodiments, the depth estimation network determines the distance from each pixel in the input segmented mask image 502 to the emulated camera and generates a depth map source image 504 representing the predicted shape of the object depicted in the input segmented mask source image 502. The engine further inputs a corresponding segmented mask source image 506 to the depth estimation network. The depth estimation network also outputs a depth map for a reference image 508, which represents the predicted shape of the object depicted in the reference image.

[0062] As shown in the flowchart 600 of FIG. 6, during the scaling step 222, the engine identifies one or more scale factors 604 associated with the source image 402. For example, one or more of the scale factors 604 may be a focal length associated with the camera that captured the source image, a type of light sensor associated with the camera that captured the source image, and a predefined size measurement associated with an instance of an object depicted in the source image. The scaling step 222 applies the scale factors 604 to the depth map reference image 602 to correct and scale the image 602 to generate a scaled depth map reference image 606. By generating a scaled depth map reference image, the engine provides a reference image that is scaled to include one or more specifications of an image captured by the same camera that captured the source image currently being processed for pose correction.

[0063] As shown in flow chart 700 of FIG. 7A, during the alignment stage 224, a depth map source image 702 and a scaled depth map reference image 704 are received as input. The engine determines disparities between the images 702, 704 by aligning the respective portions of the images 702, 704. The engine identifies the occurrence of misalignment between the respective image portions that the engine attempted to align. The engine generates one or more pose correction parameters 706. For example, the pose correction parameters 706 can be one or more translation parameters and one or more rotation parameters.

[0064] The engine applies the pose correction parameters 706 to the corresponding original source images to generate the pose-corrected images 214. The application of the pose correction parameters 706 emulates a schematic rotation of various portions of the depiction of the object in the source images to generate additional object portions that are integrated with content from the source images, such that the integrated additional object portions create the visual appearance of the object in the source images as being precisely positioned at a particular predefined pose. Thus, the integrated additional object portions represent segments or regions of the object that were not depicted in the source images (i.e., obscured) but would have been depicted if the object depicted in the source images had been precisely positioned at the particular predefined pose.

[0065] As shown in the flowchart 708 of FIG. 7B, during the registration stage 224, the engine predicts a source 3D map of the object's shape based on the depth map source image 710 and predicts a reference 3D map of the object based on the scaled depth map reference image 712. The engine projects each of the source 3D map and the reference 3D map into 3D space and aligns the projected source 3D map and the reference 3D map to identify one or more differences. In various embodiments, the engine performs an iterative closest point algorithm to align the projected source 3D map and the reference 3D map. In other embodiments, the engine may perform a shift algorithm for the alignment. The misalignment may be, for example, one or more respective differences between portions of the source 3D map and corresponding portions of the reference 3D map. In various embodiments, the engine further identifies the misalignment due to color inconsistencies present in various portions of the non-overlapping source 3D map and the reference 3D map.

[0066] According to one or more embodiments, the engine generates source point cloud data 714 based on the depth map source image 710 and generates reference point cloud data 716 based on the scaled depth map reference image 712. It will be appreciated that the point cloud data 714, 716 may be based on a segmentation representative of the object depicted in the depth map images 710, 712. The engine attempts to align the point cloud data 714, 716 to identify differences between the point cloud data 714, 716. For example, the engine identifies misalignments 718 between the point cloud data 714, 716. In one or more embodiments, the identified misalignments 718 may be respective portions of the point cloud data 714 that do not overlap respective portions of the point cloud data 716. Such identified misalignments 718 (i.e., respective non-overlapping portions) are identified by the engine as differences for which pose correction parameters 706 are generated to correct.

[0067] As shown in flow chart 730 of FIG. 7C, during the registration step 224, the engine may receive a depth map source image 732 and a scaled depth map reference image 736, and may further segment a particular predefined portion of the object depicted in the images 732, 736. For example, if the object is a shoe, the heel tab shoe portion may be predefined as the registration portion to be analyzed during the registration step 224. The engine generates heel tab segmentation masks 734, 738 from both images 732, 736. In generating the segmentation masks 734, 738, the engine attempts to align the heel tab segmentation masks 734, 738 to identify differences between the heel tab segmentation masks 734, 738.

[0068] The engine generates pose correction parameters 706 for adjusting the pose of the heel tab in the source segmentation mask 734 to align with the pose of the heel tab depicted in the reference segmentation mask 738. The engine further applies the pose correction parameters 706 generated for the heel tab in the source segmentation mask 734 to various other portions of the corresponding source image to generate the pose-corrected image 214.

[0069] According to various embodiments, the registration step 224 aligns two or more images at different positions. The purpose of the registration step 224 is to geometrically align a source image (depicting an object in a pose that has changed from a predefined pose) with a reference image (depicting the same type of object in exactly the predefined pose). The engine uses a monocular depth estimation network to calculate the depth for both the source and reference images.

[0070] The predicted depth is obtained for the source image and the reference image, or the depth for the pose of the reference image may be obtained by a depth sensing sensor or a computer generated 3D model may be used. According to one or more embodiments, the depth is based on a distance value for each pixel in the image, the distance value representing a measurement of the distance from the respective pixel and a predefined position and orientation of a simulated camera that is represented as being outside the image.

[0071] The engine reprojects the predicted depths to obtain a 3D point cloud for each of the source and reference images. The engine executes one or more alignment algorithms to compute a transformation matrix for aligning the 3D point clouds. The engine may perform a two-stage alignment including a global alignment followed by a local alignment. Global alignment methods include algorithms that do not require a suitable (or appropriate) initialization measurement. Such global alignment methods produce alignment results with more error and are used as initialization for the local methods. Local alignment methods use the initial alignment to produce alignment methods with less error.

[0072] According to various embodiments, the engine performs global alignment with one or more of the following algorithms: scale invariant feature transform, fast point feature histogram, random sample consensus. For local alignment following global alignment, the engine performs one or more of the following algorithms: iterative closest point (ICP), colored iterative closest point.

[0073] In various embodiments, the engine performs the ICP algorithm by matching the closest point in the reference point cloud to each point in the source point cloud during the local registration. The engine then estimates a combination of rotation and translation parameters that minimizes a mean squared inter-point distance metric that optimizes (i.e., best aligns) the source point at each of the corresponding matching reference points identified in the previous closest matching step. The engine transforms the source points with a transformation based on the rotation and translation parameters.

[0074] In various embodiments, the engine performs a color ICP algorithm during local registration by segmenting the reference image to extract one or more segments and modifying the RGB (Red-Blue-Green) value of each pixel in the segment based on the average RGB value of the pixels in the source image. Alternatively, the engine utilizes a deep colorization network.

[0075] FIG. 8 shows a diagram 800 of a light box. According to various embodiments, the light box can be a physical structure in which any kind of object can be placed in various predefined physical positions and orientations. For example, a shoe can be placed in a fixed position on a platform 814 within the light box. The light box can have multiple imaging devices (e.g., cameras, smartphones, etc.) 802, 804, 806, 808, 810, 812, 814 at fixed or adjustable positions to capture one or more source images of various parts and portions of the shoe. In various embodiments, the source image of the shoe can be captured by a specific imaging device 802. Since the relative positions of all other imaging devices 804, 806, 808, 810, 812 are known, the engine can perform pose correction of source images associated with various defined poses that are different from the specific defined pose corresponding to a single source image, such as the single source image captured by the specific imaging 802. In other words, a first source image may depict a shoe in a first defined pose, and other source images may depict the same shoe in various different defined poses. During the alignment phase, the engine generates respective translation and rotation parameters for a transformation to generate a pose-corrected image for the first source image. In pre-processing other source images depicting the same shoe in various different defined poses, the engine does not need to generate additional translation and rotation parameters and does not need to transform each of the other source images. Rather, the engine utilizes the translation and rotation parameters generated for the first source image to further generate respective translation and rotation parameters for each of the other source images based on the source image fixed position.

[0076] For example, the imaging device 802 for a first source image may be defined as a reference device having reference coordinates based on the known position of the device 802 in a light box relative to the shoes and other devices 804, 806, 808, 810, 812 on the platform 814. As a result, a particular different device 804 may be defined as having relative coordinates relative to the reference coordinates of the reference device 802. The engine generates a reference transform for generating a posture-corrected image for the first source image from the reference device 802. To generate a transform to be used in generating a posture-corrected image for an image captured with the different device 804, the engine maps the reference transform used for the first source image to a relative transform corresponding to the image from the different device 804.

[0077] The engine calculates the change in orientation from a central position on the reference device 802 to a central position on the different device 804. The engine adjusts a reference transform for pose correction that derives the change in orientation from the central device position of the reference device 802 to the central device position of the different device 804. The resulting adjusted reference transform represents pose correction parameters for images acquired by the different device 804 that were generated without the need to perform global and local registrations for the images acquired by the different devices.

[0078] It will be appreciated that the various embodiments of the engine described herein may use any suitable machine learning training technique for training the machine learning network 130 for each sensor, including, but not limited to, artificial neural networks, neural net-based algorithms such as deep learning, robust linear regression algorithms such as random sample consensus, Huber regression, or Theil-Sen estimator, kernel-based approaches such as support vector machines and kernel ridge regression, tree-based algorithms such as classification and regression trees, random forests, extra trees, gradient boosting machines, or alternating model trees, naive Bayes classifiers, or other suitable machine learning algorithms.

[0079] 9 illustrates an example machine of a computer system in which a set of instructions may be executed to cause the machine to perform any one or more of the methodologies described herein. In another embodiment, the machine may be connected (e.g., networked) to other machines in a LAN, an intranet, an extranet, and / or the Internet. The machine may operate in the capacity of a server or a client machine in a client-server network environment, as a peer machine in a peer-to-peer (or distributed) network environment, or as a server or a client machine in a cloud computing infrastructure or environment.

[0080] The machine may be a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), mobile phone, web appliance, server, network router, switch or bridge, or other machine capable of executing a set of instructions (sequential or otherwise) that specify operations to be performed by the machine. Moreover, although a single machine is illustrated, the term "machine" should be interpreted to include a collection of machines that individually or together execute a set (or sets) of instructions that implement any one or more of the methods described herein.

[0081] The example computer system 900 includes a processing device 902, a main memory 904 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), etc.), a static memory 906 (e.g., flash memory, static random access memory (SRAM), etc.), and a data storage device 918, which are in communication with each other via a bus 930.

[0082] Processing device 902 represents one or more general-purpose processing devices, such as a microprocessor, a central processing unit, or the like. More specifically, processing device may be a complex instruction set computer (CISC) microprocessor, a reduced instruction set computer (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets, or a processor implementing a combination of instruction sets. Processing device 902 may also be one or more special-purpose processing devices, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, or the like. Processing device 902 is configured to execute instructions 926 to perform the operations and steps described herein.

[0083] The computer system 900 may further include a network interface device 908 for communicating over a network 920. The computer system 900 may also include a video display unit 910 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device 912 (e.g., a keyboard), a cursor control device 914 (e.g., a mouse), a graphics processing unit 922, a signal generation device 916 (e.g., a speaker), a video processing unit 928, and an audio processing unit 932.

[0084] The data storage device 918 may include a machine-readable storage medium 924 (also known as a computer-readable medium) on which is stored one or more set of instructions or software 926 that embody one or more of the methods or functions described herein. Additionally, the instructions 926 may reside, completely or at least partially, within the main memory 904 and / or within the processing device 902 during execution thereof by the computer system 900, the main memory 904 and the processing device 902 also constituting a machine-readable storage medium.

[0085] In one embodiment, instructions 926 include instructions for implementing functions corresponding to components of a device implementing the disclosure herein. Although machine-readable storage medium 924 is shown in the embodiment as being a single medium, the term "machine-readable storage medium" should be interpreted to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) that store one or more sets of instructions. The term "machine-readable storage medium" should also be interpreted to include any medium capable of storing or encoding a set of instructions for execution by a device that causes the device to perform one or more of the methods of the present disclosure. Thus, the term "machine-readable storage medium" should be interpreted to include, but is not limited to, solid-state memory, optical media, and magnetic media.

[0086] Some of the preceding detailed descriptions have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is herein, and generally, conceived to be a self-consistent sequence of operations leading to a desired result. These operations require physical manipulations of physical quantities. Usually, though not necessarily, these physical quantities take the form of electrical or magnetic signals capable of being stored, combined, compared and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0087] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these physical quantities. Unless otherwise stated, which is clear from the above discussion, descriptions throughout the specification using terms such as "identifying" or "determining" or "executing" or "performing" or "collecting" or "generating" or "transmitting" are understood to refer to operations and processes of a computer system or similar electronic computing device that manipulates data represented as physical (electronic) quantities in the registers and memory of the computer system and converts data into other data that are similarly represented as physical quantities in the memory or registers of the computer system or other such information storage devices.

[0088] The present disclosure also relates to an apparatus for performing the operations herein. The apparatus may be specially configured for the intended purposes. Alternatively, the apparatus may include a general purpose computer selectively activated or reconfigured by a computer program stored within the computer. Such computer programs may be stored on a computer readable storage medium, such as, but not limited to, any type of floppy disk, optical disk, CD-ROM, magneto-optical disk, read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic or optical card, or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus.

[0089] Various general-purpose systems may be used with programs in accordance with the teachings herein. Alternatively, it may prove convenient to construct more specialized apparatus to carry out the methods. The structure for these various systems is as set forth in the above description. In addition, the present disclosure has not been described with reference to any particular programming language. It will be understood that a variety of programming languages ​​can be used to implement the teachings of the disclosure as described herein.

[0090] The present disclosure may be provided as a computer program product, or software, that may include a machine-readable medium having instructions recorded thereon, which may be used to program a computer system (or other electronic device) to perform a process according to the present disclosure. A machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). For example, machine-readable (e.g., computer-readable) media includes machine (e.g., computer) readable media such as read-only memory ("ROM"), random access memory ("RAM"), magnetic disk storage media, optical storage media, flash memory devices, and the like.

[0091] In the foregoing disclosure, embodiments of the present disclosure have been described with reference to specific embodiments thereof. It will be apparent that various modifications may be made thereto without departing from the broader spirit and scope of the embodiments of the present disclosure as set forth in the following claims. Accordingly, the present disclosure and drawings are to be regarded in an illustrative and not a restrictive sense.

Claims

1. 1. A computer-implemented method comprising: determining disparities between a reference image depicting an object in a first pose and a source image depicting said object in a second pose, said second pose comprising a pose that has changed from said first pose; identifying one or more portions of a three-dimensional (3D) map of the object's shape that are obscured by the changed pose from the first pose depicted in the source image based on the determined differences; generating a pose-corrected image of the object depicting at least a portion of the source image and at least one identified obscured portion of the shape of the object; method.

2. 2. The computer-implemented method of claim 1, further comprising: Before determining said differences, generating a plurality of reference images of the object, each reference image depicting the object in a different pose; receiving the source image; identifying the reference image depicting the first pose relative to the source image; method.

3. 2. The computer-implemented method of claim 1, wherein determining a disparity between the first pose of the reference image and the pose changed from the first pose of the source image comprises: generating a segmented mask image of the reference image (a "segmented reference image") and a segmented mask image of the source image (a "segmented source image"); generating a depth map image based on the segmented reference image (a "depth map reference image") and a depth map image based on the segmented source image (a "depth map source image"); Including, method.

4. 4. The computer-implemented method of claim 3, further comprising: identifying one or more scale factors corresponding to the source images upon receiving the source images prior to determining the differences; modifying the depth map reference image by the identified one or more scale factors to generate a scaled depth map reference image during generation of the depth map reference image and the depth map source image. method.

5. 5. The computer-implemented method of claim 4, wherein the magnification factor is: (i) a focal length associated with a camera that captured the source image; (ii) the type of optical sensor associated with the camera that captured the source image; and (iii) predefined size measurements associated with instances of the object depicted in the source images; Including one or more of the following: method.

6. Identifying one or more portions of a three-dimensional (3D) map of the object's shape that are obscured by the changed pose from the first pose based on the determined disparities; predicting a source 3D map of the shape of the object based on the depth map source image; predicting a reference 3D map of the object based on the scaled depth map reference image; projecting each of the source 3D map and the reference 3D map into a 3D space; Aligning the projected source 3D map and the reference 3D map to identify one or more differences. wherein the one or more differences include a misregistration between a portion of the source 3D map and a corresponding portion of the reference 3D map.

5. The computer-implemented method of claim 4.

7. Aligning the projected source 3D map and the reference 3D map to identify one or more differences includes: determining at least one of a rotation parameter and a translation parameter based on the positional deviation; Applying each of the parameters to the portion of the source 3D map to generate a portion of the pose corrected image. Including, 7. The computer-implemented method of claim 6.

8. 1. A system including one or more processors and a non-transitory computer readable medium including one or more instructions that, when executed by the one or more processors, cause the system to perform an operation, the operation comprising: determining differences between a reference image depicting an object in a first pose and a source image depicting said object in a second pose, said second pose comprising a pose that has changed from said first pose; identifying one or more portions of a three-dimensional (3D) map of the object's shape that are obscured by the changed pose from the first pose depicted in the source image based on the determined differences; generating a pose-corrected image of the object depicting at least a portion of the source image and at least one identified obscured portion of the shape of the object; The system includes:

9. Before determining said differences, generating a plurality of reference images of the object, each reference image depicting the object in a different pose; receiving the source image; Identifying the reference image depicting the first pose relative to the source image. The system of claim 8 further comprising:

10. Determining a disparity between the first pose of the reference image and the pose changed from the first pose of the source image includes: generating a segmented mask image of the reference image (a "segmented reference image") and a segmented mask image of the source image (a "segmented source image"); generating a depth map image based on the segmented reference image (a "depth map reference image") and a depth map image based on the segmented source image (a "depth map source image"); The system of claim 8 , comprising:

11. identifying one or more scale factors corresponding to the source images upon receiving the source images prior to determining the differences; modifying the depth map reference image by the identified one or more scale factors to generate a scaled depth map reference image during generation of the depth map reference image and the depth map source image. The system of claim 10 further comprising:

12. The magnification is (i) a focal length associated with a camera that captured the source image; (ii) the type of optical sensor associated with the camera that captured the source image; and (iii) predefined size measurements associated with instances of the object depicted in the source images; The system of claim 11 , comprising one or more of:

13. Identifying one or more portions of a three-dimensional (3D) map of the object's shape that are obscured by the changed pose from the first pose based on the determined disparities; predicting a source 3D map of the shape of the object based on the depth map source image; predicting a reference 3D map of the object based on the scaled depth map reference image; projecting each of the source 3D map and the reference 3D map into a 3D space; Aligning the projected source 3D map and the reference 3D map to identify one or more differences. wherein the one or more differences include a misregistration between a portion of the source 3D map and a corresponding portion of the reference 3D map. The system of claim 11.

14. Aligning the projected source 3D map and the reference 3D map to identify one or more differences includes: determining at least one of a rotation parameter and a translation parameter based on the positional deviation; Applying each of the parameters to the portion of the source 3D map to generate a portion of the pose corrected image. The system of claim 13 , comprising:

15. 1. A computer program product comprising a non-transitory computer readable medium having computer readable program code embodied therein for execution by one or more processors, the program code comprising: determining disparities between a reference image depicting an object in a first pose and a source image depicting said object in a second pose, said second pose comprising a pose that has changed from said first pose; identifying one or more portions of a three-dimensional (3D) map of the object's shape that are obscured by the changed pose from the first pose depicted in the source image based on the determined differences; generating a pose-corrected image of the object depicting at least a portion of the source image and at least one identified obscured portion of the shape of the object; Includes instructions for Computer program products.

16. Before determining said differences, generating a plurality of reference images of the object, each reference image depicting the object in a different pose; receiving the source image; Identifying the reference image depicting the first pose relative to the source image.

16. The computer program product of claim 15, further comprising:

17. Determining a disparity between the first pose of the reference image and the pose changed from the first pose of the source image includes: generating a segmented mask image of the reference image (a "segmented reference image") and a segmented mask image of the source image (a "segmented source image"); generating a depth map image based on the segmented reference image (a "depth map reference image") and a depth map image based on the segmented source image (a "depth map source image"); 16. The computer program product of claim 15, comprising:

18. identifying one or more scale factors corresponding to the source images upon receiving the source images prior to determining the differences; modifying the depth map reference image by the identified one or more scale factors to generate a scaled depth map reference image during generation of the depth map reference image and the depth map source image.

20. The computer program product of claim 17, further comprising:

19. The magnification is (i) a focal length associated with a camera that captured the source image; (ii) the type of optical sensor associated with the camera that captured the source image; and (iii) predefined size measurements associated with instances of the object depicted in the source images; Including one or more of the following:

20. A computer program product according to claim 18.

20. Identifying one or more portions of a three-dimensional (3D) map of the object's shape that are obscured by the changed pose from the first pose based on the determined disparities; predicting a source 3D map of the shape of the object based on the depth map source image; predicting a reference 3D map of the object based on the scaled depth map reference image; projecting each of the source 3D map and the reference 3D map into a 3D space; Aligning the projected source 3D map and the reference 3D map to identify one or more differences. the one or more differences include a misalignment between portions of the source 3D map and corresponding portions of the reference 3D map, and aligning the projected source 3D map and the reference 3D map includes: determining at least one of a rotation parameter and a translation parameter based on the positional deviation; applying said respective parameters to said portion of said source 3D map to generate a portion of said pose corrected image.

20. The computer program product of claim 18, comprising:

Images