Multi-sensor capture for virtual try-on

The multi-sensor capture system addresses the inaccuracies in virtual try-on by using both cameras to generate a lightmap for realistic reflections, improving the virtual try-on experience and user confidence in remote purchases.

WO2024232857A1PCT designated stage expired Publication Date: 2024-11-14GOOGLE LLC
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
PCT/US2023/021169
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-05
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

Conventional virtual try-on techniques fail to accurately depict reflections on reflective surfaces of wearable items, leading to unrealistic composite images that may deter users from making purchases due to inaccuracies in lighting effects and surface representations.

Method used

A multi-sensor capture system using a front and rear camera to capture diffuse and specular reflections, generating a lightmap that accurately represents environmental lighting, enabling realistic rendering of virtual items on users, even with highly reflective surfaces.

Benefits of technology

The system provides highly accurate and realistic virtual try-on experiences by accurately rendering reflections from both diffuse and specular surfaces, enhancing user confidence in remote purchases and reducing dissatisfaction with physical items.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure US2023021169_14112024_PF_FP_ABST
    Figure US2023021169_14112024_PF_FP_ABST
Patent Text Reader

Abstract

Described techniques enable virtual try on (VTO) scenarios using multiple image sensors, such as a front camera and a rear camera of a smartphone. Accordingly, a simulated composite image may be generated in which a virtual wearable item is superimposed on a user in a realistic and accurate manner.
Need to check novelty before this filing date? Find Prior Art

Description

MULTI-SENSOR CAPTURE FOR VIRTUAL TRY-ONTECHNICAL FIELD

[0001] This description relates to simulation of a wearable item on a user.BACKGROUND

[0002] Virtual try-on generally refers to the use of augmented reality techniques to render and simulate virtual versions of glasses, earbuds, or other consumer items on an image of a user. For example, a consumer wishing to purchase a pair of eyeglasses or sunglasses may use a smartphone to view a selfie image while an image of glasses being considered for purchase is rendered on the consumer's face within the selfie image.

[0003] Accordingly, a consumer or other user may be provided with a rendered or simulated composite image to assist in an evaluation of the glasses, e g., for purchase. More particularly, the user may be provided with an ability to evaluate each of a plurality of glasses in context, and without requiring physical access to the glasses being considered. As a result, for example, users may be more likely to purchase items having the virtual try-on feature available, even when making remote purchases.SUMMARY

[0004] In a general aspect, a computer program product is tangibly embodied on a non-transitory computer-readable storage medium and comprises instructions. When executed by at least one computing device (i.e., by at least one processor of the computing device), the instructions are configured to cause the at least one computing device to capture an object image of an object and reflected light from at least one light source providing incident light onto the object, using a first image sensor on a first side of a device, and capture direct light from the at least one light source, using a second image sensor on a second side of the device. When executed by the at least one computing device, the instructions are configured to cause the at least one computing device to generate a lightmap based on the object image and the direct light, and generate a composite image of the object with a virtual item at least partially superimposed thereon. When executed by the at least one computing device, the instructions are configured to cause the at least one computing device to generate, within the composite image and based on the lightmap, virtual reflected light as being reflected from the virtual item.

[0005] A lightmap in the present disclosure generally refers to a data structure including calculated and stored data on brightness of surfaces of a virtual object, such as the virtual item. The data structure of the lightmap may thus allow for rendering reflections of the virtual item in the composite image taking into account the lighting conditions in a real-world environment of the object for which the object image is captured. The lightmap may for example be composed of lumels. The composite image may thus include the captured object image and the virtual item at least partially superimposed on the object image, wherein the virtual item is rendered with reflections based on a lightmap that was generated using both the object image and the direct light captured via different image sensors.

[0006] In another general aspect, a device includes at least one memory, at least one processor, at least one display, a first image sensor on a first side of the device and positioned to capture an object image of an object and reflected light from at least one light source providing incident light onto the object, and a second image sensor on a second side of the device and positioned to capture direct light from the at least one light source. A simulation engine includes instructions stored using the at least one memory, which, when executed by the at least one processor, causes the device to generate a lightmap based on the object image and the direct light, generate a composite image of the object with a virtual item at least partially superimposed thereon, generate, within the composite image and based on the lightmap, virtual reflected light as being reflected from the virtual item, and display the composite image on the at least one display.

[0007] In another general aspect, a method includes capturing an obj ect image of an object and reflected light from at least one light source providing incident light onto the object, using a first image sensor on a first side of a device, and capturing direct light from the at least one light source, using a second image sensor on a second side of the device. The method further includes generating a lightmap based on the object image and the direct light, generating a composite image of the object with a virtual item at least partially superimposed thereon, and generating, within the composite image and based on the lightmap, virtual reflected light as being reflected from the virtual item.

[0008] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] FIG. 1 A is a block diagram of a system for multi-sensor capture for virtual try-on.

[0010] FIG. IB illustrates an alternate implementation of the system of FIG. 1A.

[0011] FIG. 1C illustrates additional example implementations of the system of FIG.1A.

[0012] FIG. ID is a flowchart illustrating example operations of the systems of FIG. 1A-1C.

[0013] FIG. 2A is a front view of an example wearable device that may be tried on virtually using the techniques of FIGS. 1A-1D.

[0014] FIG. 2B is a rear view of the example wearable device show n in FIG. 2A.

[0015] FIGS. 3 A and 3B illustrate the capture of a base image of a test subject, in accordance with implementations described herein.

[0016] FIGS. 4 A and 4B illustrate the capture of a reference image of the test subject shown in FIGS. 3A and 3B, including fitting frames, in accordance with implementations described herein.

[0017] FIG. 5 is a flowchart of a first example method for training and validating a machine learning model, in accordance with implementations described herein.

[0018] FIG. 6 is a flowchart of a second example method for training and validating a machine learning model, in accordance with implementations described herein.DETAILED DESCRIPTION

[0019] Described systems and techniques enable accurate and realistic virtual try-on of wearable items, even when the wearable items being considered have high reflectivity. Consequently, the virtual try-on experience may be improved across a wider range of use cases, and goals of virtual try-on techniques (including, e.g., consumer satisfaction with remote purchases) may be more likely to be realized than in conventional techniques.

[0020] As referenced above, conventional virtual try-on techniques are intended to provide contextually accurate views of glasses and other wearable consumer items being worn by users, even when the users do not have physical access to the wearable items. In more detail, such conventional techniques generally attempt to generate a composite image of a user and a virtual wearable item being worn by the user, in real time, and while using relighting techniques to display the composite image in an accurate and realistic manner.

[0021] As described in detail, below, the term relighting in this context refers generally to the use of a lightmap (also referred to as a lighting map) to display the composite image in a manner that is realistic with respect to current lighting conditions. For example, toappear realistic, the composite image should include the virtual wearable item in a manner that is accurate with respect to directions and sizes of shadows, as well as intensities and locations of specular highlights.

[0022] It is possible to use a camera that captures the image of the user to detect illumination on the user’s face (e.g., skin) to obtain insight into the lighting of the environment of the user. For example, the user’s eye may be used as a de facto light probe to evaluate and characterize the environmental lighting. That is, a reflection of a light source incident on the user’s eye may be captured by the camera and used to characterize the light source in a resulting lightmap.

[0023] For example, a camera (e.g., a front camera) of a smartphone may capture an image of the user and a surrounding environment, which may then be displayed on a screen of the smartphone while the user manually holds the smartphone in a desired position(s) and at a desired angle(s). Processing resources of the smartphone may then be used to generate a lightmap from the captured image, as referenced above. For example, a trained neural network may be configured to input the captured image(s) and generate a corresponding lightmap. The resulting lightmap may then be used by the smartphone to render the virtual glasses or other virtual wearable item on the user within a composite image displayed on a screen of the smartphone.

[0024] Conventional systems, however, are not sufficiently reliable or accurate with respect to high frequency lighting effects needed to represent reflections on reflective surfaces of virtual glasses (e g., lenses) or other reflective portions of wearable items being evaluated using virtual try-on techniques. For example, existing techniques may only capture relatively diffuse estimates of lighting directions that are not suitable for modeling reflections on shiny surfaces (e.g., lenses of glasses). In other examples, a resolution of a camera being used may not be reliably adequate to generate a sufficiently high-quality lightmap. Such difficulties may be exacerbated by many difficult-to-predict details regarding the manner in which the user may position the camera during the virtual try-on, variations across users with respect to skin tone, and / or variations in lighting conditions across many different use case scenarios.

[0025] If a resulting composite image is not sufficiently realistic, then the composite image may be insufficient to provide the user with confidence in making the purchase being considered. Consequently, the user may forgo the purchase entirely.

[0026] In other examples, composite images that are not sufficiently realistic may convince the user to complete the purchase based on an inaccurate perception(s). Forexample, in the case of glasses, the composite image (including virtual glasses) in conventional implementations may misrepresent a size, material, and / or finish of the frame of the physical glasses. In such cases, the user may purchase the glasses, but may be more likely to be dissatisfied with the glasses and / or return the glasses once the user is able to physically wear and evaluate the glasses.

[0027] To improve a virtual try-on experience and result, described techniques use multiple sensors (e.g., multiple cameras, such as a front camera and a rear camera of a smartphone) to obtain a high-resolution lighting estimation for a relevant environment. As a result, described techniques create realistic lighting effects on virtual glasses in the ty pes of composite images referenced above, even when the virtual wearable items include highly reflective surfaces.

[0028] In more detailed examples, a first (e.g., front) camera may be used to capture a user’s face. The resulting image capture may be sufficient to enable a trained neural network to provide a diffuse estimate of the environment, which may be used for global illumination effects on diffuse components of the wearable item (e.g., a diffuse component such as a frame of virtual glasses).

[0029] A second (e.g., rear) camera may be used to capture information regarding light sources within the environment that are not in a field of view of the first / front camera, but that are incident on the user’s face. Described techniques may use the trained neural network to provide a high frequency light estimate of the environment, using the output of the second / rear camera. By detecting and processing such light sources with respect to the captured image, described techniques provide improved and reliable accuracy with respect to rendering reflections (i.e., virtual reflections) from lenses of the virtual glasses (or other wearable item) within a composite image that includes the captured object image (e.g., user) and the virtual wearable item.

[0030] FIG. 1 is a block diagram of a device 106 for multi-sensor capture for virtual try-on. As shown, a user 102 may be located in an environment (e.g., in a room, or outdoors) with one or more light sources, represented by a light source 104. The device 106 may include a front camera 108 that is positioned (e.g., held by the user 102) to capture reflected light 110 representing incident light 112 that has been reflected from the face of the user 102. Meanwhile, a rear camera 114 (not typically visible from the angle of illustration of FIG. 1 but shown in exploded view in FIG. 1 for the sake of clarity ) is positioned (e.g., held by the user 102) to capture direct light 116 from the light source 104 that is directly incident upon the rear camera 114.

[0031] Accordingly, a display 118 of the device 106 may be used to display a composite image 120 that includes an image 122 of the user 102, captured by the front camera 108, as well as virtual glasses 124 that are superimposed on the user’s face, as shown. The virtual glasses 124 include a virtual frame 126 and virtual lenses 128. For the sake of the example of FIG. 1., it is assumed that the virtual frame 126 is generally a source of diffuse reflections, while the virtual lenses 128 are considered a source of specular reflections.

[0032] In general, a specular reflection as used herein refers to a high frequency angular response to incident light, which only or primarily reflects incident illumination from a reflected direction. In contrast, a diffuse reflection as used herein refers to a low frequency response that tends to average out incident illumination over many incident directions. As noted above, a face of the user 102 will tend to reflect the reflected light 110 as a diffuse reflection, which may be included in the composite image 120 as example diffuse reflection 130. Consequently, conventional techniques that depend on the captured image 122 of the user 102 (as captured by the front camera 108) for lightmap creation may be suitable for including a corresponding diffuse reflection 130 from a face of the user 102 within the image 122, as well as for including a virtual diffuse reflection 132 from the virtual frame 126.

[0033] However, as also noted above, conventional techniques are typically unable to accurately depict specular virtual reflection 134 from the virtual lenses 128. In contrast, the example of FIG. 1 uses the direct light 116, captured from the light source 104 using the rear camera 114, when generating the composite image 120. As described in more detail, below, including use of the direct light 116 enables reliable and accurate generation of the composite image 120, including both virtual diffuse reflections 132 from the virtual frame 126 and virtual specular reflections 134 from the virtual lenses 134.

[0034] For example, as shown in the exploded view of FIG. 1, the device 106 may include a memory 136 representing one or more memories (e.g., non-transitory computer readable storage media), as well as a processor 138 representing one or more processors. The memory 136 may be used to store instructions that are executable by the processor 138, which, when executed, cause the processor 138 to generate the composite image 120 as described herein.

[0035] More detailed examples of the device 106 and associated hardware / software resources are provided below, e.g., with respect to FIG. 1C. For purposes of described the simplified example of FIG. 1, the memory 136 is illustrated as storing a lightmap model 140 representing any machine learning (ML) model that may be trained to input image data from the front camera 108 (including diffuse reflected light 110) and from the rear camera 114(including direct light 116), to thereby provide a lightmap 142.

[0036] For example, the lightmap model 140 may include a neural network trained using various types and quantities of training data. For example, in the example of FIG. 1, such training data may include light stage training data captured in a physical environment in which one or more users perform a physical try-on of physical glasses within the context of a known and well-defined lighting environment. In other examples, synthetic training data may be generated and used to train the lightmap model 140. Additional example details related to training the lightmap model 140 are described in more detail, below, with respect to FIGS. 3A-6.

[0037] A simulation engine 144 may utilize the lightmap 142 to generate the composite image 120. For example, the simulation engine 144 may be implemented in conjunction with a suitable browser application, camera application, and / or other application(s).

[0038] FIG. 1 illustrates a simplified example to illustrate a dual-camera approach to generating the composite image 120 including the virtual specular reflection 134. In addition to the techniques described above, many additional or alternative techniques may be used to supplement or modify the approach of FIG. 1.

[0039] For example, the diffuse reflected light 110 is described above with respect to reflections from a face of the user 102. In some implementations, more specific reflections from an eye of the user 102 may be captured, and included as such within inputs to the lightmap model 140. For example, the user 102 may be instructed to move the device 106 sufficiently close to enable the front camera 108 to capture a detailed image of an eye of the user 102.

[0040] Somewhat similarly, the user 102 may be instructed to move the device 106 in a specified pattern, to enable additional inputs to the lightmap model 140, with associated improvements in the lightmap 142. Such patterned movements may also enable a more detailed and nuanced capture of the face of the user 102, to further improve a quality of the composite image 120.

[0041] A background tracker 146 may be included and configured to track the above types of movements of the device 106, and other movements of the device 106, to further facilitate and improve a quality of the composite image 120. For example, while viewing and considering the virtual glasses 124, the user 102 may change an angle of the user’s face to consider the virtual glasses from difierent angles. The background tracker 146 may enable updating of the lightmap 142 and use background tracking to update the virtual reflections130, 132, 134 on the virtual glasses 124 in real-time.

[0042] In some example implementations, the user 102 may be provided with an option to exclude use of the rear camera 114 when executing the virtual try-on. In such scenarios, the lightmap model 140 may generate the composite image 120 using only input from the front camera 108. In some such scenarios, the lightmap model 140 may represent two or more lightmap models, where one such lightmap model is trained for use with a single camera, corresponding to the front camera 108.

[0043] FIG. IB illustrates an alternate implementation of the system of FIG. 1A. In FIG. 1A, the user 102 uses the front camera 106 to capture the image 120 of the user 102, as described, while the rear camera 108 captures the direct light 116. Accordingly, the composite image 120 is generated with the virtual glasses 124 simulated on the image 122 of the user 102.

[0044] In FIG. IB, the user 102 is illustrated as holding the device 106 to capture an image of a second user 102a using the rear camera 114 (not visible in FIG. IB). Thus, in the example of FIG. IB, the front camera 108 captures direct light reaching the front camera 108 from a direction of the user 102, and corresponding to the direct light 116 of FIG. 1, while the rear camera 114 captures reflected light that is reflected, e.g., from a face of the user 102a.

[0045] Thus, FIG. IB illustrates a VTO scenario in which a first user (the user 102) is provided with the composite image 120a in which virtual glasses 124a are projected onto an image 122a of the user 102a. In other words, the user 102 is provided with an ability to evaluate an appearance of the virtual glasses 124a when worn by the user 102a. Moreover, as already described with respect to FIG. 1A, the resulting composite image 120a will be highly accurate, including accurate representation of virtual reflected light from virtual lenses and other specular (virtual) surfaces of the composite image 120a.

[0046] FIG. 1C is a third person view of the user 102 in an ambient environment 1000, with one or more external computing systems shown as additional resources 152 that are accessible to the user 102 via a network 1200. FIG. 1C illustrates numerous different wearable devices that are operable by the user 102 on one or more body parts of the user 102, including a first wearable device 150 in the form of glasses worn on the head of the user, a second wearable device 154 in the form of ear buds worn in one or both ears of the user, a third wearable device 156 in the form of a watch worn on the wrist of the user, and an implementation of the device 106 of FIG. 1 as a handheld computing device held by the user 102.

[0047] Thus, in some examples, the device 106 may be used to capture an image of atleast a portion of an arm of the user 102, and a virtual item superimposed in a simulated composite image may include an image of a virtual watch, corresponding to the third wearable device 156, superimposed on the portion of the arm within the composite image. Similarly, the device 106 may be used to capture an image of at least a portion of an ear of the user 102, and a virtual item superimposed in a simulated composite image may include an image of a virtual earbud(s), corresponding to the second wearable device 154, superimposed on the portion of the ear within the composite image.

[0048] In some examples, the first wearable device 150 is in the form of a pair of smart glasses including, for example, a display, one or more images sensors that can capture images of the ambient environment, audio input / output devices, user input capability, computing / processing capability and the like. In some examples, the second wearable device 154 is in the form of an ear worn computing device such as headphones, or earbuds, that can include audio input / output capability, an image sensor that can capture images of the ambient environment, computing / processing capability, user input capability and the like. In some examples, the third wearable device 156 is in the form of a smart watch or smart band that includes, for example, a display, an image sensor that can capture images of the ambient environment, audio input / output capability, computing / processing capability, user input capability and the like. In some examples, the handheld computing device 106 can include a display, one or more image sensors that can capture images of the ambient environment, audio input / output capability, computing / processing capability, user input capability, and the like, such as in a smartphone. In some examples, the example wearable devices 150, 154, 156 and the example handheld computing device 106 can communicate with each other and / or with the external computing system(s) 152 to exchange information, to receive and transmit input and / or output, and the like. The principles to be described herein may be applied to other types of wearable devices not specifically shown in FIG. 1C or described herein.

[0049] The user 102 may choose to use the device 106, or other suitable device, for the virtual selection, sizing and fitting of any of the wearable items or wearable devices 150, 154, 156 of FIG. 1C, or other wearable items, while obtaining the benefits of using multiple image sensors, as described herein. For example, the user 102 may use an application executing on the device 106 to select glasses for virtual try on, and for the virtual sizing and fitting of selected glasses. In order to provide for the virtual sizing and / or fitting of a wearable device such as the example glasses 150, the user 102 may use two or more image sensors of the device 106 to capture images, for example a series of images, of the face / head of the user 102.

[0050] In some examples, the images may be captured by the multiple image sensors via an application executing on the device 106. In some examples, fixed features, or landmarks, may be detected within the series of images captured by one or more image sensors of the device 106. In some examples, position and / or orientation data provided by a sensor of the device 106 may be combined with the detection of landmarks and / or fixed features in the series of images. The combination of the detected landmarks and / or features in the series of images together with the position and / or orientation data associated with the device 106 as the series of images are captured, may allow a depth map to be developed without the use of specialized equipment such as, for example a depth sensor in operation as the images are captured. A three-dimensional mesh, for example, of the face / head of the user, may be developed from the depth data collected in this manner, as the series of images is captured, and the detected landmarks and / or features in the series of images may be combined with the position and / or orientation data associated with the device 106 as the series of images are captured. The resulting three-dimensional mesh may be processed, for example by a sizing simulator, to predict sizing and / or fitting of the wearable device, such as the example wearable glasses 150. The ability to accurately predict fit in this manner may simplify the process associated with the fitting of a wearable device such as, for example the wearable glasses 150, making such wearable devices more easily accessible to a wide variety of users.

[0051] As referenced above, the device 106 may access the additional resources 152 to facilitate the sizing and / or fitting of any wearable item. In some examples, the additional resources 152 may be partially or completely available locally on the device 106. In some examples, some of the additional resources 152 may be available locally on the device 106, and some of the additional resources 152 may be available to the device 106 via the network 1200. As shown, the additional resources 152 may include, for example, server computer systems, processors, databases, memory storage, and the like. In some examples, the processor(s) may include object recognition engine(s) and / or module(s), pattern recognition engine(s) and / or module(s), data aggregation engine(s) and / or modules(s), simulation engine(s) and / or module(s), sizing / fittmg engine(s) and / or module(s), and other such processors. In some examples, the additional resources may include ML model(s), such as mesh algorithms for the types of three-dimensional mesh representations of the user 102 referenced above.

[0052] The device 106 may operate under the control of a control system 160. The device 106 can communicate with one or more external devices, either directly (via wiredand / or wireless communication), or via the network 1200. In some examples, the one or more external devices may include various ones of the illustrated wearable computing devices 150, 154, 156, another mobile computing device similar to the device 106, and the like. In some implementations, the device 106 includes a communication module 162 to facilitate external communication. In some implementations, the device 106 includes a sensing system 164 including various sensing system components. The sensing system components may include, for example one or more image sensors 322, one or more position / orientation sensor(s) 164 (including for example, an inertial measurement unit, an accelerometer, a gyroscope, a magnetometer and other such sensors), one or more audio sensors 166 that can detect audio input, one or more touch input sensors 168 that can detect touch inputs, and other such sensors. The device 106 can include more, or fewer, sensing devices and / or combinations of sensing devices.

[0053] In some implementations, the one or more image sensor(s) 322 may include, for example, cameras such as, for example, one or more forward facing cameras such as the front camera 108 of FIG. 1A, and one or more outward, or world facing, cameras, such as the rear camera 114 of FIG. 1A. The one or more image sensor(s) 166 can capture still and / or moving images of an environment outside of the device 106. The still and / or moving images may be displayed by a display device of an output system 172, and / or transmitted externally via a communication module 162 and the network 1200, and / or stored in a memory 170 of the device 106. The device 106 may include one or more processor(s) 174. The processors 174 may include various modules or engines configured to perform various functions. In some examples, the processor(s) 174 may include object recognition engine(s) and / or module(s), pattern recognition engine(s) and / or module(s), configuration identification engine(s) and / or modules(s), simulation engine(s) and / or module(s), sizing / fitting engine(s) and / or module(s), and other such processors. The processor(s) 174 may be formed in a substrate configured to execute one or more machine executable instructions or pieces of software, firmware, or a combination thereof. The processor(s) 174 can be semiconductorbased including semiconductor material that can perform digital logic. The memory 170 may include any type of storage device that stores information in a format that can be read and / or executed by the processor(s) 174. The memory 170 may store applications and modules that, when executed by the processor(s) 174, perform certain operations. In some examples, the applications and modules may be stored in an external storage device and loaded into the memory 170.

[0054] FIG. ID is a flowchart illustrating example operations of the systems of FIGS.1A-1C. In the example of FIG. ID, operations 180-188 are illustrated as separate, sequential operations. However, in various example implementations, the operations 180-188 may be implemented in a different order than illustrated, in an overlapping or parallel manner, and / or in a nested, iterative, looped, or branched fashion. Further, various operations or suboperations may be included, omitted, or substituted.

[0055] In FIG. ID, an object image of an object and reflected light from at least one light source providing incident light onto the object may be captured, using a first image sensor on a first side of a device (180). For example, the image 122 of the user 102 and the reflected light 110 from the light source 104 may be captured using the front camera 108 of the device 106, e.g., a smartphone.

[0056] Direct light from the at least one light source may be captured, using a second image sensor on a second side of the device (182). For example, the direct light 116 from the light source 104 may be captured using the rear camera 114 of the device 106.

[0057] A lightmap may be generated, based on the object image and the direct light (184). For example, the lightmap model 140 may be trained to generate the lightmap 142, using inputs from both the front camera 108 and the rear camera 114.

[0058] A composite image of the object with a virtual item superimposed thereon may be generated (186). For example, the simulation engine 144 may be configured to use the lightmap 142 and the image 120 to generate the composite image 122, including the virtual glasses 124.

[0059] Within the composite image and based on the lightmap, virtual reflected light may be generated as being reflected from the virtual item (188). For example, the virtual diffusion reflected light 132 and the virtual specular reflected light 134 may be included in the composite image 122. In this way, as described above, the user 102 may reliably be provided with an accurate, realistic representation of the user 102 wearing the virtual glasses 124, so that the user 102 may make an informed decision with respect to purchasing one or more pairs of physical glasses corresponding to the virtual glasses 124, even if the user 102 does not have access to the physical glasses to try them on.

[0060] Hereinafter, systems and methods will be described with respect to the sizing and / or fitting or a wearable device from images of the face / head of the user 102, together with position and / or acceleration data provided by the device 106, for example, for the fitting of a head mounted wearable device, simply for purposes of discussion and illustration. Principles to be described herein may be similarly used for sizing and / or fitting from images captured by a computing device, together with position / acceleration data provided by thecomputing device, for other purposes such as, for example, the sizing and / or fitting of other types of wearable devices including apparel, the insertion of augmented reality items into an augmented reality scene and / or a real world scene, and the like.

[0061] An example head mounted wearable device 200 in the form of a pair of smart glasses is shown in FIGS. 2A and 2B, for purposes of discussion and illustration. The example head mounted wearable device 200 includes a frame 202 having rim portions 203 surrounding glass portion, or lenses 207, and arm portions 230 coupled to a respective rim portion 203. In some examples, the lenses 207 may be corrective / prescription lenses. In some examples, the lenses 207 may be glass portions that do not necessarily incorporate corrective / prescription parameters. A bridge portion 209 may connect the rim portions 203 of the frame 202. In the example shown in FIGS. 2A and 2B, the wearable device 200 is in the form of a pair of smart glasses, or augmented reality glasses, simply for purposes of discussion and illustration. The principles to be described herein can be applied to the sizing and / or fitting of a head mounted w earable device in the form of glasses that do not include the functionality typically associated with smart glasses. The principles to be described herein can be applied to the sizing and / or fitting of a head mounted wearable device in the form of glasses (including smart glasses, or eyewear that does not include the functionality typically associated with smart glasses) that include corrective / prescription lenses.

[0062] In some examples, the wearable device 200 includes a display device 204 that can output visual content, for example, at an output coupler 205, so that the visual content is visible to the user. In the example shown in FIGS. 2A and 2B, the display device 204 is provided in one of the two arm portions 230, simply for purposes of discussion and illustration. Display devices 204 may be provided in each of the two ami portions 230 to provide for binocular output of content. In some examples, the display device 204 may be a see through near eye display. In some examples, the display device 204 may be configured to project light from a display source onto a portion of teleprompter glass functioning as a beamsplitter seated at an angle (e.g., 30-45 degrees). The beamsplitter may allow for reflection and transmission values that allow the light from the display source to be partially reflected while the remaining light is transmitted through. Such an optic design may allow a user to see both physical items in the world, for example, through the lenses 207, next to content (for example, digital images, user interface elements, virtual content, and the like) output by the display device 204. In some implementations, waveguide optics may be used to depict content on the display device 204.

[0063] The example wearable device 200, in the form of smart glasses as shown inFIGS. 2A and 2B, includes one or more of an audio output device 206 (such as, for example, one or more speakers), an illumination device 208, a sensing system 210, a control system 212, at least one processor 214, and an outward facing image sensor 216 (for example, a camera). In some examples, the sensing system 210 may include various sensing devices and the control system 212 may include various control system devices including, for example, the at least one processor 214 operably coupled to the components of the control system 212. In some examples, the control system 212 may include a communication module providing for communication and exchange of information between the wearable device 200 and other external devices. In some examples, the head mounted wearable device 200 includes a gaze tracking device 215 to detect and track eye gaze direction and movement. Data captured by the gaze tracking device 215 may be processed to detect and track gaze direction and movement as a user input. In the example shown in FIGS. 2A and 2B, the gaze tracking device 215 is provided in one of two arm portions 230, simply for purposes of discussion and illustration. In the example arrangement shown in FIGS. 2A and 2B, the gaze tracking device 215 is provided in the same arm portion 230 as the display device 204, so that user eye gaze can be tracked not only with respect to objects in the physical environment, but also with respect to the content output for display by the display device 204. In some examples, gaze tracking devices 215 may be provided in each of the two arm portions 230 to provide for gaze tracking of each of the two eyes of the user. In some examples, display devices 204 may be provided in each of the two arm portions 230 to provide for binocular display of visual content.

[0064] The example wearable device 200 can include more, or fewer features than described above. The principles to be described herein are applicable to the virtual sizing and / or fitting of head mounted wearable devices including display capability and / or computing capability, i.e., smart glasses, and also to head mounted wearable devices that do not include display and / or computing capabilities, and to head mounted wearable devices with or without corrective lenses.

[0065] The wearable device 200 is described in detail herein, e.g., because the various included components may impose particular imaging requirements in virtual try-on scenarios, which may not be present, e.g., in the context of conventional glasses that do not have any computing abilities. For example, an appearance of the lenses 207 may be affected by the various display abilities of the wearable device 200, and it may be desirable to include, and accurately reproduce, any such specialized appearances when simulating the composite image 120 of FIG. 1A, or the composite image 120a of FIG. IB. Otherwise, as alreadydescribed, the user 102 may be more prone to forgo purchasing the wearable device 200, or more prone to returning or replacing the wearable device 200.

[0066] Additionally, in some implementations, the user 102 may be wearing the wearable device 200 while using the device 106 during VTO scenarios. For example, when considering purchase of a new pair of glasses, which may be conventional glasses without computing abilities or augmented reality glasses (e.g., such as an upgraded or updated version of the wearable device 200), the device 106, using any of the internal or external computing resources described herein, may generate the composite image 120 or 120a by replacing the physical glasses being worn by the user 102 with virtual glasses being virtually tried on.

[0067] Moreover, in some implementations, computing abilities of the wearable device 200 may be used to supplement, augment, or replace computing processes of the device 106. For example, image sensors of the device 106 may be used to generate a lightmap as described herein, and the resulting composite image may be displayed for the user 102 using the wearable device 200, rather than the device 106.

[0068] The wearable device 200 is illustrated as glasses, such as smartglasses, augmented reality (AR) glasses, or virtual reality (VR) glasses. More generally, the wearable device 200 may represent any head-mounted device (HMD), including, e.g., a hat, helmet, or headband. Even more generally, the wearable device 200 and the computing device 106 may represent any wearable device(s), handheld computing device(s), or combinations thereof.

[0069] In a head mounted wearable device such as the example w earable device 200, w earable fit parameters take into account how a frame 202 fits on a particular user and / or looks and / or feels on a particular user. Wearable fit parameters may take into account, for example whether the frame 202 is wide enough to be comfortable with respect to the user’s temples, whether the rim portions 203 and bridge portion 209 are sized so that the bridge portion 209 can rest comfortably on the bridge of the user’s nose, w hether the arm portions 205 are sized to comfortably rest on the user’s ears, and other such comfort related considerations. Wearable fit parameters may take into account as-worn parameters including how the user naturally wears the w earable device 200, such as, for example, head posture / how the user 102 naturally holds his / her head, how the user 102 positions the wearable device 200 relative to his / her face, and the like. Wearable fit parameters may also take into account whether the size and / or shape and / or contour of the frame 202 is aesthetically pleasing to the user, and is compatible with the user’s facial features.

[0070] In contrast with wearable fit parameters, display fit parameters may provide an indication of placement of the display device 204 so that content displayed by the displaydevice 204 is visible to the user and / or that color and / or brightness of content viewed by the user 102 is substantially uniform and / or substantially accurately perceived by the user 102, and other such factors. For example, display fit parameters may provide an indication of placement of the display device 204 so that content displayed by the display device 204 is captured within at least a set portion of the field of view of the user 102. For example, the display fit parameters may be used to ensure that at least a set level of gazability is provided. This may correspond to an amount, or portion, or percentage of the display of content that is visible to the user at a set brightness level and a set pupil size at a periphery (for example, a least visible comer) of the field of view of the user. In a wearable device configured as smart glasses, or augmented reality glasses, the display fit parameters may be used to configure the display device 204 to provide the best possible placement of the display of content to the user.

[0071] Ophthalmic fit parameters, or ophthalmic fit measurements, may be taken into account when the lenses 207 to be incorporated into the wearable device 200 are prescription, or corrective lenses. Ophthalmic fit measurements may include, for example, a pupil height PH (a distance from a center of the pupil to a bottom of the lens 207), an interpupillary distance IPD (a distance between the pupils), a monocular pupil distance, for example, a left pupil distance LPD (a distance from a central portion of the bridge of the nose to the left pupil) and a right pupil distance RPD (a distance from the central portion of the bridge of nose to right pupil), a pantoscopic angle PA (an angle defined by the tilt of the lens 207 with respect to vertical), a vertex distance V (a distance from the cornea to the lens 207), and other such parameters, or measures. Ophthalmic fit measurements may be taken into account when fitting the wearable device 200 including the display device 204, to provide for placement of the display of content by the display device 204 within an eye box defined by a three- dimensional volume extending between the lens 207 and the eye of the user, where the display of content will be within the field of view of the user, and thus visible to the user. Ophthalmic fit measurements may also be taken into account when fitting the wearable device 200 including the prescription, or corrective lenses into the frame 202 of a wearable device that does not include computing and / or display capability.

[0072] FIGS. 3A, 3B, 4A, and 4B illustrate a process, in accordance with implementations described herein, for the capture of three-dimensional images, or scans, of a test subject, for use in the validation of a simulation module of a computing system that provides for the simulated placement of glasses on a head of a user. For example, the techniques of FIGS. 3A, 3B, 4A, and 4B may be used to train the lightmap model 140 of FIG.1A.

[0073] FIG. 3 A is a front view, and FIG. 3B is a rear view, of a test subject in an example capture space 350 in which a first three-dimensional image, or scan, of the head of the test subject may be captured. An image capture assembly positioned in the example capture space 350 may include a plurality of example capture devices 355, or cameras 355. The example capture devices 355, or cameras 355, may be positioned throughout the example capture space 350 to provide for the capture of 360-degree image data of the head of the test subject The example capture devices 355 may include, for example, image sensors, light sensors, depth sensors, light sources, and other such devices involved in the capture of a three-dimensional image or scan, and various combinations and / or arrangements thereof. The arrangement of the capture devices 355 of the image capture assembly shown in FIGS. 3A and 3B presents one example arrangement of example capture devices 355 in an example capture space 350. The image capture assembly may include other types and / or arrangements of components, in a similar space or other ty pes of spaces, to capture the three-dimensional image, or scan, of the head of the test subject.

[0074] In some examples, the plurality of capture devices 355 may be operated to capture a three-dimensional scan of the head of the test subject in the capture space 350. In the example shown in FIGS. 3A and 3B, the test subject is not wearing glasses, and the three- dimensional image, or scan, captured by the capture devices 355 may yield a three- dimensional base image 310, as shown in FIG. 3C. The example base image 310 shown in FIG. 3C presents a relevant portion of the three-dimensional image, or scan, that may be captured by the capture devices 355. In particular, as this example is described with respect to the validation of the predicted placement and / or simulation of glasses on the head of the test subject, a portion of the head of the test subject relevant to the placement of glasses is shown in FIG. 3C, simply for purposes of discussion and illustration. The three-dimensional image, or scan, produced based on the image data captured by the plurality of capture devices 355 in the capture space 350 may include a full 360-degree capture of the head of the test subject.

[0075] FIG. 4A is a front view, and FIG. 4B is a rear view, of the test subject in the example capture space 350. In the example shown in FIGS. 4A and 4B, the test subject is wearing a head mounted wearable device. The example wearable device shown in FIGS. 4 A and 4B is an example fitting frame 400. The example fitting frame 400 may be physically, or geometrically representative of actual frames of a head mounted wearable device to be worn by a user, but simply for sizing / fitting purposes, and thus non-functional. The example fitting frame 400 may correspond generally to the example wearable device 200 of FIGS. 2A and2B. For example, the example fitting frame 400 may include rim portions defining lens areas in which glass portion(s), or lenses may be received. The example fitting frame 400 may include a bridge portion that connects the rim portions, and arm portions may be coupled to a respective rim portion. In some examples, visual markers may be included on the fitting frame 400, to be identified, or recognized, in an image-based data capture of the fitting frame 400. The visual markers may include a detectable pattem(s), and one or more of the detectable patterns may provide for the identification of a configuration of the fitting frame 400. The configuration may include, for example, a size of the fitting frame, a contour of rim portions / lens areas, a length of bridge portion, a length of arm portions, a distance between the ami portions, atilt angle of the rim portions relative to the arm portions, and other such information.

[0076] In some examples, the plurality of capture devices 355 may be operated to capture a three-dimensional scan of the head of the test subject in the capture space 350 as shown in FIGS. 4A and 4B, in which the test subject is wearing the fitting frame 400. The three-dimensional image, or scan, captured by the capture devices 355 may yield a three- dimensional reference image. The visual markers may be detected within the reference image, and the reference image may present a relevant portion of the three-dimensional image, or scan, that may be captured by the capture devices 355 (and also, for example, corresponding to a base image captured by the capture devices 355). A resulting three- dimensional image, or scan, produced based on the image data captured by the plurality of capture devices 355 in the capture space 350 may thus include a full 360-degree capture of the head of the test subject wearing the fitting frame 400.

[0077] Thus, FIGS. 3A-4B illustrate example techniques for training and validation, in which, for example, the capture devices 355 capture the illustrated test subject using one light at a time of the plurality of lights of the capture devices 355, so that many images are captured in which the test subject has not substantially moved and is lit from different directions. Accordingly, relighting may be performed so that, for each image, a direction from which the test subject was lit may be determined, and corresponding analyses of reflections, intensities, color variations, and other lighting characteristics may be performed. Consequently, in the example of FIG. 1 A, the composite image 120 may be generated with a desired lighting using an appropriate combination of these lighting characteristics that correspond to lighting provided by the light source 104, relative to camera angles of the front camera 108 and the rear camera 114 at the time the composite image 120 is being generated.

[0078] FIG. 5 is a flowchart of a method 500, in accordance with implementationsdescribed herein. The method 500 may provide for the validation and adjustment of algorithms and / or machine learning models associated with the simulation of a wearable device in the form of, for example, a head mounted wearable device, such as smart glasses including a display device. The principles to be described herein may be applied to the validation and adjustment of algorithms associated with the simulation of other types of wearable devices.

[0079] Abase image may be captured by, for example, an image capture assembly (block 510). The base image may include an image of a body portion of a user, on which a wearable device is to be worn. For example, the base image may include an image capturing a face and head of the user, in which the user is not wearing the head mounted wearable device.

[0080] A reference image may be captured by the image capture assembly (block520). The reference image may include an image of the body portion of the user, wearing the wearable device. For example, the reference image may include an image capturing the face and head of the user wearing a reference head mounted wearable device.

[0081] The head mounted wearable device may include one or more visual markers and / or one or more known features that are detectable from the reference image captured by the image capture assembly (block 530). One or more measurements associated with the head mounted wearable device may be determined based on the detected one or more visual markers and / or the one or more known features (block 540). The one or more measurements may include wearable fit measurements and / or display fit measurements and / or ophthalmic fit measurements. The one or more measurements may be indicative of a fit of the head mounted wearable device on the user.

[0082] A composite image included the rendered virtual item may be generated by, for example, a simulation engine (block 550). The simulation engine may retrieve, for example, from a database, a three-dimensional model of the head mounted wearable device, and may generate a virtual rendering of the head mounted wearable device that is superimposed on the base image to generate the composite image.

[0083] One or more measurements detected within the rendered image may be compared to the corresponding one or more measurements associated with the head mounted wearable device detected in the reference image (block 560). For example, a pose (i.e., a position and / or an orientation) of rendered virtual frames on the face of the user in the rendered composite image may be compared to an actual physical position of reference frames on the face of the user as captured in a reference image, to validate the virtualsimulation. In some examples, a plurality of features detected in the reference image (i.e., pupil center, facial landmarks and the like) may be used to approximate a three-dimensional mesh of the head and / or face of the user to generate the virtual simulation. In some examples, the generation of this three-dimensional mesh may be generated by a simulation engine of the external resources 152. This comparison may be made based on measurements extracted from the reference image, compared to measurements extracted from the composite image. That is, a level of accuracy of the simulation modeling may be assessed and / or validated by comparing one or more of the wearable fit measurements and / or ophthalmic fit measurements described above. In some examples, the comparison may include a comparison of features of the reference frames with respect to facial landmarks and / or pupil center detected in the reference image, with the corresponding features of the virtual frames with respect to the corresponding pupil center(s) and / or facial features in the rendered composite image. As referenced above, visual markers and / or other known features of the reference frames, pupil center(s), and various facial landmarks from the reference image may provide for a determination of measurements such as vertex distance, interpupillary distance and / or right pupil distance and / or left pupil distance, ins size, pantoscopic height, pantoscopic angle, and other such measurements. In a similar manner, corresponding measurements may be extracted for pose of the virtual frames on the face of the user in the rendered composite image. In some examples, the measurements may be based on detection of known features of the virtual frames, detected pupil center(s), facial landmarks and the like as previously described. In some examples, corresponding measurements may be extracted from the three- dimensional models of the virtual frames and the head of the user forming the basis of the rendered image.

[0001] A magnitude in differences in the corresponding measurements may provide an indication of the accuracy of the modeling and simulation engines used for virtual try on sessions. In some instances, a magnitude in the difference between a set of corresponding measurements may be great enough to cause an adjustment in one or more of the modeling engines and / or in one or more of the simulation engines.

[0084] When it is determined that a difference between one or more of the measurements detected from the rendered image and the corresponding measurement detected from the reference image is greater than or equal to a set threshold value (block 570), an associated algorithm and / or machine learning module may be adjusted (block 580). The adjustment of the algorithm and / or machine learning module may continue to learn and adjust an association between detected features (for example, facial features of the testsubject) and detected features of the wearable device (for example, features of the glasses) based on learned association(s) between the detected features of the face of the test subject and the features of the head mounted wearable device may reduce differences betw een actual fit of the head mounted wearable device and the simulated fit of the head mounted wearable device, thus improving accuracy of the modeling and simulation of the fit of wearable devices during a virtual try on session.

[0085] FIG. 6 is a flowchart of a method 600, in accordance with implementations described herein. The method 600 may provide for the validation and adjustment of algorithms and / or machine learning models associated with the simulation of a wearable device (or other wearable item) as described with respect to FIG. 5, using additional or alternative techniques. As with FIG. 5, the principles to be described herein with respect to FIG. 6 may be applied to the validation and adjustment of algorithms associated with the simulation of other types of wearable devices.

[0086] In FIG. 6, a base image may be captured by, for example, an image capture assembly (block 610). The image capture assembly may include one or more image capture devices positioned in an image capture space so as to capture a three-dimensional image of a test subject in the image capture space. The base image may include an image of a body portion of a user, on which a wearable device is to be worn. For example, the base image may include a three-dimensional image capturing a face and a head of the user, in which the user is not wearing the head mounted wearable device.

[0087] A reference image may be captured by the image capture assembly (block 620). The reference image may include an image of the body portion of the user, wearing the wearable device. For example, the reference image may include a three-dimensional image capturing the face and the head of the user wearing a fitting frame including one or more visual markers and / or one or more known features that are detectable from the reference image captured by the image capture assembly. A combined image may be generated, including a three-dimensional model of the fitting frame superimposed on the reference image. The three-dimensional model of the fitting frame may be retrieved from a database including modeling and configuration information associated with a plurality of fitting frames. Markers on the three-dimensional model of the fitting frame may be aligned with markers on the fitting frame worn by the test subject in the reference image to position the three-dimensional model of the fitting frame in the combined image.

[0088] An aligned image may be generated based on the base image and the reference image (block 630). The combined image may include a region of interest extracted from thereference image superimposed on a region of interest extracted from the base image. The region of interest may be a region corresponding to a worn region of the wearable device.

[0089] An aligned image may be generated to align the base image and the reference image in a common three-dimensional space, or common frame of reference (block 640). A transform image may be generated (block 650), based on the alignment calculated from the combined image and reflected in the aligned image, positioning simulated glasses onto the reference image. A simulated composite image may then be generated (block 660), based on the transform image. The positioning of the simulated glasses on the reference image may be transferred onto the base image, as the base image and the reference image have been brought into a common three-dimensional space, or common frame of reference.

[0090] Many different variations of the above techniques may be implemented. For example, during the training and validation techniques of FIGS. 3A-6, the test subject of FIGS. 3A and 3B may sometimes wear glasses and sometimes not wear glasses during the training, so that the lightmap model 140, and / or another trained model, may be configured to determine actual reflected light from the glasses being worn, as well as to simulate an image of a user not wearing glasses from an image of the user wearing glasses (i.e., to execute a ML-based generation of a user image with glasses removed). Then, when the lightmap model 140 is trained and deployed, the user 102 of FIG. 1A may be wearing a physical pair of glasses. The lightmap model 140, and / or another trained model, may then first simulate removal of the physical glasses, and then utilize actual reflected light from the physical glasses in generating virtual reflected light 134 from subsequently generated virtual glasses 124 in the composite image 120. That is, for example, the actual reflected light from lenses or other shiny surfaces of the physical glasses being worn by the user 102 may be used as a basis for generating the virtual reflected light 134.

[0091] In addition to the examples of FIGS. 3A-6, in which lighting conditions are synthetically generated, training data may be captured by using a device similar to the device 106 of FIG. 1 to capture training images, while a reflective object (e.g., reflective sphere) is in the camera view(s) of camera(s) of the device. In still other examples, training data may be generated in a fully synthetic manner, using analysis of existing training data that includes lighting information.

[0092] In some examples, during training and / or live image capture at a time of VTO, a maximum frequency of virtual reflected light 134 may be set. In some examples, during a face scan performed prior to generation of the composite image 120, the rear camera 114 may be used for lighting information collection to ensure optimal fitting of the virtual glasses 124on the face of the user 102 (i.e., to ensure optimal determination of a geometry of the face of the user 102), in addition to ensuring accurate generation of the virtual reflected light 134.

[0093] As described herein, in a first example implementation referred to as example 1 includes a computer program product, the computer program product being tangibly embodied on a non-transitory computer-readable storage medium and comprising instructions that, when executed by at least one computing device, are configured to cause the at least one computing device to: capture an object image of an object and reflected tight from at least one light source providing incident light onto the object, using a first image sensor on a first side of a device; capture direct light from the at least one light source, using a second image sensor on a second side of the device; generate a lightmap based on the object image and the direct tight; generate a composite image of the obj ect with a virtual item at least partially superimposed thereon; and generate, within the composite image and based on the lightmap, virtual reflected light as being reflected from the virtual item.

[0094] In a second example, in the computer program product of example 1, the first image sensor includes a front camera of a smartphone, and the second image sensor includes a rear camera of the smartphone.

[0095] In a third example, in the computer program product of example 1, the first image sensor includes a rear camera of a smartphone, and the second image sensor includes a front camera of the smartphone.

[0096] In a fourth example, in the computer program product of examples 1-3, the object includes at least one body part, and the virtual item includes an image of a virtual wearable item superimposed on the at least one body part within the composite image.

[0097] In a fifth example, in the computer program product of example 4, the object includes a face, and the virtual item includes an image of virtual glasses superimposed on the face within the composite image.

[0098] In a sixth example, in the computer program product of example 4, the object includes a portion of an arm, and the virtual item includes an image of a virtual watch superimposed on the portion of the arm within the composite image.

[0099] In a seventh example, in the computer program product of example 4, the object includes a portion of an ear, and the virtual item includes an image of a virtual earbud superimposed on the portion of the ear within the composite image.

[0100] In an eighth example, in the computer program product of any of the preceding examples, the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to generate the lightmap using a trained neural network.

[0101] In a ninth example, in the computer program product of any of the preceding examples, the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to: track a relative movement of the device with respect to the object; and update the composite image in response to the relative movement.

[0102] In a tenth example, in the computer program product of any one of the preceding examples, the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to: determine the direct light using the first image sensor during a scan of the object captured prior to the capture of the object image; determine a geometry of the object, based on the scan; and determine a fit of the item with respect to the object within the composite image, based on the geometry.

[0103] In an eleventh examples, a device includes: at least one memory; at least one processor; at least one display; a first image sensor on a first side of the device and positioned to capture an object image of an object and reflected light from at least one light source providing incident light onto the object; a second image sensor on a second side of the device and positioned to capture direct light from the at least one light source; and a simulation engine including instructions stored using the at least one memory, which, when executed by the at least one processor, causes the device to generate a lightmap based on the object image and the direct light; generate a composite image of the object with a virtual item at least partially superimposed thereon; generate, within the composite image and based on the lightmap, virtual reflected light as being reflected from the virtual item; and display the composite image on the at least one display.

[0104] In a twelfth example, in the device of example 11, the simulation engine, when executed by the at least one processor, is further configured to cause the device to generate the lightmap using a trained neural network.

[0105] In a thirteenth example, in the device of example 11 or 12, the device includes a smartphone, the first image sensor includes a front camera of the smartphone, and the second image sensor includes a rear camera of the smartphone.

[0106] In a fourteenth example, in the device of any one of examples 11 to 13, the object includes at least one body part of a user, and the virtual item includes an image of a virtual wearable item superimposed on the at least one body part of the user within the composite image.

[0107] In a fifteenth example, in the device of example 14, the object includes a face, and the virtual item includes an image of virtual glasses superimposed on the face within the composite image.

[0108] In a sixteenth example, a method comprises: capturing an object image of an object and reflected light from at least one light source providing incident light onto the object, using a first image sensor on a first side of a device; capturing direct light from the at least one light source, using a second image sensor on a second side of the device; generating a lightmap based on the object image and the direct light; generating a composite image of the object with a virtual item at least partially superimposed thereon; and generating, within the composite image and based on the lightmap, virtual reflected light as being reflected from the virtual item.

[0109] In a seventeenth example, in the method of example 16, the first image sensor includes a front camera of a smartphone, and the second image sensor includes a rear camera of the smartphone.

[0110] In an eighteenth example, in the method of example 16 or 17, the object includes at least one body part, and the virtual item includes an image of a virtual wearable item superimposed on the at least one body part within the composite image.

[0111] In a nineteenth example, in the method of example 18, wherein the object includes a face, and the virtual item includes an image of virtual glasses superimposed on the face within the composite image.

[0112] In a twentieth example, the method of any one of examples 16 to 19, furthercomprises generating the lightmap using a trained neural network.

[0113] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0114] These computer programs (also known as modules, programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms “machine- readable medium” “computer-readable medium” refers to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0115] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, or LED (light emitting diode)) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well. For example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0116] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components ofthe system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (“LAN”), a wide area network (“WAN”), and the Internet.

[0117] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0118] A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the description and claims.

[0119] In addition, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. In addition, other steps may be provided, or steps may be eliminated, from the described flows, and other components may be added to, or removed from, the described systems. Accordingly, other implementations are within the scope of the following claims.

[0120] Further to the descriptions above, a user is provided with controls allowing the user to make an election as to both if and when systems, programs, devices, networks, or features described herein may enable collection of user information (e.g., information about a user’s social network, social actions, or activities, profession, a user’s preferences, or a user’s current location), and if the user is sent content or communications from a server. In addition, certain data may be treated in one or more ways before it is stored or used, so that user information is removed. For example, a user’s identity may be treated so that no user information can be determined for the user, or a user’s geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined. Thus, the user may have control over what information is collected about the user, how that information is used, and what information is provided to the user.

[0121] The computer system (e.g., computing device) may be configured to wirelessly communicate with a network server over a network via a communication link established with the network server using any known wireless communications technologies and protocols including radio frequency (RF), micro wave frequency (MWF), and / or infrared frequency (IRF) wireless communications technologies and protocols adapted for communication over the network.

[0122] In accordance with aspects of the disclosure, implementations of various techniques described herein may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. Implementations may be implemented as a computer program product (e.g., a computer program tangibly embodied in an information carrier, a machine-readable storage device, a computer-readable medium, a tangible computer-readable medium), for processing by, or to control the operation of, data processing apparatus (e.g., a programmable processor, a computer, or multiple computers). In some implementations, a tangible computer-readable storage medium may be configured to store instructions that when executed cause a processor to perform a process. A computer program, such as the computer program(s) described above, may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may be deployed to be processed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.

[0123] Specific structural and functional details disclosed herein are merely representative for purposes of describing example implementations. Example implementations, however, may be embodied in many alternate forms and should not be construed as limited to only the implementations set forth herein.

[0124] The terminology used herein is for the purpose of describing particular implementations only and is not intended to be limiting of the implementations. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises," "comprising," "includes," and / or "including," when used in this specification, specify the presence of the stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0125] It will be understood that when an element is referred to as being "coupled," "connected," or "responsive" to, or "on," another element, it can be directly coupled, connected, or responsive to, or on, the other element, or intervening elements may also be present. In contrast, when an element is referred to as being "directly coupled," "directly connected," or "directly responsive" to, or "directly on," another element, there are no intervening elements present. As used herein the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0126] Spatially relative terms, such as "beneath," "below," "lower," "above," "upper," and the like, may be used herein for ease of description to describe one element or feature in relationship to another element(s) or feature(s) as illustrated in the figures. It will be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as "below" or "beneath" other elements or features would then be oriented "above" the other elements or features. Thus, the term "below" can encompass both an orientation of above and below. The device may be otherwise oriented (rotated 130 degrees or at other orientations) and the spatially relative descriptors used herein may be interpreted accordingly.

[0127] Example implementations of the concepts are described herein with reference to cross-sectional illustrations that are schematic illustrations of idealized implementations (and intermediate structures) of example implementations. As such, variations from the shapes of the illustrations as a result, for example, of manufacturing techniques and / or tolerances, are to be expected. Thus, example implementations of the described concepts should not be construed as limited to the particular shapes of regions illustrated herein but are to include deviations in shapes that result, for example, from manufacturing. Accordingly, the regions illustrated in the figures are schematic in nature and their shapes are not intended to illustrate the actual shape of a region of a device and are not intended to limit the scope of example implementations.

[0128] It will be understood that although the terms "first," "second," etc. may be used herein to describe various elements, these elements should not be limited by these terms. These temis are only used to distinguish one element from another. Thus, a "first" element could be termed a "second" element without departing from the teachings of the present implementations.

[0129] Unless otherwise defined, the terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which these concepts belong. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and / or the present specification and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0130] While certain features of the described implementations have been illustrated as described herein, many modifications, substitutions, changes, and equivalents will nowoccur to those skilled in the art. It is, therefore, to be understood that the appended claims are intended to cover such modifications and changes as fall within the scope of the implementations. It should be understood that they have been presented by way of example only, not limitation, and various changes in form and details may be made. Any portion of the apparatus and / or methods described herein may be combined in any combination, except mutually exclusive combinations. The implementations described herein can include various combinations and / or sub-combinations of the functions, components, and / or features of the different implementations described

Claims

WHAT IS CLAIMED IS:

1. A computer program product, the computer program product being tangibly embodied on anon-transitory computer-readable storage medium and comprising instructions that, when executed by at least one computing device, are configured to cause the at least one computing device to: capture an object image of an object with at least one light source providing incident light onto the object, using a first image sensor on a first side of a device; capture direct light from the at least one light source, using a second image sensor on a second side of the device; generate a lightmap based on the object image and the direct light; generate a composite image of the obj ect with a virtual item at least partially superimposed thereon; and generate, within the composite image and based on the lightmap, virtual reflected light as being reflected from the virtual item.

2. The computer program product of claim 1, wherein the first image sensor includes a front camera of a smartphone, and the second image sensor includes a rear camera of the smartphone.

3. The computer program product of claim 1, wherein the first image sensor includes a rear camera of a smartphone, and the second image sensor includes a front camera of the smartphone.

4. The computer program product of any one of claims 1 to 3, wherein the object includes at least one body part, and the virtual item includes an image of a virtual wearable item superimposed on the at least one body part within the composite image.

5. The computer program product of claim 4, wherein the object includes a face, and the virtual item includes an image of virtual glasses superimposed on the face within the composite image.

6. The computer program product of claim 4, wherein the object includes a portion of an arm, and the virtual item includes an image of a virtual watch superimposed on the portion of the arm within the composite image.

7. The computer program product of claim 4, wherein the object includes a portion of an ear, and the virtual item includes an image of a virtual earbud superimposed on the portion of the ear within the composite image.

8. The computer program product of any one of the preceding claims, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to: generate the lightmap using a trained neural network.

9. The computer program product of any one of the preceding claims, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to: track a relative movement of the device with respect to the object; and update the composite image in response to the relative movement.

10. The computer program product of any one of the preceding claims, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to: determine the direct light using the first image sensor during a scan of the object captured prior to the capture of the object image; determine a geometry of the object, based on the scan; and determine a fit of the item with respect to the object within the composite image, based on the geometry.

11. A device comprising: at least one memory; at least one processor; at least one display; a first image sensor on a first side of the device and positioned to capture an object image of an object with at least one light source providing incident light onto the object;a second image sensor on a second side of the device and positioned to capture direct light from the at least one light source; and a simulation engine including instructions stored using the at least one memory, which, when executed by the at least one processor, causes the device to generate a lightmap based on the obj ect image and the direct light; generate a composite image of the object with a virtual item at least partially supenmposed thereon; generate, within the composite image and based on the lightmap, virtual reflected light as being reflected from the virtual item; and display the composite image on the at least one display.

12. The device of claim 11, wherein the simulation engine, when executed by the at least one processor, is further configured to cause the device to generate the lightmap using a trained neural network.

13. The device of claim 11 or 12, wherein the device includes a smartphone, the first image sensor includes a front camera of the smartphone, and the second image sensor includes a rear camera of the smartphone.

14. The device of any one of claims 11 to 13, wherein the object includes at least one body part of a user, and the virtual item includes an image of a virtual wearable item superimposed on the at least one body part of the user within the composite image.

15. The device of claim 14, wherein the object includes a face, and the virtual item includes an image of virtual glasses superimposed on the face within the composite image.

16. A method comprising: capturing an object image of an object with at least one light source providing incident light onto the object, using a first image sensor on a first side of a device; capturing direct light from the at least one light source, using a second image sensor on a second side of the device; generating a lightmap based on the object image and the direct light; generating a composite image of the object with a virtual item at least partially superimposed thereon; andgenerating, within the composite image and based on the lightmap, virtual reflected light as being reflected from the virtual item.

17. The method of claim 16, wherein the first image sensor includes a front camera of a smartphone, and the second image sensor includes a rear camera of the smartphone.

18. The method of claim 16 or 17, wherein the object includes at least one body part, and the virtual item includes an image of a virtual wearable item superimposed on the at least one body part within the composite image.

19. The method of claim 18, wherein the object includes a face, and the virtual item includes an image of virtual glasses superimposed on the face within the composite image.

20. The method of any one of claims 16 to 19, further comprising: generating the lightmap using a trained neural network.

Citation Information

Patent Citations

  • Method and system for representing a virtual object in a view of a real environment

    EP2933781A2