System and method for eyewear removal and virtual try-on

Abstract

A system includes a computing device including a processor communicatively coupled to a camera. The computing device is configured to, in response to receiving the request, capture an image by the camera and detect a first plurality of locations of the first object within the image. The computing device is further configured to segment the first plurality of locations of the first object by determining, for each location of the first plurality of locations, a likelihood that the corresponding location includes a portion of the first object. The computing device is configured to repair a second plurality of locations having an associated likelihood that corresponding locations above a threshold include a portion of the first object. The computing device is additionally configured to generate an enhanced image by superimposing the selected second object on the image, and to display the enhanced image on a user interface of the computing device.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Patent Application No. 17 / 856,482, filed on July 1, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to augmented reality by implementing a neural network, and more particularly, to augmented reality to remove an item and replace the item with a virtual item. Background Art

[0004] In order to visualize what a particular item would look like on an individual, virtual try-on methods have been implemented to place a virtual item on an individual's items or videos. Depending on the individual, there may be other items in the image or video that obstruct the visualization of the virtual item.

[0005] The background description provided here is for the purpose of generally presenting the background of the present disclosure. To the extent described in this background section, the work of the inventors of the present application and aspects of the description that may not qualify as prior art at the time of filing the application are not explicitly or implicitly admitted as prior art for the present disclosure. Summary of the invention

[0006] A system includes a computing device. The computing device includes at least one processor that is communicatively coupled to a camera. The computing device is configured to capture at least one image by the camera and detect a first plurality of locations of a first object within the at least one image in response to receiving a request. The computing device is further configured to segment the first plurality of locations of the first object by determining, for each of the first plurality of locations, a likelihood that the corresponding location includes a portion of the first object. The computing device is further configured to inpaint a second plurality of locations that have an associated likelihood that the corresponding location includes a portion of the first object above a threshold. The computing device is additionally configured to generate an augmented image by superimposing a selected second object on the at least one image, and display the augmented image on a user interface of the computing device.

[0007] In further aspects, the first plurality of locations of the first object are pixels within at least one image.

[0008] In further aspects, the first plurality of positions includes more positions than the second plurality of positions.

[0009] In further aspects, the first plurality of positions includes fewer positions than the second plurality of positions.

[0010] In further aspects, the first plurality of locations and the second plurality of locations are the same.

[0011] In a further aspect, the computing device is configured to capture real-time video via a camera and, for a subset of a plurality of frames of the real-time video, inpaint a set of positions among the plurality of positions.

[0012] In further aspects, the subset of the plurality of frames is based at least in part on a computing capability of the computing device.

[0013] In some embodiments, the computing capabilities of the computing device are determined based on the model of the computing device.

[0014] In additional aspects, the computing device is configured to reduce stray noise by blending a previous image with the current image before generating the enhanced image.

[0015] In further aspects, in response to detecting the absence of the plurality of locations of the first object, the computing device is configured to skip the segmentation step and the inpainting step to generate the enhanced image.

[0016] In further aspects, in response to the second plurality of locations having a correlation likelihood below a lower threshold, the computing device is configured to skip the inpainting step to generate the enhanced image.

[0017] In a further aspect, the system includes a remote database configured to store a set of objects including the selected second object, wherein the computing device is configured to obtain the selected second object from the remote database.

[0018] In further aspects, the selected second object is indicated in the request.

[0019] In further aspects, the computing device is a mobile computing device.

[0020] In another aspect, the first object is a pair of glasses.

[0021] In further aspects, the portion of the first object is a frame of a pair of eyeglasses.

[0022] In further aspects, the computing device is configured to detect a first plurality of locations of a first object within at least one image by implementing a first neural network. In further aspects, the computing device is configured to segment the first plurality of locations of the first object by determining, for each location in the first plurality of locations, a likelihood that the corresponding location includes a portion of the first object using a second neural network. In further aspects, the computing device is configured to inpaint a second plurality of locations having an associated likelihood that the corresponding location includes a portion of the first object above a threshold by implementing a third neural network.

[0023] A method includes capturing at least one image via a camera by a computing device including at least one processor and communicatively coupled to the camera in response to receiving a request. The method also includes detecting a first plurality of locations of a first object within the at least one image. The method also includes segmenting the first plurality of locations of the first object by determining, for each of the plurality of locations, a likelihood that the corresponding location includes a portion of the first object. The method additionally includes inpainting a second plurality of locations having an associated likelihood that the corresponding location includes a portion of the first object above a threshold. The method also includes generating an augmented image by superimposing a selected second object on the at least one image, and displaying the augmented image on a user interface of the computing device.

[0024] In some aspects, detection is performed by a first module, segmentation is performed by a second module, and repair is performed by a third module.

[0025] In some aspects, the first module is different from the second module and the third module.

[0026] In some respects, the second module is different from the first module and the third module.

[0027] In some aspects, the third module is different from the first module and the second module.

[0028] In some aspects, the first and second modules are the same module.

[0029] In some respects, the second and third modules are identical.

[0030] In some aspects, the first, second, and third modules are separate modules, each module configured to execute a respective artificial neural network.

[0031] In some aspects, the first and second modules are the same module.

[0032] In some aspects, a method includes determining a computing capability of a computing device; capturing, via a camera, a real-time video comprising a first plurality of frames; and repairing a second plurality of locations in a second plurality of frames. The second plurality of frames may be based on the computing capability of the computing device.

[0033] In some aspects, the second plurality of frames may be different than the first plurality of frames.

[0034] In some aspects, a method may include blending a previous image with a current image to reduce stray noise before generating an enhanced image.

[0035] In some aspects, a method may include, in response to detecting a lack of a plurality of locations of a first object, skipping a segmentation step and an inpainting step to generate an enhanced image.

[0036] A non-transitory computer-readable medium having instructions stored thereon. The instructions, when executed by at least one processor, cause a device to perform operations. The operations may include, in response to receiving a request, capturing at least one image via a camera by a computing device including at least one processor and communicatively coupled to the camera. The operations may include detecting a first plurality of locations of a first object within the at least one image. The operations may include segmenting the first plurality of locations of the first object by determining, for each of the first plurality of locations, a likelihood that the corresponding location includes a portion of the first object. The operations may include inpainting a second plurality of locations having an associated likelihood that the corresponding location includes a portion of the first object above a threshold. The operations may include generating an augmented image by superimposing a selected second object on the at least one image, and displaying the augmented image on a user interface of the computing device.

[0037] In some aspects, detection may be performed by a first module, segmentation may be performed by a second module, and repair may be performed by a third module.

[0038] In some aspects, the operations may include determining a computing capability of a device; capturing, via a camera, a real-time video comprising a first plurality of frames; and repairing a second plurality of locations in a second plurality of frames. The second plurality of frames may be based on the computing capability of the device.

[0039] In some aspects, the operations may include blending a previous image with the current image to reduce stray noise before generating the enhanced image.

[0040] Further areas of applicability of the present disclosure will become apparent from the detailed description, claims, and drawings.The detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The present disclosure will be more fully understood from the detailed description and accompanying drawings.

[0042] Figure 1-4 is an example user interface for a user device for navigating a user to virtually try on a pair of glasses.

[0043] Figure 5 is a functional block diagram of an example architecture for implementing a virtual try-on device.

[0044] Figure 6 is an example flow chart depicting the execution of virtual glasses removal and replacement.

[0045] In the drawings, reference numerals may be repeated to identify similar and / or identical elements. DETAILED DESCRIPTION

[0046] A virtual try-on system can be implemented on a user device to remove glasses or other objects present in an image or video of the user and superimpose a selected pair of glasses or other objects on the user over the image or video with the existing glasses removed. Typically, virtual try-on technology uses augmented reality to show a user what a pair of glasses frames would look like on the user's face. For example, a user can browse multiple pairs of glasses on a user device and select a pair for virtual try-on. To perform a virtual try-on, an image of the customer is captured by a camera included in or communicatively connected to the user device, and the glasses frames are superimposed on the captured image. The user interface of the user device displays the composite image back to the user. The process can be performed on a single image, a sequence of images (i.e., a video), or in a real-time streaming manner. Although the following description is provided with reference to a glasses try-on system, it should be understood that the concepts disclosed herein can be used for other purposes. For example, the systems and methods described herein can be used to detect a first object in an image and replace the first object with a second object that is different from the first object. An augmented image, i.e., an image in which the first object has been replaced by the second object, can then be displayed or otherwise presented to a user or other individual.

[0047] In various cases, the user may already be wearing glasses and need to continue wearing their existing glasses in order to properly see the composite image or video displayed back to the user. However, superimposing the virtual glasses on top of the existing glasses does not produce a clear or realistic image for the user of what the virtual glasses will look like on the user.

[0048] The present virtual try-on system not only overlays a selected pair of glasses on one or more images of the user, but also uses a combination of multiple modules or neural networks to detect and remove any existing glasses before overlaying. In some embodiments, three neural networks are implemented in a series of steps to (1) detect glasses, (2) segment the frames of the glasses, and (3) repair the identified frames of the glasses before overlaying the selected pair of glasses. The segmentation step analyzes each pixel in the image to determine the probability that the pixel belongs to a specific object class or a background class. The repair step involves a process of intervening the content of the missing portion of the image based on neighboring parts of the image, patterns learned from a training set of images, or some combination of the two. It should be understood that although three neural networks are described in the following example, fewer or more neural networks can be used. The benefit of using multiple separate neural networks for each of the above steps in detecting, segmenting, and repairing an existing pair of glasses is that a specific neural network can be targeted to a specific step or task, rather than a single neural network performing each step overall. That is, by breaking down the virtual try-on system into using three different neural networks, the virtual try-on system is able to run in real time (e.g., while the user is shooting a streaming video of themselves) and use less memory than using a single neural network to perform the entire analysis. This is advantageous when the processing power of the user device or other computing device may be limited, such as when the computing device is a mobile phone, tablet computer, or lightweight computer. In addition, in various embodiments, the virtual try-on system may also incorporate a method for reducing transient stray noise introduced by the restoration process. The method reduces noise between superimposed images by averaging or blending the images to reduce blur and more seamlessly transition from the first image to the second image.

[0049] In some embodiments, the first step in removing existing real glasses that the user is wearing is to detect a pair of glasses, which is implemented using a first neural network, for example, using a yolov4-tiny object detection neural network pre-trained on an image dataset. Although reference is made to the yolov4-tiny object detection neural network as an example, a person of ordinary skill in the art will understand that other neural networks may be used to perform object detection. The first neural network may be trained on an image dataset of manually annotated images of individuals wearing glasses and identifying regions of interest including glasses. The identified region of interest may be, for example, a frame surrounding the frame of the glasses. The detection step not only simplifies the analysis when the glasses are not detected, but also reduces the area in which the frames are segmented / identified, making the segmentation operation faster. In various embodiments, if no glasses are detected, the virtual try-on system may be directed to superimpose a selected pair of glasses since the existing glasses do not need to be removed.

[0050] Once the glasses are detected and the region of interest is identified, the virtual try-on system can segment the glasses by determining, for each pixel in the region of interest, the likelihood of including a portion of a glasses frame. Segmentation can be accomplished using a second neural network having, for example, a Unet neural network architecture and a ReLU activation function. Again, while the Unet neural network architecture and the ReLU activation function are references, other neural network architectures and / or activation functions can be implemented. The second neural network can be trained using a custom dataset that includes images with synthetic glasses and images with real glasses that were manually segmented using a custom tool. In various embodiments, the second neural network returns a percentage for each pixel in the region of interest indicating the likelihood that the pixel includes a portion of a glasses frame. For example, if the percentage is above 50%, it can be determined that the pixel includes a portion of a glasses frame. It should be understood that other percentages can be used to determine whether a pixel includes a portion of a glasses frame or other objects that may need to be removed from one or more images.

[0051] Next, a third neural network can be implemented to repair pixels that are determined to include a portion of a glasses frame. The repair can be accomplished using an AOT-GAN neural network with a VGG16 encoder pre-trained on an image dataset; however, other neural networks and / or encoders may be used. In some embodiments, to ensure per-pixel reconstruction accuracy and visual fidelity of the repaired image, the third neural network can be trained using the following four optimization objectives: L1 loss, style loss, perceptual loss, and adversarial loss. The training set may include thousands of paired images with and without glasses. To generate training data, synthetic glasses may be superimposed on images in a dataset of individuals without glasses. The result of the repair step may include replacing pixels that include a frame with what the third neural network identifies as what the frame might look like in that pixel if it were not located at the pixel based on the image used to train the third neural network.

[0052] Since each of the detection, segmentation, repair, etc. steps can be computationally intensive, the virtual try-on system may not implement repair on every video frame of the video, but still achieve real-time operation. To overcome this problem, the repair step can be implemented on less than all video frames or images, for example, on every third image, but the repair can be implemented on other numbers of images (e.g., two, four, etc.). To achieve this, on images where repair is not performed, the segmented pixels are filled with information from the most recently repaired image. In various embodiments, the virtual try-on system can have a specific number of images, and the device based on which the virtual try-on is performed on the image skips the repair step. For example, on a device with better processing speed or capability, the repair step can be completed on every image or every other image, while those user devices with lower processing capabilities can only perform repair on every three, every four, or every other image.

[0053] The virtual try-on system can then overlay the selected pair of glasses identified in the user request. For example, as described in U.S. Patent No. 10,825,260 to Goldberg et al. and U.S. Publication No. 2021 / 0065285 to Goldberg et al., which are hereby incorporated by reference in their entirety. In addition, in various embodiments, the virtual try-on system can implement a form of stray noise reduction between the repair and overlay steps. For example, the output of the segmentation and repair neural network may cause visual instability or lag that is significant and visible to the user over time. To mitigate instability, the virtual try-on system can display a mixture of a previously repaired image and a currently repaired image. Reducing stray noise estimates the motion of the existing glasses by comparing the previous and current segmentation masks to transform the previous repair results before mixing them with the current repair results.

[0054] refer to Figure 1-4 , shows an example user interface of a user device for navigating a user to virtually try on a pair of glasses. Figure 1 A user device 104 including a front-facing camera 108 is depicted. For example, a user may browse an e-commerce platform including a plurality of eyeglasses and select a pair of eyeglasses 112 for viewing. Figure 1-4 As shown, the user can select a style of glasses 37, and the user interface can display the price of the glasses. The user can also select a user interface button 116 to virtually try on a pair of glasses 112 selected.

[0055] like Figure 2As shown, once selected, the camera 108 can begin capturing images or videos of the user 120 wearing a pair of existing glasses 124. As previously described, if the user is not wearing any glasses, the virtual try-on system may not detect any glasses and continue to simply superimpose the selected pair of glasses 112 on the user 120.

[0056] Continue to refer Figure 3 As previously described, the virtual try-on system identifies a region of interest 128 including existing glasses 124 on the user 120 using a first neural network. As previously described, within the region of interest 128, a plurality of pixels are identified and segmented as including a frame of the existing glasses 124 or not including a frame. As described above, a percentage likelihood may be associated with each pixel in the region of interest 128, indicating how likely it is that the pixel includes a portion of a frame of the existing glasses 124 using a second neural network.

[0057] exist Figure 4 In the example of FIG. 1 , an image including glasses 132 superimposed on user 120 may be presented to the user. As discussed, after segmentation, the frames of existing glasses 124 may be inpainted using a third neural network to fill in those pixels with content that would be displayed in those pixels if no frames were there. The selected glasses 112 are then superimposed on user 120.

[0058] Reference now Figure 5 , a functional block diagram of an example architecture for implementing a virtual try-on module 504 is shown. The virtual try-on module 504 may be an application stored on the user device 104 and implemented on the user device 104 using at least one processor of the user device 104. The virtual try-on module 504 may include an image acquisition module 508, which receives a try-on request from a user interface of the user device 104 (e.g., by selecting Figure 1 Once the image acquisition module 508 receives a request from a user selection of a user interface item, the image acquisition module 508 may turn on the camera of the user device 104 and receive a real-time image from the camera of the user device 104. The image is forwarded to the glasses detection module 512.

[0059] In some embodiments, the glasses detection module 512 may implement a first neural network, as previously described, to detect an existing pair of glasses and identify a region of interest that includes the existing pair of glasses. The region of interest may include only the existing glasses and / or the frames of the existing glasses, as this is the area of ​​the one or more images that will be enhanced or adjusted. Because the camera of the user device 104 can provide a real-time video feed (e.g., a continuous plurality of images), the user may be constantly moving. Thus, the glasses detection module 512 may be configured to detect the existing glasses and track the glasses.

[0060] To this end, the first neural network can implement an object detector called yolov4-tiny, which is a small neural network capable of operating in real time on a user device 104 (e.g., a mobile phone, a tablet computer, etc.). The first neural network can also be capable of operating in real time on other devices, such as a laptop computer, a desktop computer, and / or one or more servers communicatively connected to the user device (e.g., via one or more networks). In some embodiments, to train the first neural network, the glasses detection module 512 can use the Darknet framework, then convert the model to Keras, and finally convert to CoreML format using a custom conversion pipeline. The first neural network can be trained for an image resolution of 416x416 to provide a compromise between quality and performance, but higher or lower resolutions can be used. The first neural network can be trained using a training dataset of approximately 2000 manually annotated images to identify locations and regions of interest in images of individuals wearing glasses, but larger or smaller training datasets can be used.

[0061] Once the region of interest is identified, the image and region can be forwarded to the segmentation module 516 to apply a second neural network to a plurality of pixels within the region of interest and determine, for each pixel, the likelihood that the pixel includes a portion of a frame of an existing pair of glasses. The second neural network can be a UNet neural network architecture with a ReLU activation function. The second neural network can be trained using the Keras framework and using the Adam optimizer and a binary cross entropy loss function. In various embodiments, the captured image identifying the region of interest can be resized to 256x256 and grayscale and then used as input to the second neural network. However, it should be understood that the resizing of the image can be omitted, or the image can be resized to other sizes and / or color formats or image types.

[0062] The output of the second neural network can be a binary mask that highlights the location of the eyeglass frame including the temples of the eyeglasses. In various embodiments, the binary mask is constructed by including those pixels having a resulting probability of including the frame that is higher than a threshold percentage (e.g., 50%). The size of the resulting binary mask can be adjusted using bicubic interpolation to match the original portion of the image. The second neural network can be trained using a training data set of approximately 73,000 pairs of images of individuals wearing glasses and manually identified binary masks of eyeglass frames, but other training sets can also be used. In various embodiments, if all pixels have a corresponding probability below a threshold (e.g., 30%), or the average of all probabilities is below a threshold, the segmentation module 516 can generate a reminder and send a reminder to the display module 520 to display on the user interface of the user device 104 that an existing pair of glasses was not found. Additionally or alternatively, the reminder or indication can be forwarded to the overlay module 524 to overlay the selected glasses on the user because the existing glasses were not found with sufficient confidence.

[0063] The binary mask of the frame of the existing glasses can be forwarded to the restoration module 528 that implements the third neural network. The third neural network can be implemented using AOT-GAN, which is a generated adversarial network. The specificity of the third neural network is a special AOT block that is optimized for larger missing areas and includes an enhanced discriminator that better distinguishes the detailed appearance of real and synthetic patches. The third neural network can be implemented on the Pytorch framework, where the VGG16 encoder is pre-trained on a set of images (approximately 6700 pairs) of individuals with binary masks and without glasses. Here again, other types of neural networks and / or training sets can be used.

[0064] In various embodiments, the repair module 528 can choose to repair each image or a subset of these images in the real-time images received from the user device 104 camera based on the type of user device 104. For example, if the user device 104 is an older model or has reduced processing power (e.g., the processing power is less than a threshold processing power), the repair module 528 can repair every third image and use the most recently repaired pixels for any image that has not been repaired. In some embodiments, the number of images repaired is based on the processing power of the user device. For example, if the processing power or computing power exceeds a first threshold, the repair is applied to each image. If the processing power or computing power of the user device is less than the first threshold but greater than the second threshold, the repair can be applied to every other image. In another example, if the processing power or computing power of the user device is less than the second threshold but greater than the third threshold, the repair can be applied to every third image.

[0065] In various embodiments, the inpainted image may be stored in an image database 532 that stores image threshold time periods to reduce spurious noise using previously inpainted images. The inpainted image may also be forwarded to a spurious noise reduction module 536 that combines the previously inpainted image with the current inpainted image to avoid lag in the image displayed to the user. The spurious noise reduction module 536 may implement a specific algorithm to blend the previous image with the current image. For example, for the first frame (i=1), the output mask is the same as the input mask, and the output image is the same as the inpainted image.

[0066] Otherwise, you can use OpenCV’s findTransformECC() function to find the representation of the mask M′ from the previous output i-1 To the current input mask M i The warping operation H(·) of the transformation is applied with OpenCV’s warpPerspective() function. The output mask can be M′ i , which is a mixture of the current input mask and the distorted version of the previous output mask: M′ i =α M M i +(1-α M )H(M′ i-1 ). If there is an inpainted image available, for example from the image database 532, the spurious noise reduction module 536 blends the previous image with the current image so that the area corresponding to the output mask is the current inpainted image Y i and the distorted version H(Y′) of the previous output image i-1 ). The rest of the image is composed of the repaired image Y i Given, as shown in the following equation: i =B(M′ i )⊙[α Y Y i +(1-α Y )H(Y′ i-1 )]+[1-B(M′ i )]⊙Y i , where B(·) represents a binarization operation and ⊙ represents a pixel-wise multiplication. However, if no inpainted image is available in the image database 532, the spurious noise reduction module 536 instead has a distorted version H(Y′) of the previously output image. i-1 ) corresponds to the output mask given by , and the rest of the image is the input image X i Given, as shown in the following equation: i =B(M′ i )⊙H(Y′ i-1 )+[1-B(M′ i )]⊙Xi .

[0067] In various embodiments, the stray noise reduction module 536 is optional. Once the image is repaired and the stray noise is reduced, the overlay module 524 imposes the selected pair of glasses indicated in the try-on request on the user. The overlay module 524 can obtain the corresponding pair of glasses from the glasses database 540, which is remote and accessible to the user device 104 via the distributed communication system 544. The overlay module 524 can forward the overlaid image to the display module 520 for display on the user interface of the user device 104.

[0068] Reference now Figure 6 , shows an example flow chart depicting virtual glasses removal and replacement execution. For example, Figure 6 The flowchart may be implemented on the user device 104. It should be understood that Figure 6 The flowchart of can be implemented in a distributed processing system, where multiple devices (eg, processors and / or computers) implement Figure 6 In some embodiments, control begins in response to receiving a try-on request. Control continues to 604 to turn on a camera of a user device associated with the try-on request. Control continues to 608 to obtain a real-time image or video feed from a camera, which may be included in the user device or in other cameras communicatively coupled to the user device. Control continues to 612 to identify a pair of glasses in the obtained image using a first neural network, as previously described. Control continues to 616 to determine whether a pair of glasses is detected.

[0069] If a pair of glasses is detected, control continues to 620. Otherwise, control continues to 624 to obtain selected glasses based on the try-on request. Returning to 620, control continues to segment a group of pixels corresponding to a region of interest for the detected pair of glasses. When glasses are detected, the first neural network can identify the region of interest. Control continues to 628 to determine, for each pixel, a likelihood that the pixel is part of a frame of a pair of glasses using a second neural network. Control continues to 632 to select a first pixel in the group of pixels. Then, control continues to 636 to determine whether the likelihood for the selected pixel is above a threshold. For example, as previously described, if the likelihood that a particular pixel includes a frame is above a threshold, then the pixel is identified as including a frame.

[0070] If so, control continues to 640 to indicate that the selected pixel is part of the frame. Otherwise, control continues to 644 to determine whether another pixel is in the pixel group. If the other pixel is in the pixel group, control continues to 648 to select the next pixel in the pixel group and return to 636. Otherwise, if the other pixel is not in the pixel group, control continues to 652 to inpaint each pixel indicated as comprising the frame using a third neural network. Then, control continues to 624 to obtain the selected glasses from the remote database based on the try-on request. Control continues to 656 to superimpose the selected glasses on the obtained image. Control continues to 660 to display the image on the user interface of the user device and returns to 608 to obtain another real-time image from the camera of the user device.

[0071] Although the above method refers to the flowchart shown, it is understandable that many other ways can be used to perform the actions associated with the method. For example, the order of some operations can be changed, and some operations described can be optional.

[0072] In addition, the methods and systems described herein may be embodied at least in part in the form of computer-implemented processes and devices for practicing these processes. The disclosed methods may also be embodied at least in part in the form of tangible, non-temporary machine-readable storage media encoded with computer program code. For example, the steps of the method may be embodied in hardware, in executable instructions executed by a processor (e.g., software), or in a combination of the two. The medium may include, for example, RAM, ROM, CD-ROM, DVD-ROM, BD-ROM, hard drive, flash memory, or any other non-temporary machine-readable storage medium. When the computer program code is loaded into a computer and executed by the computer, the computer becomes a device for practicing the method. The method may also be embodied at least in part in the form of a computer into which the computer program code is loaded or executed, making the computer a dedicated computer for practicing the method. When implemented on a general-purpose processor, the computer program code segment configures the processor to create a specific logic circuit. The method may alternatively be embodied at least in part in a dedicated integrated circuit for performing the method.

[0073] The term model used in this disclosure includes a data model created using machine learning. Machine learning may involve training a model in a supervised or unsupervised environment. Machine learning may include a model that can be trained to learn the relationship between various data groups. The machine learning model can be based on a set of algorithms designed to model abstractions in the data using multiple processing layers. The processing layers can be composed of nonlinear transformations. The model can include, for example, artificial intelligence, neural networks, deep convolutions, and recurrent neural networks. Such a neural network can be composed of trainable filters, transformations, projections, hashing, pooling, and regularization at various levels. The model can be used for large-scale relationship identification tasks. The model can be created using various open source and proprietary machine learning tools known to those skilled in the art.

[0074] In this application, including the following definitions, the term "module" or the term "controller" may be replaced by the term "circuit". The term "module" may refer to processor hardware (shared, dedicated, or group) that executes code and memory hardware (shared, dedicated, or group) that stores code executed by the processor hardware, and may be part of or include the processor hardware and the memory hardware.

[0075] The module may include one or more interface circuits. In some examples, the interface circuit(s) may implement a wired or wireless interface connected to a local area network (LAN) or a wireless personal area network (WPAN). Examples of LANs are the Institute of Electrical and Electronics Engineers (IEEE) standard 802.11-2016 (also known as the WIFI wireless networking standard) and the IEEE standard 802.3-2015 (also known as the Ethernet wired networking standard). Examples of WPANs are the Bluetooth wireless networking standard from the Bluetooth Special Interest Group and the IEEE standard 802.15.4.

[0076] Modules may communicate with other modules using (one or more) interface circuits. Although modules may be depicted as logically communicating directly with other modules in the present disclosure, modules may actually communicate through a communication system in various embodiments. A communication system includes physical and / or virtual network devices, such as hubs, switches, routers, and gateways. In some embodiments, a communication system is connected to or spans a wide area network (WAN) such as the Internet. For example, a communication system may include multiple LANs interconnected via the Internet or point-to-point dedicated lines using technologies including multi-protocol label switching (MPLS) and virtual private networks (VPNs).

[0077] The foregoing is provided to illustrate, explain and describe the embodiments of the present disclosure. Modifications and adjustments to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments shown herein, but should be given the widest scope consistent with the novel features and principles disclosed herein.

Claims

1. A system comprising: A computing device comprising at least one processor and communicatively coupled to the camera, the computing device being configured to, in response to receiving the request: capturing at least one image via the camera; detecting a first plurality of locations of a first object within the at least one image; segmenting a first plurality of locations of the first object by determining, for each location in the first plurality of locations, a likelihood that the corresponding location includes a portion of the first object; inpainting a second plurality of locations, each location in the second plurality of locations having an associated likelihood above a threshold that the corresponding location includes a portion of the first object; generating an enhanced image by superimposing a selected second object on the at least one image; and The augmented image is displayed on a user interface of the computing device.

2. The system according to claim 1, wherein: The first plurality of locations of the first object are pixels within the at least one image.

3. The system according to claim 1, wherein: The first plurality of locations includes more locations than the second plurality of locations.

4. The system according to claim 1, wherein: The first plurality of positions includes fewer positions than the second plurality of positions.

5. The system according to claim 1, wherein: The first plurality of locations and the second plurality of locations are the same.

6. The system according to claim 1, wherein: The computing device is configured to capture real-time video via a camera, and for a subset of a plurality of frames of the real-time video, inpaint a set of positions among a plurality of positions.

7. The system according to claim 6, wherein: The subset of the plurality of frames is based at least in part on a computing capability of the computing device.

8. The system according to claim 1, wherein: The computing device is configured to reduce stray noise by blending a previous image with a current image before generating the enhanced image.

9. The system according to claim 1, wherein: In response to detecting the absence of multiple locations of the first object, the computing device is configured to skip a segmentation step and a restoration step to generate the enhanced image.

10. The system according to claim 1, wherein: In response to the second plurality of locations having an associated likelihood below a lower threshold, the computing device is configured to skip a repair step to generate the enhanced image.

11. The system of claim 1 , further comprising a remote database configured to store a set of objects including the selected second object, wherein: The computing device is configured to obtain the selected second object from the remote database.

12. The system of claim 1, wherein: The selected second object is indicated in the request.

13. The system of claim 1, wherein: The computing device is a mobile computing device.

14. The system of claim 1, wherein: The first object is a pair of glasses.

15. The system of claim 1, wherein: The portion of the first object is a frame of a pair of glasses.

16. The system of claim 1, wherein: The computing device is configured to: detecting a first plurality of locations of the first object within the at least one image by implementing a first neural network; segmenting a first plurality of locations of the first object by determining, for each location in the first plurality of locations, a likelihood that the corresponding location includes a portion of the first object using a second neural network; and The second plurality of locations having an associated likelihood above the threshold that a corresponding location includes a portion of the first object is inpainted by implementing a third neural network.

17. A method comprising: capturing, in response to receiving the request, at least one image via the camera by a computing device including at least one processor and communicatively coupled to the camera; detecting a first plurality of locations of a first object within the at least one image; segmenting a first plurality of locations of the first object by determining, for each location in the plurality of locations, a likelihood that the corresponding location includes a portion of the first object; inpainting a second plurality of locations having an associated likelihood above a threshold that a corresponding location includes a portion of the first object; generating an enhanced image by superimposing a selected second object on the at least one image; as well as The augmented image is displayed on a user interface of the computing device.

18. The method according to claim 17, wherein: The detection is performed by a first module, the segmentation is performed by a second module, and the repair is performed by a third module.

19. The method according to claim 18, wherein: The first module is different from the second module and the third module.

20. The method according to claim 18, wherein: The second module is different from the first module and the third module.

21. The method according to claim 18, wherein: The third module is different from the first module and the second module.

22. The method according to claim 21, wherein: The first module and the second module are the same module.

23. The method of claim 18, further comprising: determining a computing capability of the computing device; capturing, by the camera, real-time video comprising a first plurality of frames; and A second plurality of locations in the second plurality of frames are inpainted, wherein the second plurality of frames is based on a computing capability of the computing device.

24. The method according to claim 23, wherein: The second plurality of frames is different from the first plurality of frames.

25. The method of claim 18, further comprising blending a previous image with a current image to reduce stray noise before generating the enhanced image.

26. The method of claim 18, further comprising skipping a segmentation step and a restoration step to generate the enhanced image in response to detecting a loss of a plurality of locations of the first object.

27. A non-transitory computer readable medium having stored thereon instructions, wherein: When the instructions are executed by at least one processor, the device performs operations including: capturing, in response to receiving the request, at least one image via the camera by a computing device including the at least one processor and communicatively coupled to the camera; detecting a first plurality of locations of a first object within the at least one image; segmenting a first plurality of locations of the first object by determining, for each location in the plurality of locations, a likelihood that the corresponding location includes a portion of the first object; inpainting a second plurality of locations having an associated likelihood above a threshold that a corresponding location includes a portion of the first object; generating an enhanced image by superimposing a selected second object on the at least one image; as well as The augmented image is displayed on a user interface of the computing device.

28. The non-transitory computer readable medium of claim 27, wherein: The detection is performed by a first module, the segmentation is performed by a second module, and the repair is performed by a third module.

29. The non-transitory computer readable medium of claim 28, wherein: The operations include: determining the computing capabilities of the device; capturing, by the camera, real-time video comprising a first plurality of frames; and The second plurality of locations in a second plurality of frames are repaired, wherein the second plurality of frames is based on a computing capability of the device.

30. The non-transitory computer readable medium of claim 28, wherein: The operation includes blending a previous image with a current image to reduce stray noise before generating the enhanced image.

Images