Devices, methods and computer programs for virtual eyeglasses try-on

Abstract

Devices, methods and computer programs for virtual eyeglasses try-on are disclosed. The invention allows virtual eyeglasses try-on with an adjustable lens degree. A lens degree indication included in input information based on which the virtual eyeglasses in an output image are generated may be adjusted by a user. Furthermore, at least an eye area of a detected face is modifiable based on the lens degree indication, thus providing the users intuitive information about how they look when wearing various prescription eyeglasses.

Description

[0001] DEVICES, METHODS AND COMPUTER PROGRAMS FOR VIRTUAL EYEGLASSES TRY-ON

[0002] TECHNICAL FIELD

[0003] The present disclosure relates to the field of object detection, and, more particularly, to virtual eyeglasses try-on, and related devices, methods and computer programs.

[0004] BACKGROUND

[0005] Typically, virtual try-on applications of eyeglasses attempt to align eyeglasses to human faces using face detection and alignment algorithms. Typically, existing virtual try-on applications of eyeglasses assume that the user is wearing zero-degree eyeglasses. However, in real life, users often wear prescription eyeglasses which are eyeglasses with a prescribed lens degree.

[0006] Accordingly, at least in some situations, there may be a need for virtual eyeglasses try-on that allows an adjustable lens degree.

[0007] SUMMARY

[0008] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0009] It is an object of the invention to allow virtual eyeglasses try-on. The foregoing and other objects are achieved by the features of the independent claims. Further implementation forms are apparent from the dependent claims, the description and the figures.

[0010] According to a first aspect, a device for virtual eyeglasses try-on is provided. The device is configured to obtain an input image comprising a face of a user. The device is further configured to perform face detection for the obtained input image. The device is further configured to obtain input information related to virtual eyeglasses. The input information comprises a lens degree indication for the virtual eyeglasses. The device is further configured to generate an output image comprising the face of the user detected by the performed face detection, the detected face wearing the virtual eyeglasses corresponding with the lens degree indication, such that at least an area including eyes of the detected face is modified in accordance with the lens degree indication. The device is further configured to display the generated output image. The present disclosure allows virtual eyeglasses try-on with an adjustable lens degree. The lens degree indication included in the input information based on which the virtual eyeglasses in the output image are generated may be adjusted by the user. Furthermore, at least the eye area of the detected face is modifiable based on the lens degree indication, thus providing the users intuitive information about how they look when wearing various prescription eyeglasses.

[0011] In an implementation form of the first aspect, the device is further configured to obtain the input information by detecting physical eyeglasses worn by the detected face in the obtained input image and estimating the lens degree indication based at least on an eye shape of the detected face. This implementation form allows automatically obtaining the lens degree indication from the input image.

[0012] In an implementation form of the first aspect, the device is further configured to obtain an auxiliary image comprising the face of the user without the physical eyeglasses, wherein the estimating of the lens degree indication is further based on an eye shape difference between the input image and the auxiliary image. This implementation form allows estimating the lens degree indication with enhanced accuracy.

[0013] In an implementation form of the first aspect, the device is further configured to perform the estimating of the lens degree indication via applying a neural network. This implementation form allows estimating the lens degree indication with enhanced efficiency.

[0014] In an implementation form of the first aspect, the device is further configured to obtain a confirmation or a correction for the estimated lens degree indication before the generation of the output image. This implementation form allows estimating the lens degree indication with enhanced accuracy.

[0015] In an implementation form of the first aspect, the device is further configured to remove the detected physical eyeglasses from the detected face before the generation of the output image. This implementation form allows an enhanced user experience virtual eyeglasses try-on, as the generated virtual eyeglasses rather than the physical eyeglasses are visible.

[0016] In an implementation form of the first aspect, the device is further configured to remove the detected physical eyeglasses via applying an eyeglasses removal generative adversarial network. This implementation form allows removing the detected physical eyeglasses with enhanced efficiency.

[0017] In an implementation form of the first aspect, the device is further configured to obtain the input information comprising the lens degree indication from user input. This implementation form allows the user to manually enter the lens degree indication. In an implementation form of the first aspect, the device is further configured to obtain at least one visual virtual eyeglasses parameter. The output image is further generated such that the virtual eyeglasses worn by the detected face are modified in accordance with the obtained at least one visual virtual eyeglasses parameter. This implementation form allows an enhanced user experience virtual eyeglasses try-on, as the user is able to change various parameters affecting the visual appearance of the virtual eyeglasses.

[0018] In an implementation form of the first aspect, the obtained at least one visual virtual eyeglasses parameter comprises at least one of an eyeglasses type, a color of a lens, a reflective index of the lens, or a light-sensitivity of the lens. This implementation form allows an enhanced user experience virtual eyeglasses try-on, as the user is able to change various parameters affecting the visual appearance of the virtual eyeglasses.

[0019] In an implementation form of the first aspect, the device is further configured to obtain at least one visual face parameter. The output image is further generated such that the detected face is modified in accordance with the obtained at least one visual face parameter. This implementation form allows an enhanced user experience virtual eyeglasses try-on, as the user is able to change various parameters affecting the visual appearance of the face of the user.

[0020] In an implementation form of the first aspect, the obtained input information further comprises at least one of a distance between pupils or a degree of astigmatism. This implementation form allows estimating the lens degree indication with enhanced accuracy and generating an output image with a more realistic visual appearance of the face of the user.

[0021] In an implementation form of the first aspect, the device is further configured to generate and display a comparison image in addition to the generated output image. The comparison image comprises the detected face without the virtual eyeglasses. This implementation form allows an enhanced user experience virtual eyeglasses try-on, as the users are able to easily compare their looks with and without the virtual eyeglasses.

[0022] According to a second aspect, a method for virtual eyeglasses try-on is provided. The method comprises obtaining, by a device for virtual eyeglasses try-on, an input image comprising a face of a user. The method further comprises performing, by the device, face detection for the obtained input image. The method further comprises obtaining, by the device, input information related to virtual eyeglasses. The input information comprises a lens degree indication for the virtual eyeglasses. The method further comprises generating, by the device, an output image comprising the face of the user detected by the performed face detection, the detected face wearing the virtual eyeglasses corresponding with the lens degree indication, such that at least an area including eyes of the detected face is modified in accordance with the lens degree indication. The method further comprises displaying, by the device, the generated output image. The present disclosure allows virtual eyeglasses try-on with an adjustable lens degree. The lens degree indication included in the input information based on which the virtual eyeglasses in the output image are generated may be adjusted by the user. Furthermore, at least the eye area of the detected face is modifiable based on the lens degree indication, thus providing the users intuitive information about how they look when wearing various prescription eyeglasses.

[0023] In an implementation form of the second aspect, the obtaining of the input information comprises detecting, by the device, physical eyeglasses worn by the detected face in the obtained input image and estimating, by the device, the lens degree indication based at least on an eye shape of the detected face. This implementation form allows automatically obtaining the lens degree indication from the input image.

[0024] In an implementation form of the second aspect, the method further comprises obtaining, by the device, an auxiliary image comprising the face of the user without the physical eyeglasses. The estimating of the lens degree indication is further based on an eye shape difference between the input image and the auxiliary image. This implementation form allows estimating the lens degree indication with enhanced accuracy.

[0025] In an implementation form of the second aspect, the estimating of the lens degree indication is performed, by the device, via applying a neural network. This implementation form allows estimating the lens degree indication with enhanced efficiency.

[0026] In an implementation form of the second aspect, the method further comprises obtaining, by the device, a confirmation or a correction for the estimated lens degree indication before the generation of the output image. This implementation form allows estimating the lens degree indication with enhanced accuracy.

[0027] In an implementation form of the second aspect, the method further comprises removing, by the device, the detected physical eyeglasses from the detected face before the generation of the output image. This implementation form allows an enhanced user experience virtual eyeglasses try-on, as the generated virtual eyeglasses rather than the physical eyeglasses are visible.

[0028] In an implementation form of the second aspect, the removing of the detected physical eyeglasses is performed, by the device, via applying an eyeglasses removal generative adversarial network. This implementation form allows removing the detected physical eyeglasses with enhanced efficiency. In an implementation form of the second aspect, the input information comprising the lens degree indication is obtained, by the device, from user input. This implementation form allows the user to manually enter the lens degree indication.

[0029] In an implementation form of the second aspect, the method further comprises obtaining, by the device, at least one visual virtual eyeglasses parameter. The output image is further generated such that the virtual eyeglasses worn by the detected face are modified in accordance with the obtained at least one visual virtual eyeglasses parameter. This implementation form allows an enhanced user experience virtual eyeglasses try-on, as the user is able to change various parameters affecting the visual appearance of the virtual eyeglasses.

[0030] In an implementation form of the second aspect, the obtained at least one visual virtual eyeglasses parameter comprises at least one of an eyeglasses type, a color of a lens, a reflective index of the lens, or a light-sensitivity of the lens. This implementation form allows an enhanced user experience virtual eyeglasses try-on, as the user is able to change various parameters affecting the visual appearance of the virtual eyeglasses.

[0031] In an implementation form of the second aspect, the method further comprises obtaining, by the device, at least one visual face parameter. The output image is further generated such that the detected face is modified in accordance with the obtained at least one visual face parameter. This implementation form allows an enhanced user experience virtual eyeglasses try-on, as the user is able to change various parameters affecting the visual appearance of the face of the user.

[0032] In an implementation form of the second aspect, the obtained input information further comprises at least one of a distance between pupils or a degree of astigmatism. This implementation form allows estimating the lens degree indication with enhanced accuracy and generating an output image with a more realistic visual appearance of the face of the user.

[0033] In an implementation form of the second aspect, the method further comprises generating and displaying, by the device, a comparison image in addition to the generated output image. The comparison image comprises the detected face without the virtual eyeglasses. This implementation form allows an enhanced user experience virtual eyeglasses try-on, as the users are able to easily compare their looks with and without the virtual eyeglasses.

[0034] According to a third aspect, a computer program product is provided. The computer program product comprises program code configured to perform a method according to the second aspect, when the program code is executed on a computer. The present disclosure allows virtual eyeglasses try-on with an adjustable lens degree. The lens degree indication included in the input information based on which the virtual eyeglasses in the output image are generated may be adjusted by the user. Furthermore, at least the eye area of the detected face is modifiable based on the lens degree indication, thus providing the users intuitive information about how they look when wearing various prescription eyeglasses.

[0035] Many of the attendant features will be more readily appreciated as they become better understood by reference to the following detailed description considered in connection with the accompanying drawings.

[0036] DESCRIPTION OF THE DRAWINGS

[0037] In the following, example embodiments are described in more detail with reference to the attached figures and drawings, in which:

[0038] Fig. 1 includes diagrams illustrating eyeglasses with a variety of lens degrees;

[0039] Fig. 2 is a block diagram illustrating a device for virtual eyeglasses try-on;

[0040] Fig. 3 is a diagram illustrating a general overview of virtual eyeglasses try-on with an adjustable lens degree;

[0041] Fig. 4 is a diagram illustrating an augmented reality -based eyeglasses try-on block of Fig. 3 in more detail;

[0042] Fig. 5 is a diagram illustrating a first embodiment;

[0043] Fig. 6 is a diagram further illustrating the first embodiment;

[0044] Fig. 7 is a diagram illustrating a second embodiment;

[0045] Fig. 8 is a diagram further illustrating the second embodiment;

[0046] Fig. 9A is a flow chart illustrating a method; and

[0047] Fig. 9B is a flow chart illustrating another method.

[0048] In the following, identical reference signs refer to identical or at least functionally equivalent features.

[0049] DETAILED DESCRIPTION

[0050] In the following description, reference is made to the accompanying drawings, which form part of the disclosure, and in which are shown, by way of illustration, specific aspects in which the invention may be placed. It is understood that other aspects may be utilized, and structural or logical changes may be made without departing from the scope of the invention. The following detailed description, therefore, is not to be taken in a limiting sense, as the scope of the invention is defined in the appended claims.

[0051] For instance, it is understood that a disclosure in connection with a described method may also hold true for a corresponding device or system configured to perform the method and vice versa. For example, if a specific method step is described, a corresponding device may include a unit to perform the described method step, even if such unit is not explicitly described or illustrated in the figures. On the other hand, for example, if a specific apparatus is described based on functional units, a corresponding method may include a step performing the described functionality, even if such step is not explicitly described or illustrated in the figures. Further, it is understood that the features of the various example aspects described herein may be combined with each other, unless specifically noted otherwise.

[0052] Eyeglasses may change the appearance of a user wearing them depending on their lens degree. Particularly eye shape and / or an area including eyes may be changed. Fig. 1 includes diagrams illustrating this. Diagram 100 A illustrates a user wearing eyeglasses with farsighted lenses, diagram 100B illustrates a user wearing eyeglasses with zero-degree lenses, and diagram 100C illustrates a user wearing eyeglasses with nearsighted lenses. As can be seen, the eye shape differs in each of diagrams 100A to 100C.

[0053] In recent years, interest in various face related applications has increased. Among these applications, augmented reality (AR) -based virtual eyeglasses try-on is a new way to shop for eyeglasses online. This can help customers to find out how they will look like with various eyeglasses, and thus improve the interaction between buyers and online sellers.

[0054] At least some embodiments of the present disclosure may provide virtual try-on of eyeglasses for users with prescription lenses. Lens degree may be specified, e.g., via user input (e.g., via a slide bar), or via estimation performed, e.g., by a pre-trained neural network, such as a convolutional neural network (CNN) -based pre-trained neural network.

[0055] At least some embodiments of the present disclosure may allow easy integration, e.g., into a website or a mobile application.

[0056] At least some embodiments of the present disclosure may be executed in realtime, thereby functioning as a virtual mirror showing the user’s face wearing virtual eyeglasses.

[0057] At least some embodiments of the present disclosure may allow removal of physical eyeglasses form an output image. That is, the user may wear the physical eyeglasses which are replaced with selected virtual eyeglasses in the output image without the user having to actually take off the physical eyeglasses.

[0058] At least some embodiments of the present disclosure may allow photo realistic eye shape generation in accordance with lens degrees - from minus to plus degrees, with or without eyeglasses.

[0059] Diagram 300 of Fig. 3 illustrates a general overview of the disclosed virtual eyeglasses try-on with an adjustable lens degree. The lens degree may be given by a user at block 304 (using, e.g., a slide bar provided by a suitable user interface) or evaluated by, e.g., an algorithm at blocks 302, 303 from an input image obtained at block 301.

[0060] If the input image comprises the user wearing physical eyeglasses, at block 305, the physical eyeglasses may be removed to be replaced with virtual eyeglasses.

[0061] At block 303, the lens degree may be estimated, e.g., by comparing the user’s eye shape with and without the physical eyeglasses.

[0062] At block 306, the user may provide additional input, such as visual virtual eyeglasses parameters and / or visual face parameters described in more detail below.

[0063] At block 307, an output image may be generated. The generation of the output image is described in more detail below in connection with Fig. 2. Furthermore, an implementation example of block 307 is described in detail below in connection with Fig. 4.

[0064] At block 308, the generated output image may be displayed to the user.

[0065] Next, example embodiments of a device 200 for virtual eyeglasses try-on are described based on Fig. 2. Some of the features of the described devices are optional features which may provide further advantages.

[0066] Fig. 2 is a block diagram illustrating the device 200 for virtual eyeglasses try-on, according to an embodiment of the present disclosure. In an embodiment, the device 200 may comprise any of various types of client devices used directly by an end user entity and capable of communication in a wireless network, such as user equipment (UE). Such devices include but are not limited to smartphones, tablet computers, smart watches, lap top computers, intemet- of-things (loT) devices, etc. In another embodiment, the device 200 may comprise a server device (e.g. a server device providing one or more cloud services) communicatively connected (e.g., wirelessly) to a client device.

[0067] The device 200 comprises at least one processor or a processing unit 202 and at least one memory 204 coupled to the at least one processor 202, which may be used to implement the functionalities described later in more detail. The device 200 may further comprise a digital camera 206. The camera 206 may be configured, e.g., to capture input images and / or auxiliary images described later in more detail. The camera 206 may comprise, e.g., a frontfacing camera or a rear-facing camera. The device 200 may further comprise a display 208 configured to, e.g., display output images and / or prompts to a user, described later in more detail. In an example, the display 208 may comprise a touch screen.

[0068] The device 200 may also include other elements, such as a transceiver configured to enable the device 200 to transmit and / or receive information to / from other devices, and / or input means (such as a physical or virtual keyboard), as well as other elements not shown in Fig. 2. In one example, the device 200 may use the transceiver to transmit or receive signalling information and data in accordance with at least one cellular communication protocol. The transceiver may be configured to provide at least one wireless radio connection, such as for example a 3GPP mobile broadband connection (e.g., 5G). The transceiver may comprise, or be configured to be coupled to, at least one antenna to transmit and / or receive radio frequency signals.

[0069] The at least one processor 202 may include, e.g., one or more of various processing devices, such as a co-processor, a microprocessor, a controller, a digital signal processor (DSP), a processing circuitry with or without an accompanying DSP, or various other processing devices including integrated circuits such as, for example, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a microcontroller unit (MCU), a hardware accelerator, a special-purpose computer chip, or the like.

[0070] The memory 204 may be configured to store e.g. computer programs and the like. The memory may include one or more volatile memory devices, one or more non-volatile memory devices, and / or a combination of one or more volatile memory devices and non-volatile memory devices. For example, the memory 204 may be embodied as semiconductor memories (such as mask ROM, PROM (programmable ROM), EPROM (erasable PROM), flash ROM, RAM (random access memory), etc.).

[0071] The device 200 is configured to obtain an input image comprising a face of a user. The input image may comprise, for example, a still image or an image frame of a video stream, e.g. captured by the camera 206.

[0072] The device 200 is further configured to perform face detection for the obtained input image.

[0073] The device 200 is further configured to obtain input information related to virtual eyeglasses. The input information comprises a lens degree indication for the virtual eyeglasses. At least in some embodiments, the obtained input information may further comprise a distance between pupils and / or a degree of astigmatism. For example, the lens degree indication may include one or more values in an eyeglass prescription of the user, such as one or more diopter related values.

[0074] The device 200 is further configured to generate an output image. The output image comprises the face of the user detected by the performed face detection, the detected face wearing the virtual eyeglasses corresponding with the lens degree indication, such that at least an area including eyes of the detected face is modified in accordance with the lens degree indication.

[0075] The device 200 is further configured to display the generated output image, e.g., on the display 208.

[0076] In a first embodiment, the device 200 may be further configured to obtain the input information by detecting physical eyeglasses worn by the detected face in the obtained input image and estimating the lens degree indication based at least on an eye shape of the detected face. For example, the device 200 may be further configured to perform the estimating of the lens degree indication via applying a neural network.

[0077] At least in some implementations, the device 200 may be further configured to display guidance information in response to failing to detect the physical eyeglasses. For example, the guidance information may comprise a prompt to make sure the device 200 is stable and / or to make sure the face in view of the camera 206.

[0078] In the first embodiment, the device 200 may be further configured to obtain an auxiliary image comprising the face of the user without the physical eyeglasses. The estimating of the lens degree indication may further be based on an eye shape difference between the input image and the auxiliary image.

[0079] In the first embodiment, the device 200 may be further configured to obtain a confirmation or a correction for the estimated lens degree indication before the generation of the output image.

[0080] In the first embodiment, the device 200 may be further configured to remove the detected physical eyeglasses from the detected face before the generation of the output image. For example, the device 200 may be further configured to remove the detected physical eyeglasses via applying an eyeglasses removal generative adversarial network (ERGAN).

[0081] In a second embodiment, the device 200 may be further configured to obtain the input information comprising the lens degree indication from user input (e.g., via a text entry box or a slide bar).

[0082] At least in some implementations, the user may be prompted to take off the physical eyeglasses (e.g., to allow the display of the virtual eyeglasses on the user’s face without the physical eyeglasses), and / or to wear the physical eyeglasses (e.g., to allow the lens degree estimation of the first embodiment).

[0083] In the first and / or the second embodiment, the device 200 may be further configured to obtain at least one visual virtual eyeglasses parameter. The output image may further be generated such that the virtual eyeglasses worn by the detected face are modified in accordance with the obtained at least one visual virtual eyeglasses parameter. For example, the obtained at least one visual virtual eyeglasses parameter may comprise an eyeglasses type, a color of a lens, a reflective index of the lens, and / or a light-sensitivity of the lens.

[0084] In the first and / or the second embodiment, the device 200 may be further configured to obtain at least one visual face parameter. The output image may further be generated such that the detected face is modified in accordance with the obtained at least one visual face parameter. For example, the obtained at least one visual face parameter may comprise eye color, hair color, and / or skin tone.

[0085] In the first and / or the second embodiment, the device 200 may be further configured to generate and display a comparison image in addition to the generated output image, e.g., simultaneously or successively. The comparison image may comprise the detected face without the virtual eyeglasses.

[0086] Diagram 500 of Fig. 5 and diagram 600 of Fig. 6 further illustrate the first embodiment.

[0087] As described above in connection with Fig. 2, the first embodiment allows a virtual eyeglasses try-on with an adjustable lens degree, e.g., when the user is wearing physical eyeglasses, as shown at 200 A, and wants to find out how he / she looks like wearing various virtual eyeglasses, as shown at 200B. When the physical eyeglasses are present in the input image / video (e.g., the user’s portrait) at 200A, the lens degree will be estimated and used to generate the user’s face with the virtual eyeglasses. As discussed above in more detail, various virtual eyeglasses may be selected / switched, block 501, and the lens degree estimate provided by the lens degree estimation may be edited, block 502.

[0088] As also discussed above in more detail, a face detector may be applied to an input image or video of the user.

[0089] When no face is detected in the input image / video, it may be that the virtual eyeglasses try-on cannot proceed. In this case, the device 200 may be further configured to prompt the user to enter the lens degree, thereby triggering a switch to the second embodiment. Alter- natively / additionally, as discussed above, at least in some implementations, the device 200 may be further configured to display guidance information in response to failing to detect the physical eyeglasses. For example, the guidance information may comprise a prompt to make sure the device 200 is stable and / or to make sure the face in view of the camera 206.

[0090] When a face is detected in the input image / video, the eyeglasses detector 601 may be launched (e.g., for each detected face in the input image / video) to check if the user(s) is / are already wearing eyeglasses or not. When eyeglasses are detected, then the lens degree estimation 602 may be performed, e.g., using a trained CNN architecture.

[0091] The estimated lens degree may be confirmed by the user. Additionally, the user may edit the estimated lens degree value, e.g., to correspond with an eyeglasses prescription, block 603.

[0092] Then, eyeglasses removal 604 may be performed, e.g., by using a trained ERGAN to generate the user’s face without glasses.

[0093] The ERGAN output, the lens degree value(s), and / or the eyeglasses style or the like (block 605) may be input to the AR glasses degree block 606 to generate the user’s face with the virtual eyeglasses.

[0094] Diagram 700 of Fig. 7 and diagram 800 of Fig. 8 further illustrate the first embodiment.

[0095] As described above in connection with Fig. 2, the second embodiment allows a virtual eyeglasses try-on with an adjustable lens degree, e.g., when the user is not wearing physical eyeglasses, as shown at 200C, and wants to find out how he / she looks like wearing various virtual eyeglasses, as shown at 200D. When the physical eyeglasses are absent in the input image / video (e.g., the user’s portrait) at 200C, the lens degree will be obtained from user input and used to generate the user’s face with the virtual eyeglasses. As discussed above in more detail, various virtual eyeglasses may be selected / switched, block 701. An objective of the second embodiment includes providing a good virtual eyeglasses experience to the user with manual lens degree input.

[0096] As also discussed above in more detail, a face detector may be applied to an input image or video of the user.

[0097] When no face is detected in the input image / video, it may be that the virtual eyeglasses try-on cannot proceed. As discussed above, at least in some implementations, the device 200 may be further configured to display guidance information in response to failing to detect the physical eyeglasses. For example, the guidance information may comprise a prompt to make sure the device 200 is stable and / or to make sure the face in view of the camera 206.

[0098] When a face is detected in the input image / video, the eyeglasses detector 801 may be launched (e.g., for each detected face in the input image / video) to check if the user(s) is / are already wearing eyeglasses or not.

[0099] When eyeglasses are not detected, then the user may be prompted to manually enter the lens degree, block 802. The input image or video, the entered lens degree value(s), and / or the eyeglasses style or the like (block 803) may be input to the AR glasses degree block 804 to generate the user’s face with the virtual eyeglasses.

[0100] Fig. 4 illustrates the AR -based eyeglasses try-on block 307 of Fig. 3 in more detail.

[0101] For example, the block 307 may take three inputs: user’s face image 401, the lens degree 402 of the user, and the eyeglasses model 403.

[0102] At operation 307 A, face and facial key point detection may be performed for the images with faces.

[0103] At operation 307B, scaling of the face region with the lens degree (either plus or minus) inside the eyeglasses frame may be performed for the faces.

[0104] At operation 307C, the face region within the eyeglasses frame inner contour may be cropped.

[0105] At operation 307D, estimation of the eye pose may be performed.

[0106] At operation 307E, a three-dimensional (3D) face transform module may handle face pose estimation, eye point alignment, and 3D eyeglasses rendering.

[0107] At operation 307F, physically based rendering (PBR) and image-based lighting (IBL, operation 307G) may be used to generate realistic effects for the selected eyeglasses frame, which is aligned to the face of the user. PBR refers to a computer graphics approach that seeks to render images in a way that models the flow of light in the real world. IBL refers to a 3D rendering technique which involves capturing an omnidirectional representation of real- world light information as an image. This image may then be used to simulate the lighting for objects in a scene. Output 404 may include the user’s face with the virtual eyeglasses.

[0108] Fig. 9A is a flow chart illustrating a method 900A for virtual eyeglasses try-on, according to an embodiment of the present disclosure.

[0109] At operation 901 A, the device 200 for virtual eyeglasses try-on obtains the input image comprising the face of the user. In the embodiment of Fig. 9A, the user may be wearing physical eyeglasses in the obtained input image. At least in some embodiments, the user may be prompted to wear the physical eyeglasses before operation 901A, e.g., via a user interface of the device 200.

[0110] At optional operation 902, the device 200 may obtain the at least one visual virtual eyeglasses parameter.

[0111] At optional operation 903, the device 200 may obtain the at least one visual face parameter. At operation 904, the device 200 performs the face detection for the obtained input image.

[0112] Next, the device 200 obtains the input information related to the virtual eyeglasses. The input information comprises a lens degree indication for the virtual eyeglasses. More specifically, at optional operation 905 Al, the device 200 may obtain the auxiliary image comprising the face of the user without the physical eyeglasses. At least in some embodiments, the user may be prompted to take off the physical eyeglasses before operation 905A, e.g., via the user interface of the device 200. At optional operation 905 A2, the device 200 may detect the physical eyeglasses worn by the detected face in the obtained input image and estimate, at optional operation 905 A3, the lens degree indication based at least on the eye shape of the detected face. At optional operation 905 A4, the device 200 may obtain the confirmation or correction for the estimated lens degree indication.

[0113] At optional operation 905 A5, the device 200 may remove the detected physical eyeglasses from the detected face.

[0114] At operation 906, the device 200 generates the output image comprising the face of the user detected by the performed face detection, the detected face wearing the virtual eyeglasses corresponding with the lens degree indication, such that at least the area including eyes of the detected face is modified in accordance with the lens degree indication.

[0115] At operation 907, the device 200 displays the generated output image.

[0116] At optional operation 908, the device 200 the comparison image in addition to the generated output image. The comparison image comprises the detected face without the virtual eyeglasses.

[0117] The method 900A may be performed by the device 200. The operations 901-908 can, for example, be performed by the at least one processor 202 and the memory 204. Further features of the method 900A directly result from the functionalities and parameters of the device 200 and thus are not repeated here. The method 900A can be performed by computer programs.

[0118] The operations 901 A-908 of Fig. 9 A may be carried out in any suitable order, or simultaneously where appropriate. For example, operation 903 may be carried out after operation 905A1. For another example, operation 905A1 may be carried out before operation 901 A.

[0119] Fig. 9B is a flow chart illustrating a method 900B for virtual eyeglasses try-on, according to an embodiment of the present disclosure.

[0120] At operation 90 IB, the device 200 for virtual eyeglasses try-on obtains the input image comprising the face of the user. In the embodiment of Fig. 9B, the user may be without physical eyeglasses in the obtained input image. At optional operation 902, the device 200 may obtain the at least one visual virtual eyeglasses parameter.

[0121] At optional operation 903, the device 200 may obtain the at least one visual face parameter.

[0122] At operation 904, the device 200 performs the face detection for the obtained input image.

[0123] Next, the device 200 obtains the input information related to the virtual eyeglasses. The input information comprises a lens degree indication for the virtual eyeglasses. More specifically, at optional operation 905B, the device 200 may obtain the input information comprising the lens degree indication from user input.

[0124] At operation 906, the device 200 generates the output image comprising the face of the user detected by the performed face detection, the detected face wearing the virtual eyeglasses corresponding with the lens degree indication, such that at least the area including eyes of the detected face is modified in accordance with the lens degree indication.

[0125] At operation 907, the device 200 displays the generated output image.

[0126] At optional operation 908, the device 200 the comparison image in addition to the generated output image. The comparison image comprises the detected face without the virtual eyeglasses.

[0127] The method 900B may be performed by the device 200. The operations 901-908 can, for example, be performed by the at least one processor 202 and the memory 204. Further features of the method 900B directly result from the functionalities and parameters of the device 200 and thus are not repeated here. The method 900B can be performed by computer programs.

[0128] The operations 901B-908 of Fig. 9B may be carried out in any suitable order, or simultaneously where appropriate. For example, operation 903 may be carried out after operation 905B. For another example, operation 905B may be carried out before operation 901B.

[0129] The device 200 may comprise means for performing at least one method described herein. In one example, the means may comprise the at least one processor 202, and the at least one memory 204 including program code configured to, when executed by the at least one processor, cause the device 200 to perform the method.

[0130] The functionality described herein can be performed, at least in part, by one or more computer program product components such as software components. According to an embodiment, the device 200 may comprise a processor or processor circuitry, such as for example a microcontroller, configured by the program code when executed to execute the embod- iments of the operations and functionality described. Alternatively, or in addition, the functionality described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), and Graphics Processing Units (GPUs).

[0131] Any range or device value given herein may be extended or altered without losing the effect sought. Also, any embodiment may be combined with another embodiment unless explicitly disallowed.

[0132] Although the subject matter has been described in language specific to structural features and / or acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as examples of implementing the claims and other equivalent features and acts are intended to be within the scope of the claims.

[0133] It will be understood that the benefits and advantages described above may relate to one embodiment or may relate to several embodiments. The embodiments are not limited to those that solve any or all of the stated problems or those that have any or all of the stated benefits and advantages. It will further be understood that reference to 'an' item may refer to one or more of those items.

[0134] The steps of the methods described herein may be carried out in any suitable order, or simultaneously where appropriate. Additionally, individual blocks may be deleted from any of the methods without departing from the spirit and scope of the subject matter described herein. Aspects of any of the embodiments described above may be combined with aspects of any of the other embodiments described to form further embodiments without losing the effect sought.

[0135] The term 'comprising' is used herein to mean including the method, blocks or elements identified, but that such blocks or elements do not comprise an exclusive list and a method or apparatus may contain additional blocks or elements.

[0136] It will be understood that the above description is given by way of example only and that various modifications may be made by those skilled in the art. The above specification, examples and data provide a complete description of the structure and use of exemplary embodiments. Although various embodiments have been described above with a certain degree of particularity, or with reference to one or more individual embodiments, those skilled in the art could make numerous alterations to the disclosed embodiments without departing from the spirit or scope of this specification.

Claims

CLAIMS1. A device (200) for virtual eyeglasses try-on, configured to: obtain an input image comprising a face of a user; perform face detection for the obtained input image; obtain input information related to virtual eyeglasses, wherein the input information comprises a lens degree indication for the virtual eyeglasses; generate an output image comprising the face of the user detected by the performed face detection, the detected face wearing the virtual eyeglasses corresponding with the lens degree indication, such that at least an area including eyes of the detected face is modified in accordance with the lens degree indication; and display the generated output image.

2. The device (200) according to claim 1, further configured to obtain the input information by detecting physical eyeglasses worn by the detected face in the obtained input image and estimating the lens degree indication based at least on an eye shape of the detected face.

3. The device (200) according to claim 2, further configured to obtain an auxiliary image comprising the face of the user without the physical eyeglasses, wherein the estimating of the lens degree indication is further based on an eye shape difference between the input image and the auxiliary image.

4. The device (200) according to claim 2 or 3, further configured to perform the estimating of the lens degree indication via applying a neural network.

5. The device (200) according to any of claims 2 to 4, further configured to obtain a confirmation or a correction for the estimated lens degree indication before the generation of the output image.

6. The device (200) according to any of claims 2 to 5, further configured to remove the detected physical eyeglasses from the detected face before the generation of the output image.

7. The device (200) according to claim 6, further configured to remove the detected physical eyeglasses via applying an eyeglasses removal generative adversarial network.

8. The device (200) according to claim 1, further configured to obtain the input information comprising the lens degree indication from user input.

9. The device (200) according to any of claims 1 to 8, further configured to obtain at least one visual virtual eyeglasses parameter, wherein the output image is further generated such that the virtual eyeglasses worn by the detected face are modified in accordance with the obtained at least one visual virtual eyeglasses parameter.

10. The device (200) according to claim 9, wherein the obtained at least one visual virtual eyeglasses parameter comprises at least one of an eyeglasses type, a color of a lens, a reflective index of the lens, or a light-sensitivity of the lens.

11. The device (200) according to any of claims 1 to 10, further configured to obtain at least one visual face parameter, wherein the output image is further generated such that the detected face is modified in accordance with the obtained at least one visual face parameter.

12. The device (200) according to any of claims 1 to 11, wherein the obtained input information further comprises at least one of a distance between pupils or a degree of astigmatism.

13. The device (200) according to any of claims 1 to 12, further configured to generate and display a comparison image in addition to the generated output image, the comparison image comprising the detected face without the virtual eyeglasses.

14. A method (900A, 900B) for virtual eyeglasses try-on, comprising: obtaining (901 A, 901B), by a device for virtual eyeglasses try-on, an input image comprising a face of a user; performing (904), by the device, face detection for the obtained input image;obtaining (905 Al -905 A4, 905B), by the device, input information related to virtual eyeglasses, wherein the input information comprises a lens degree indication for the virtual eyeglasses; generating (906), by the device, an output image comprising the face of the user detected by the performed face detection, the detected face wearing the virtual eyeglasses corresponding with the lens degree indication, such that at least an area including eyes of the detected face is modified in accordance with the lens degree indication; and displaying (907), by the device, the generated output image.

15. A computer program product comprising program code configured to perform a method (900 A, 900B) according to claim 14, when the program code is executed on a computer.

Images