Apparatus, method and computer program for virtual glasses try-on

Abstract

Apparatuses, methods, and computer programs for virtual glasses try-on are disclosed. According to the invention, trying-on of the virtual glasses with adjustable lens degrees is supported. The lens power included in the input information indicates that the lens power can be adjusted by the user, and the virtual glasses in the output image are generated based on the input information. Further, at least one eye region of the detected face may be modified based on the lens power indication, thereby providing intuitive information to the user regarding their pattern when wearing various prescription eyeglasses.

Description

Technical Field

[0001] The present invention relates to the field of object detection, and more particularly, to virtual glasses trying on, and related devices, methods and computer programs. Background Art

[0002] Typically, glasses virtual try-on applications attempt to align glasses to a person's face using facial detection and alignment algorithms. Typically, existing glasses virtual try-on applications assume that the user is wearing zero-degree glasses. However, in real life, users often wear prescription glasses, i.e., glasses with a lens power after optometry.

[0003] Therefore, at least in some cases, it may be desirable to try on virtual glasses that allow for adjustable lens power. Summary of the invention

[0004] The Summary briefly introduces some concepts that will be further described in the Detailed Description. The Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0005] The object of the present invention is to support virtual eyeglass try-on. The above and other objects are achieved by the features of the independent claims. Other implementations are apparent from the dependent claims, the description and the drawings.

[0006] According to a first aspect, a device for trying on virtual glasses is provided. The device is used to obtain an input image including a user's face. The device is also used to perform facial detection on the obtained input image. The device is also used to obtain input information related to virtual glasses. The input information includes a lens power indication of the virtual glasses. The device is also used to generate an output image, the output image including the face of the user detected by the performed facial detection, the detected face wearing the virtual glasses corresponding to the lens power indication, thereby modifying at least one area including the eyes of the detected face according to the lens power indication. The device is also used to display the generated output image. The present invention supports trying on virtual glasses with adjustable lens power. The user can adjust the lens power indication included in the input information, and the virtual glasses in the output image are generated based on the input information. In addition, at least the eye area of ​​the detected face can be modified based on the lens power indication, thereby providing users with intuitive information about what they look like when wearing various prescription glasses.

[0007] In an implementation of the first aspect, the device is further configured to obtain input information by detecting physical glasses worn by the detected face in the acquired input image and estimating a lens power indication based at least on an eye shape of the detected face. This implementation supports automatically obtaining the lens power indication from the input image.

[0008] In an implementation of the first aspect, the device is further configured to acquire an auxiliary image, the auxiliary image comprising a face of a user not wearing physical glasses, wherein the estimation of the lens power indication is further based on an eye shape difference between the input image and the auxiliary image. This implementation supports estimating the lens power indication with enhanced accuracy.

[0009] In an implementation of the first aspect, the device is further configured to perform the estimation of the lens power indication by applying a neural network. This implementation supports estimating the lens power indication with enhanced efficiency.

[0010] In an implementation of the first aspect, the device is further configured to obtain confirmation or correction of the estimated lens power indication prior to generation of the output image. This implementation supports estimating the lens power indication with enhanced accuracy.

[0011] In an implementation of the first aspect, the device is further configured to remove the detected physical glasses from the detected face before the output image is generated. This implementation supports an enhanced user experience of trying on virtual glasses because the generated virtual glasses are visible instead of the physical glasses.

[0012] In an implementation of the first aspect, the device is further configured to remove the detected physical glasses by applying a glasses removal generative adversarial network. This implementation supports removing the detected physical glasses with enhanced efficiency.

[0013] In an implementation of the first aspect, the device is further configured to obtain input information including a lens power indication from a user input. This implementation supports a user manually inputting a lens power indication.

[0014] In an implementation of the first aspect, the device is further configured to obtain at least one visual virtual glasses parameter. An output image is further generated to modify the virtual glasses worn by the detected face according to the obtained at least one visual virtual glasses parameter. This implementation supports an enhanced virtual glasses try-on user experience because the user can change various parameters that affect the visual appearance of the virtual glasses.

[0015] In an implementation of the first aspect, the at least one visual virtual glasses parameter obtained includes at least one of glasses type, lens color, lens reflectivity, and lens sensitivity. This implementation supports an enhanced virtual glasses try-on user experience because the user can change various parameters that affect the visual appearance of the virtual glasses.

[0016] In an implementation of the first aspect, the device is further configured to obtain at least one visual facial parameter. An output image is further generated to modify the detected face according to the obtained at least one visual facial parameter. This implementation supports an enhanced virtual glasses try-on user experience because the user is able to change various parameters that affect the visual appearance of the user's face.

[0017] In an implementation of the first aspect, the acquired input information further includes at least one of an interpupillary distance and an astigmatism degree. This implementation supports estimating the lens power indication with enhanced accuracy and generating an output image with a more realistic visual appearance of the user's face.

[0018] In an implementation of the first aspect, in addition to the generated output image, the device is further configured to generate and display a comparison image. The comparison image includes the detected face without the virtual glasses. This implementation supports an enhanced virtual glasses try-on user experience because the user can easily compare how they look wearing the virtual glasses and how they look without the virtual glasses.

[0019] According to a second aspect, a method for trying on virtual glasses is provided. The method includes a device for trying on virtual glasses acquiring an input image including a user's face. The method also includes the device performing facial detection on the acquired input image. The method also includes the device acquiring input information related to the virtual glasses. The input information includes a lens power indication of the virtual glasses. The method also includes the device generating an output image, the output image including the user's face detected by the facial detection performed, the detected face wearing virtual glasses corresponding to the lens power indication, thereby modifying at least one area including the eyes of the detected face according to the lens power indication. The method also includes the device displaying the generated output image. The present invention supports trying on virtual glasses with adjustable lens power. The user can adjust the lens power indication included in the input information, and the virtual glasses in the output image are generated based on the input information. In addition, at least the eye area of ​​the detected face can be modified based on the lens power indication, thereby providing the user with intuitive information about what they look like when wearing various prescription glasses.

[0020] In an implementation of the second aspect, obtaining the input information includes: the device detecting, in the acquired input image, physical glasses worn by the detected face, and the device estimating the lens power indication based on at least the eye shape of the detected face. This implementation supports automatically obtaining the lens power indication from the input image.

[0021] In an implementation of the second aspect, the method further includes: the device acquires an auxiliary image, the auxiliary image including a face of the user without wearing physical glasses. The estimation of the lens power indication is also based on the eye shape difference between the input image and the auxiliary image. This implementation supports estimating the lens power indication with enhanced accuracy.

[0022] In an implementation of the second aspect, the device performs the estimation of the lens power indication by applying a neural network. This implementation supports estimating the lens power indication with enhanced efficiency.

[0023] In an implementation of the second aspect, the method further comprises obtaining, by the device, a confirmation or correction of the estimated lens power indication prior to generation of the output image. This implementation supports estimating the lens power indication with enhanced accuracy.

[0024] In an implementation of the second aspect, the method further includes, before the output image is generated, the device removing the detected physical glasses from the detected face. This implementation supports an enhanced user experience of trying on virtual glasses because the generated virtual glasses are visible instead of the physical glasses.

[0025] In an implementation of the second aspect, the device performs removal of the detected physical glasses by applying a glasses removal generative adversarial network. This implementation supports removal of the detected physical glasses with enhanced efficiency.

[0026] In an implementation of the second aspect, the device obtains input information including a lens power indication from a user input. This implementation supports a user to manually input a lens power indication.

[0027] In an implementation of the second aspect, the method further includes the device acquiring at least one visual virtual glasses parameter. An output image is also generated to modify the virtual glasses worn by the detected face according to the acquired at least one visual virtual glasses parameter. This implementation supports an enhanced virtual glasses try-on user experience because the user can change various parameters that affect the visual appearance of the virtual glasses.

[0028] In an implementation of the second aspect, the at least one visual virtual glasses parameter obtained includes at least one of glasses type, lens color, lens reflectivity, and lens sensitivity. This implementation supports an enhanced virtual glasses try-on user experience because the user can change various parameters that affect the visual appearance of the virtual glasses.

[0029] In an implementation of the second aspect, the method further includes obtaining, by the device, at least one visual facial parameter. An output image is also generated, thereby modifying the detected face according to the obtained at least one visual facial parameter. This implementation supports an enhanced virtual glasses try-on user experience because the user is able to change various parameters that affect the visual appearance of the user's face.

[0030] In an implementation of the second aspect, the acquired input information further includes at least one of a distance between pupils or a degree of astigmatism. This implementation supports estimating the lens power indication with enhanced accuracy and generating an output image with a more realistic visual appearance of the user's face.

[0031] In an implementation of the second aspect, the method further includes, in addition to the generated output image, the device further generating and displaying a comparison image. The comparison image includes the detected face without the virtual glasses. This implementation supports an enhanced virtual glasses try-on user experience because the user can easily compare how they look wearing the virtual glasses and when they do not wear the virtual glasses.

[0032] According to a third aspect, a computer program product is provided. The computer program product comprises program code, which, when executed on a computer, is used to perform the method according to the second aspect. The present invention supports trying on virtual glasses with adjustable lens power. The user can adjust the lens power indication included in the input information, and the virtual glasses in the output image are generated based on the input information. In addition, at least the eye area of ​​the detected face can be modified based on the lens power indication, thereby providing the user with intuitive information about what they look like when wearing various prescription glasses.

[0033] Many of the attendant features become better understood by reference to the following detailed description considered in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In the following, exemplary embodiments are described in more detail with reference to the accompanying drawings, in which:

[0035] Figure 1 Includes diagrams showing eyeglasses with various lens powers;

[0036] Figure 2 is a block diagram of a device for virtual eyeglasses trial wearing;

[0037] Figure 3 A diagram showing a general overview of virtual glasses fitting with adjustable lens power;

[0038] Figure 4 is shown in more detail Figure 3 Figure 1: An augmented reality-based glasses fitting frame.

[0039] Figure 5 is a diagram of a first embodiment;

[0040] Figure 6 is a diagram further illustrating the first embodiment;

[0041] Figure 7 is a diagram of a second embodiment;

[0042] Figure 8 is a diagram further illustrating the second embodiment;

[0043] Fig. 9A It is a flow chart of a method;

[0044] Fig. 9B Here is a flowchart of another approach.

[0045] In the following, identical reference signs refer to identical or at least functionally equivalent features. DETAILED DESCRIPTION

[0046] In the following description, reference is made to the accompanying drawings, which form a part of the present invention and show by way of illustration specific aspects to which the present invention may be applied. It will be appreciated that other aspects may be utilized and structural or logical changes may be made without departing from the scope of the present invention. Therefore, the following detailed description should not be understood in a restrictive sense, as the scope of the present invention is defined in the appended claims.

[0047] For example, it should be understood that the disclosure related to the described method may also be applicable to the corresponding device or system for performing the method, and vice versa. For example, if a specific method step is described, the corresponding device may include a unit for performing the described method step, even if such a unit is not explicitly described or shown in the drawings. On the other hand, for example, if a specific device is described based on functional units, the corresponding method may include steps for performing the described functions, even if such steps are not explicitly described or shown in the drawings. In addition, it should be understood that unless otherwise explicitly stated, the features of the various exemplary aspects described herein may be combined with each other.

[0048] Glasses may change the appearance of the user wearing the glasses due to the lens power. In particular, the shape of the eyes and / or the area comprising the eyes may be changed. Figure 1 Figures illustrating this are included. Figure 100A shows a user wearing glasses with farsighted lenses, Figure 100B shows a user wearing glasses with zero-degree lenses, and Figure 100C shows a user wearing glasses with nearsighted lenses. As can be seen, the eye shape is different in each of Figures 100A to 100C.

[0049] In recent years, people have shown increasing interest in various face-related applications. Among these applications, virtual glasses try-on based on augmented reality (AR) is a new way to buy glasses online. This can help customers know how they look when wearing various glasses, thereby improving the interaction between buyers and online sellers.

[0050] At least some embodiments of the present invention can provide a virtual try-on of glasses for a user with prescription lenses. For example, the lens power can be specified by user input (e.g., via a slider) or by estimation performed by a pre-trained neural network (e.g., a pre-trained neural network based on a convolutional neural network (CNN)).

[0051] At least some embodiments of the present invention may support easy integration, for example, into a website or a mobile application.

[0052] At least some embodiments of the present invention may be executed in real time, acting as a virtual mirror that displays the face of a user wearing the virtual glasses.

[0053] At least some embodiments of the present invention may support removing physical glasses from the output image. That is, the user may wear physical glasses (replaced by the selected virtual glasses in the output image) without the user actually taking off the physical glasses.

[0054] At least some embodiments of the present invention may support the generation of photo-realistic eye shapes with or without glasses based on lens power (from negative to positive).

[0055] Figure 3 FIG. 300 shows a general overview of the disclosed virtual eyeglass fitting with adjustable lens power.

[0056] The lens power may be given by a user at block 304 (eg, using a slider provided by a suitable user interface), or estimated at blocks 302 , 303 by an algorithm or the like based on the input image acquired at block 301 .

[0057] If the input image includes a user wearing physical glasses, then at block 305 the physical glasses may be removed to be replaced with virtual glasses.

[0058] At block 303 , the lens power may be estimated, for example, by comparing the shape of the user's eyes with and without the physical glasses.

[0059] At block 306 , the user may provide additional input, such as visual virtual glasses parameters and / or visual facial parameters, as described in greater detail below.

[0060] At block 307, an output image may be generated. Figure 2 The generation of the output image is described in more detail. Figure 4 An implementation example of block 307 is described in detail.

[0061] At block 308, the generated output image may be displayed to a user.

[0062] Next, based on Figure 2 An exemplary embodiment of an apparatus 200 for virtual eyeglass fitting is described. Some features of the described apparatus are optional features that may provide further advantages.

[0063] Figure 2 is a block diagram of a device 200 for virtual glasses fitting according to an embodiment of the present invention. In one embodiment, the device 200 may include any of various types of client devices, such as user equipment (UE), used directly by an end-user entity and capable of communicating in a wireless network. Such devices include, but are not limited to, smartphones, tablet computers, smart watches, laptop computers, internet-of-things (IoT) devices, and the like. In another embodiment, the device 200 may include a server device (e.g., a server device providing one or more cloud services) connected to the client device in a communicative manner (e.g., wirelessly).

[0064] The device 200 includes at least one processor or processing unit 202 and at least one memory 204 coupled to the at least one processor 202, which can be used to implement the functions described in more detail later. The device 200 can also include a digital camera 206. For example, the camera 206 can be used to capture input images and / or auxiliary images described in more detail later. The camera 206 can include a front camera or a rear camera, etc. The device 200 can also include a display 208 for displaying output images and / or prompts to a user, for example, which will be described in more detail later. In one example, the display 208 can include a touch screen.

[0065] The device 200 may also include other elements, such as a transceiver for enabling the device 200 to send and / or receive information to / from other devices and / or input components (such as a physical or virtual keyboard), and Figure 2 Other components not shown in the figure. In one example, the device 200 can use a transceiver to send or receive signaling information and data according to at least one cellular communication protocol. The transceiver can be used to provide at least one wireless radio connection, such as a 3GPP mobile broadband connection (e.g., 5G). The transceiver can include or be used to couple to at least one antenna for sending and / or receiving radio frequency signals.

[0066] At least one processor 202 may include, for example, one or more of various processing devices, such as a coprocessor, a microprocessor, a controller, a digital signal processor (DSP), a processing circuit including or not including a DSP, or various other processing devices including integrated circuits, including application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), microcontroller units (MCUs), hardware accelerators, special-purpose computer chips, and the like.

[0067] The memory 204 may be used to store computer programs, etc. The memory may include one or more volatile memory devices, one or more non-volatile memory devices, and / or a combination of one or more volatile memory devices and non-volatile memory devices. For example, the memory 204 may be implemented as a semiconductor memory (e.g., a mask ROM, a programmable ROM (PROM), an erasable PROM (EPROM), a flash ROM, a random access memory (RAM), etc.).

[0068] The device 200 is used to obtain an input image including a user's face. For example, the input image may include a still image or an image frame of a video stream captured by the camera 206.

[0069] The device 200 is also used to perform face detection on the acquired input image.

[0070] The device 200 is also used to obtain input information related to the virtual glasses. The input information includes a lens power indication of the virtual glasses. At least in some embodiments, the input information obtained may also include a distance between pupils and / or a degree of astigmatism. For example, the lens power indication may include one or more values ​​in the user's eyeglass prescription, such as one or more diopter-related values.

[0071] The device 200 is also used to generate an output image. The output image includes a face of a user detected by performing facial detection, the detected face wearing virtual glasses corresponding to the lens power indication, so as to modify at least one area including an eye of the detected face according to the lens power indication.

[0072] The device 200 is further configured to display the generated output image, for example on a display 208 .

[0073] In the first embodiment, the device 200 may also be used to obtain input information by detecting physical glasses worn by the detected face in the acquired input image and estimating the lens power indication based on at least the eye shape of the detected face. For example, the device 200 may also be used to perform the estimation of the lens power indication by applying a neural network.

[0074] In at least some implementations, the device 200 can also be used to display guidance information in response to failure to detect physical glasses. For example, the guidance information can include prompts to ensure that the device 200 is stable and / or to ensure that the face is in the field of view of the camera 206.

[0075] In the first embodiment, the device 200 may also be used to acquire an auxiliary image, the auxiliary image comprising the face of the user without wearing physical glasses.The estimation of the lens power indication may also be based on the eye shape difference between the input image and the auxiliary image.

[0076] In the first embodiment, the device 200 may also be used to obtain confirmation or correction of the estimated lens power indication prior to generation of the output image.

[0077] In a first embodiment, the device 200 may also be used to remove detected physical glasses from the detected face before the output image is generated. For example, the device 200 may also be used to remove detected physical glasses by applying an eyeglasses removal generative adversarial network (ERGAN).

[0078] In a second embodiment, the device 200 may also be used to obtain input information including a lens power indication from a user input (eg, via a text input box or a slide bar).

[0079] In at least some implementations, the user can be prompted to take off physical glasses (e.g., to support displaying virtual glasses on the user's face without wearing physical glasses), and / or to wear physical glasses (e.g., to support lens power estimation of the first embodiment).

[0080] In the first embodiment and / or the second embodiment, the device 200 may also be used to obtain at least one visual virtual glasses parameter. An output image may also be generated to modify the virtual glasses worn by the detected face according to the obtained at least one visual virtual glasses parameter. For example, the obtained at least one visual virtual glasses parameter may include glasses type, lens color, lens reflectivity, and / or lens sensitivity.

[0081] In the first embodiment and / or the second embodiment, the device 200 may also be used to obtain at least one visual facial parameter. An output image may also be generated to modify the detected face according to the obtained at least one visual facial parameter. For example, the obtained at least one visual facial parameter may include eye color, hair color and / or skin color.

[0082] In the first embodiment and / or the second embodiment, in addition to the generated output image, the device 200 can also be used to simultaneously or continuously generate and display a comparison image, for example. The comparison image can include a detected face without wearing virtual glasses.

[0083] Figure 5 Figure 500 and Figure 6 Diagram 600 also shows a first embodiment.

[0084] As above combined Figure 2 As described, the first embodiment supports virtual glasses try-on with adjustable lens power, for example, when a user wears physical glasses (as shown at 200A) and wishes to know what he / she looks like wearing various virtual glasses (as shown at 200B). At 200A, when the physical glasses are present in an input image / video (e.g., a portrait of the user), the lens power will be estimated and used to generate the user's face wearing the virtual glasses. As discussed in more detail above, various virtual glasses can be selected / switched (block 501), and the lens power estimate provided by the lens power estimation can be edited (block 502).

[0085] As discussed in more detail above, a face detector may be applied to an input image or video of a user.

[0086] When no face is detected in the input image / video, it may be that the virtual glasses fitting cannot be performed. At this time, the device 200 can also be used to prompt the user to enter the lens degree, thereby triggering a switch to the second embodiment. Alternatively / additionally, as discussed above, at least in some implementations, the device 200 can also be used to display guidance information in response to failure to detect physical glasses. For example, the guidance information may include prompts to ensure that the device 200 is stable and / or to ensure that the face is in the field of view of the camera 206.

[0087] When faces are detected in the input image / video, the glasses detector 601 may be activated (eg, for each detected face in the input image / video) to check whether one or more users are already wearing glasses.

[0088] When glasses are detected, the trained CNN architecture may be used to perform lens power estimation 602 .

[0089] The estimated lens power may be confirmed by the user. Additionally, the user may edit the estimated lens power value, for example, to correspond to an eyeglass prescription (block 603).

[0090] Then, glasses removal 604 may be performed by using the trained ERGAN to generate a face of the user without glasses.

[0091] The ERGAN output, one or more lens power values ​​and / or glasses style, etc. (box 605) can be input into the AR glasses power box 606 to generate the face of the user wearing the virtual glasses.

[0092] Figure 7 Figure 700 and Figure 8 Diagram 800 also shows a first embodiment.

[0093] As above combined Figure 2 As described, the second embodiment supports virtual glasses with adjustable lens power, for example, when the user is not wearing physical glasses (as shown at 200C) and wants to know what he / she looks like when wearing various virtual glasses (as shown at 200D). At 200C, when physical glasses are not present in the input image / video (e.g., a portrait of the user), the lens power is obtained from the user input and used to generate the face of the user wearing the virtual glasses. As discussed in more detail above, various virtual glasses can be selected / switched (box 701). The purpose of the second embodiment includes providing a good virtual glasses experience for the user by manually entering the lens power.

[0094] As discussed in more detail above, a face detector may be applied to an input image or video of a user.

[0095] When no face is detected in the input image / video, it may be that the virtual glasses try-on cannot be performed. As discussed above, at least in some implementations, the device 200 can also be used to display guidance information in response to failure to detect physical glasses. For example, the guidance information may include prompts to ensure that the device 200 is stable and / or to ensure that the face is in the field of view of the camera 206.

[0096] When faces are detected in the input image / video, the glasses detector 801 may be activated (eg, for each detected face in the input image / video) to check whether one or more users are already wearing glasses.

[0097] When the glasses are not detected, the user may be prompted to manually enter the lens power (block 802).

[0098] The input image or video, one or more lens power values ​​and / or glasses style, etc. (box 803) can be input into the AR glasses power box 804 to generate a face of the user wearing virtual glasses.

[0099] Figure 4 Shown in more detail Figure 3 AR-based glasses fitting frame 307.

[0100] For example, block 307 may take three inputs: a user's facial image 401 , the user's lens power 402 , and a glasses model 403 .

[0101] In operation 307A, face and facial landmark detection may be performed on an image having a face.

[0102] In operation 307B, scaling of the facial area having a lens power (positive or negative) within the eyeglass frame may be performed for the face.

[0103] In operation 307C, the facial area within the inner contour of the eyeglass frame may be cropped.

[0104] In operation 307D, estimation of eye pose may be performed.

[0105] In operation 307E, a three-dimensional (3D) face transformation module may process facial pose estimation, eye point alignment, and 3D glasses rendering.

[0106] In operation 307F, physically based rendering (PBR) and image-based lighting (IBL) (operation 307G) can be used to generate a realistic effect of the selected eyeglass frame aligned with the user's face. PBR refers to a computer graphics method that attempts to render images in a way that simulates the light flow in the real world. IBL refers to a 3D rendering technique that involves capturing an omnidirectional representation of real-world light information as an image. The image can then be used to simulate the lighting of objects in the scene. Output 404 can include the face of the user wearing the virtual glasses.

[0107] Fig. 9A is a flowchart of a method 900A for virtual glasses fitting according to an embodiment of the present invention.

[0108] In operation 901A, the apparatus 200 for virtual glasses fitting acquires an input image including a user's face. Fig. 9A In the embodiment, in the acquired input image, the user can wear physical glasses. In at least some embodiments, the user can be prompted to wear physical glasses before operation 901A (e.g., through the user interface of device 200).

[0109] In optional operation 902, the device 200 may obtain at least one visual virtual glasses parameter.

[0110] In optional operation 903, device 200 may acquire at least one visual facial parameter.

[0111] In operation 904, the device 200 performs face detection on the acquired input image.

[0112] Next, the device 200 obtains input information related to the virtual glasses. The input information includes a lens power indication of the virtual glasses. More specifically, in optional operation 905A1, the device 200 may obtain an auxiliary image, which includes the face of the user who is not wearing physical glasses. In at least some embodiments, the user may be prompted to take off the physical glasses before operation 905A (e.g., through a user interface of the device 200). In optional operation 905A2, the device 200 may detect physical glasses worn by the detected face in the acquired input image, and estimate the lens power indication based on at least the eye shape of the detected face in optional operation 905A3. In optional operation 905A4, the device 200 may obtain confirmation or correction of the estimated lens power indication.

[0113] In optional operation 905A5, device 200 may remove the detected physical glasses from the detected face.

[0114] In operation 906, the device 200 generates an output image, which includes a face of a user detected by performing facial detection, and the detected face wears virtual glasses corresponding to the lens power indication, thereby modifying at least one area including the eyes of the detected face according to the lens power indication.

[0115] In operation 907, the device 200 displays the generated output image.

[0116] In optional operation 908, in addition to the generated output image, the device 200 also generates a comparison image. The comparison image includes the detected face without the virtual glasses.

[0117] Method 900A may be performed by device 200. For example, operations 901 to 908 may be performed by at least one processor 202 and memory 204. Other features of method 900A are directly derived from the functions and parameters of device 200, and thus are not described in detail herein. Method 900A may be performed by a computer program.

[0118] Fig. 9AOperations 901A to 908 may be performed in any suitable order, or simultaneously where appropriate. For example, operation 903 may be performed after operation 905A1. For another example, operation 905A1 may be performed before operation 901A.

[0119] Fig. 9B is a flowchart of a method 900B for virtual glasses fitting according to an embodiment of the present invention.

[0120] In operation 901B, the apparatus 200 for virtual glasses fitting acquires an input image including a user's face. Fig. 9B In an embodiment, in the acquired input image, the user may not be wearing physical glasses.

[0121] In optional operation 902, the device 200 may obtain at least one visual virtual glasses parameter.

[0122] In optional operation 903, device 200 may acquire at least one visual facial parameter.

[0123] In operation 904, the device 200 performs face detection on the acquired input image.

[0124] Next, the device 200 obtains input information related to the virtual glasses. The input information includes the lens power indication of the virtual glasses. More specifically, in optional operation 905B, the device 200 can obtain input information including the lens power indication from the user input.

[0125] In operation 906, the device 200 generates an output image, which includes a face of a user detected by performing facial detection, and the detected face wears virtual glasses corresponding to the lens power indication, thereby modifying at least one area including the eyes of the detected face according to the lens power indication.

[0126] In operation 907, the device 200 displays the generated output image.

[0127] In optional operation 908, in addition to the generated output image, the device 200 also generates a comparison image. The comparison image includes the detected face without the virtual glasses.

[0128] Method 900B may be performed by device 200. For example, operations 901 to 908 may be performed by at least one processor 202 and memory 204. Other features of method 900B are directly derived from the functions and parameters of device 200, and thus are not described in detail herein. Method 900B may be performed by a computer program.

[0129] Fig. 9BOperations 901B to 908 may be performed in any suitable order, or simultaneously where appropriate. For example, operation 903 may be performed after operation 905B. For another example, operation 905B may be performed before operation 901B.

[0130] The device 200 may include a component for performing at least one method described herein. In one example, the component may include at least one processor 202 and at least one memory 204, and at least one memory 204 includes program code, which, when executed by at least one processor, is used to cause the device 200 to perform the method.

[0131] The functions described herein may be performed at least in part by one or more computer program product components (e.g., software components). According to one embodiment, the device 200 may include a processor or a processor circuit, such as a microcontroller, and when the program code is executed, the processor or processor circuit is configured by the program code to perform the described operations and embodiments of the functions. Alternatively or in addition, the functions described herein may be performed at least in part by one or more hardware logic components. For example, without limitation to this, exemplary types of hardware logic components that may be used include field programmable gate arrays (FPGAs), program-specific integrated circuits (ASICs), program-specific standard products (ASSPs), system-on-a-chip (SOCs), complex programmable logic devices (CPLDs), and graphics processing units (GPUs).

[0132] Any range or device value given herein may be expanded or altered without losing the effect sought. In addition, any embodiment may be combined with other embodiments unless expressly prohibited.

[0133] Although the subject matter has been described in specific language of structural features and / or actions, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are disclosed as examples of implementing the claims, and other equivalent features and actions are intended to be included within the scope of the claims.

[0134] It will be understood that the benefits and advantages described above may relate to one embodiment or to several embodiments. The embodiments are not limited to embodiments that solve any or all of the problems described, nor are they limited to embodiments that have any or all of the advantages and benefits described. It will also be understood that reference to "an" item may refer to reference to one or more of these items.

[0135] The steps of the methods described herein may be performed in any appropriate order, or simultaneously where appropriate. Additionally, individual blocks may be deleted from any of the methods without departing from the spirit and scope of the subject matter described herein. Aspects of any of the embodiments described above may be combined with aspects of any other embodiment described to form other embodiments without losing the effects sought.

[0136] The term "comprising" is used herein to mean including the identified methods, blocks or elements, but such blocks or elements do not include an exclusive list and the method or apparatus may include additional blocks or elements.

[0137] It should be understood that the above description is provided by way of example only, and that various modifications may be made by those skilled in the art. The above description, examples, and data provide a complete description of the structure and application of the exemplary embodiments. Although various embodiments have been described above with a certain degree of particularity or with reference to one or more separate embodiments, those skilled in the art may make many modifications to the disclosed embodiments without departing from the spirit or scope of this specification.

Claims

1. A device (200) for virtual glasses fitting, characterized in that: Used for: obtaining an input image including a face of a user; Performing face detection on the acquired input image; Acquire input information related to the virtual glasses, wherein the input information includes an indication of the lens power of the virtual glasses; generating an output image, the output image including the face of the user detected by the performed face detection, the detected face wearing the virtual glasses corresponding to the lens power indication, thereby modifying at least one area including the eyes of the detected face according to the lens power indication; The generated output image is displayed.

2. The device (200) according to claim 1, characterized in that The input information is also acquired by detecting physical glasses worn by the detected face in the acquired input image and estimating the lens power indication based at least on the eye shape of the detected face.

3. The device (200) according to claim 2, characterized in that Also used to obtain an auxiliary image, the auxiliary image including the face of the user without wearing the physical glasses, wherein the estimation of the lens power indication is also based on the eye shape difference between the input image and the auxiliary image.

4. The device (200) according to claim 2 or 3, characterized in that Also for performing said estimating of said lens power indication by applying a neural network.

5. The device (200) according to any one of claims 2 to 4, characterized in that Also for obtaining a confirmation or correction of said estimated lens power indication prior to said generating of said output image.

6. The device (200) according to any one of claims 2 to 5, characterized in that Also for removing said detected physical glasses from said detected face prior to said generating of said output image.

7. The device (200) according to claim 6, characterized in that It is also used to remove the detected physical glasses by applying a glasses removal generative adversarial network.

8. The device (200) according to claim 1, characterized in that It is also used to obtain the input information including the lens power indication from the user input.

9. The device (200) according to any one of claims 1 to 8, characterized in that It is also used to obtain at least one visual virtual glasses parameter, wherein the output image is also generated to modify the virtual glasses worn on the detected face according to the obtained at least one visual virtual glasses parameter.

10. The device (200) according to claim 9, characterized in that The at least one obtained visual virtual glasses parameter includes at least one of glasses type, lens color, reflectivity of the lens, and sensitivity of the lens.

11. The device (200) according to any one of claims 1 to 10, characterized in that The method is also used to obtain at least one visual facial parameter, wherein the output image is also generated so as to modify the detected face according to the obtained at least one visual facial parameter.

12. The device (200) according to any one of claims 1 to 11, characterized in that The acquired input information further includes at least one of the distance between pupils and the degree of astigmatism.

13. The device (200) according to any one of claims 1 to 12, characterized in that The method is further configured to generate and display, in addition to the generated output image, a comparison image, wherein the comparison image includes the detected face without the virtual glasses.

14. A method (900A, 900B) for virtual glasses fitting, characterized in that: include: The device for virtual glasses fitting acquires (901A, 901B) an input image including a user's face; The device performs (904) face detection on the acquired input image; The device obtains (905A1 to 905A4, 905B) input information related to the virtual glasses, wherein the input information includes an indication of the lens power of the virtual glasses; The device generates (906) an output image, the output image including the face of the user detected by the performed face detection, the detected face wearing the virtual glasses corresponding to the lens power indication, thereby modifying at least one area including an eye of the detected face according to the lens power indication; The device displays (907) the generated output image.

15. A computer program product comprising program code, characterized in that When the program code is executed on a computer, the program code is used to perform the method (900A, 900B) according to claim 14.

Images