Eyewear distortion correction

Abstract

Systems, methods, and non-transitory computer-readable media are disclosed that include a process for correcting distortions in images introduced by eyewear (i.e., where the region of the face around the eyes has boundaries that do not match the boundaries of the uncovered region of the face). The correction includes segmenting the image to detect a region of the face covered by the eyewear and a region of the face not covered by the eyewear, and altering the covered region of the face to match the covered face boundaries to the uncovered face boundaries. The altering includes processing using a machine learning model, applying an anti-aliasing algorithm, scaling the covered region of the face to match the boundaries of the uncovered region of the face, or a combination thereof.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Application No. 17 / 039,334, filed on September 30, 2020, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] This disclosure relates to image processing, and more specifically to processing images to correct distortions introduced by eyewear. Background Technology

[0004] Eye-worn devices include lenses that act as prisms to refract light in order to improve the wearer's vision. Refraction introduces distortion, which causes the wearer's eyes (and the surrounding area covered by the lens) to appear disproportionate to the rest of the head when viewed by another person, making the wearer's eyes appear out of proportion to the rest of the head not covered by the lens. This distortion is also noticeable in the wearer's images. Attached Figure Description

[0005] The accompanying drawings depict one or more specific embodiments by way of example only and not by way of limitation. In these drawings, similar reference numerals refer to the same or similar elements, and letter markings are added to distinguish them. Letter markings may be omitted when referring collectively to the same or similar elements, or when referring to non-specific elements among the same or similar elements.

[0006] Figure 1A This is a side view of an exemplary hardware configuration of an eye-wearing device, which includes a visible light camera at the corner and a speaker on the temple.

[0007] Figure 1B and Figure 1C yes Figure 1A A rear view of an exemplary hardware configuration for an eye-wearing device, which includes two different types of image displays.

[0008] Figure 2 yes Figure 1A The top cross-sectional view of the corner of the eye-wearing device depicts a visible light camera, a head movement tracker, and a circuit board.

[0009] Figure 3A It is a high-level functional block diagram of an exemplary image capture, processing, and display system that includes eye-wearing devices, mobile devices, and server systems connected via various networks.

[0010] Figure 3B yes Figure 3A A simplified block diagram illustrating the hardware configuration of the server system for an audio visualization system.

[0011] Figure 4It is a simplified block diagram of the hardware configuration of a mobile device.

[0012] Figure 5A , Figure 5B , Figure 5C and Figure 5D This is a flowchart of exemplary steps for eliminating distortions introduced by eyewear.

[0013] Figures 6A and 6B are prior art illustrations depicting image magnification due to lenses.

[0014] Figure 6C This is an illustration depicting a technique used to eliminate distortions introduced by eyewear.

[0015] Figure 7A is an illustration of an image based on the prior art, showing the distortion introduced by the eyewear.

[0016] Figure 7B This is an illustration depicting exemplary steps to eliminate distortions introduced by eyewear.

[0017] Figure 7C and Figure 7D It is an illustration of two resulting images with the altered eyewear region corrected for distortion introduced by the eyewear. Detailed Implementation

[0018] The following detailed description includes examples of methods for correcting distortions introduced by an eye-wearing device in an image (still image or video image), i.e., where the facial region around the eye covered by the eye-wearing lens has a covered facial boundary that does not match the uncovered facial boundary of the uncovered facial region. This correction includes: segmenting the image to detect the facial regions covered and uncovered by the eye-wearing device, and modifying the covered facial region to match the covered facial boundary with the uncovered facial boundary. Modifications include processing using a machine learning model, applying an anti-reflection algorithm, scaling the covered facial region to match the boundary of the uncovered facial region, or combinations thereof.

[0019] In the following detailed embodiments, numerous specific details are illustrated by way of example to provide a thorough understanding of the relevant teachings. However, it will be apparent to those skilled in the art that the teachings can be practiced without such details. In other instances, well-known methods, processes, components, and circuits are described at a higher level without detail to avoid unnecessarily obscuring various aspects of the teachings.

[0020] As used herein, the term "coupled" refers to any logical, optical, physical, or electrical connection, link, etc., through which electrical signals generated or provided by one system element are transmitted to another coupled element. Unless otherwise described, coupled elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements, or communication media that can modify, manipulate, or carry signals. The term "on" means directly supported by an element or indirectly supported by an element through another element integrated into or supported by that element. As used herein, the term "about" means within ±10% of the stated amount.

[0021] For purposes of illustration and discussion, the orientation of mobile devices, eye-wearing devices, associated components, and any complete device incorporating a camera, such as any of the figures shown in the accompanying drawings, is given by way of example only. In operation, for a particular programming, the device may be oriented in any other direction suitable for a particular application, such as up, down, sideways, or any other orientation. Furthermore, for the purposes of this document, any directional terms such as front, back, inside, outside, towards, left, right, sideways, longitudinal, up, down, high, low, top, bottom, and side are used by way of example only and do not limit the orientation or orientation of any camera or camera component constructed as otherwise described herein.

[0022] The objectives, advantages, and novel features of the examples will be set forth in part in the following detailed description, and in part will become apparent to those skilled in the art upon examination of the following description and the accompanying drawings, or may be learned by production or operation of the examples. The objectives and advantages of this subject matter may be realized and achieved by means of the methods, means, and combinations particularly pointed out in the appended claims.

[0023] Now refer in detail to the accompanying drawings and the examples discussed below.

[0024] Figure 1A An exemplary hardware configuration of a mobile device in the form of an eye-wearing device 100 for collecting and optionally processing images is depicted. The mobile device may take other forms, such as a mobile phone or a tablet. Additionally, the eye-wearing device 100 may take other forms and may be combined with other types of frames, such as a headband, headset, or helmet. The eye-wearing device 100 includes at least one visible light camera 114 on a corner 110B for capturing images in a viewing area (e.g., field of view). The illustrated eye-wearing device 100 also includes a speaker 115 and a microphone 116.

[0025] Visible light camera 114 is sensitive to wavelengths within the visible light range. As shown in this example, visible light camera 114 has a forward-facing field of view from the wearer's perspective, configured to capture an image of the scene viewed through optical assembly 180B. Examples of such visible light cameras 114 include high-resolution complementary metal-oxide-semiconductor (CMOS) image sensors and video graphics array (VGA) cameras, such as 640p (e.g., 640 × 480 pixels, totaling 0.3 megapixels), 720p, or 1080p (or greater). Eye-wearing device 100 captures image sensor data from visible light camera 114 and optionally captures other data such as geolocation data and audio data (via microphone 116), digitizes the data using one or more processors, and stores the digitized data in memory. The term "field of view" describes the viewing area that a user of a mobile device can see through optical assembly 180B or on a display of the mobile device that presents information captured by visible light camera 114.

[0026] Visible light camera 114 can be coupled to an image processor for digital processing and adding timestamps and location coordinates corresponding to the time and location of the captured scene. Figure 3A Component 312 in the image processor 312 includes receiving signals from the visible light camera 114 and processing those signals from the visible light camera 114 into a form suitable for storage in memory. Figure 3A The circuitry is in the format of element 334. The timestamp can be added by the image processor 312 or another processor that controls the operation of the visible light camera 114. The image processor 312 can also add, for example, data from the Global Positioning System (GPS). Figure 3A The position coordinates received by component 331 in the middle.

[0027] Microphone 116 may be coupled to an audio processor (not shown) for digital processing and to add a timestamp indicating when the audio was captured. The audio processor includes circuitry that receives signals from microphone 116 (or from memory) and processes these signals into a format suitable for storage in memory 334 or presentation by speaker 115. The timestamp may be added by the audio processor or by another processor that controls the operation of speaker 115 and microphone 116.

[0028] like Figure 1A , Figure 1B and Figure 1CAs shown, the eye-wearing device 100 includes a frame 105 having a left edge 107A connected to a right edge 107B via a nose bridge 106 adapted to fit a user's nose. The left and right edges 107A-B include corresponding apertures 175A-B for holding corresponding optical components 180A-B. Left and right temples 125A-B extend from corresponding sides 170A-B of the frame 105, for example, via corresponding left and right corner portions 110A-B. Each temple 125A-B is connected to the frame 105 via a corresponding hinge 126A-B. The substrate or material forming the frame 105, corner portions 110, and temples 125A-B may include plastic, acetate, metal, or combinations thereof. Corner portions 110A-B may be integrated into or connected to the frame 105 or temples 125A-B.

[0029] Although shown as having two optical components 180A-B, the eye-wearing device 100 may include other arrangements, such as a single component or three optical components, or the optical components 180A-B may have different arrangements, depending on the application of the eye-wearing device 100 or the intended user.

[0030] In one example, such as Figure 1B As depicted herein, each optical component 180A-B includes a display matrix 171 and one or more optical layers 176A-N. The display matrix 171 may include a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, or other such displays. The one or more optical layers 176 may include lenses, optical coatings, prisms, mirrors, waveguides, optical strips, and other optical components in any combination. As used herein, the term "lens" is intended to encompass a transparent or translucent sheet of glass or plastic having a curved or flat surface that causes light to converge / diverge or to cause little or no convergence or divergence.

[0031] Optical layers 176A-N may include prisms having suitable dimensions and construction and including a first surface for receiving light from a display matrix and a second surface for emitting light toward a user's eye. The prisms of optical layers 176A-N extend over all or at least a portion of corresponding apertures 175A-B formed in the left and right edges 107A-B, allowing the user to see the second surface of the prism when viewed through the corresponding left and right edges 107A-B. The first surface of the prisms of optical layers 176A-N faces upward from the frame 105, and the display matrix covers the prisms such that photons and light emitted by the display matrix illuminate the first surface. The prisms are sized and shaped such that light is refracted within the prisms and directed to the user's eye by the second surface of the prisms of optical layers 176A-N. In this respect, the second surface of the prisms of optical layers 176A-N may be convex to direct light toward the center of the eye. The size and shape of the prism can be optionally designed to magnify the image projected by the display matrix 171, and the light travels through the prism such that the image viewed from the second surface is larger than the image emitted from the display matrix 171 in one or more dimensions.

[0032] In another example, such as Figure 1C The image display device depicted in the diagram, comprising optical components 180A-B, includes a projection image display. The projection image display includes a laser projector 150 (e.g., a tri-color laser projector using a scanning mirror or galvanometer) positioned at one of the corners 110A-B adjacent to the eyewear device 100, and optical strips 155A-N spaced apart by the width of the lens across optical components 180A-B or by the depth of the lens between the front and rear surfaces of the lens.

[0033] When photons projected by laser projector 150 travel through the lenses of optical components 180A and 180B, they encounter optical strips 155A-N. When a specific photon encounters a specific optical strip, the photon is either redirected toward the user's eye or propagated to the next optical strip. A combination of modulation of laser projector 150 and modulation of the optical strips controls specific photons or beams. In this example, the processor controls optical strips 155A-N by emitting mechanical, acoustic, or electromagnetic signals.

[0034] In one example, the visual output generated on the optical components 180A-B of the eye-wear device 100 includes a superimposed image overlaid on at least a portion of the field of view through the optical components 180A-B. In one example, the optical components 180A-B are a see-through display that presents the superimposed image as a superimposed image on a scene (or features within a scene) viewed by the wearer through the lenses of the optical components. In another example, the optical components 180A-B are not a see-through display (e.g., an opaque display) and present the superimposed image by combining the superimposed image with real-time images captured by the camera 114 of the eye-wear device for presentation to the user on a display.

[0035] As described above, the eye-wearing device 100 is coupled to a processor and memory, for example, within the eye-wearing device 100 itself or in another part of the system. The eye-wearing device 100 or the system can then process the captured eye images; for example, the coupled memory and processor in the system process the captured eye images to track eye movement. This processing of the captured images establishes a scanning path to identify the user's eye movement. The scanning path comprises a series or sequence of eye movements based on changes in the captured eye's reflections. Eye movements are generally categorized into such fixations and saccades—specifically, when the eye fixates on a location and moves to another location. The resulting series of fixations and saccades is called the scanning path. Smooth following describes the eye following a moving object. Fixational eye movements include microsaccades: small, unconscious saccades that occur during an attempt to fixate. The scanning path is then used to determine field of view adjustment.

[0036] An eye orientation database can be established during calibration. Since the known fixed locations of the corresponding points of interest are known during calibration, a scan path database can be used to establish similarity with previously acquired calibration images. Because the known fixed locations of the points of interest are known from the calibration images and recorded in the eye orientation database, the eye-wearing device 100 can determine where the user's eyes are looking by comparing the currently captured image of the eye with the eye orientation database. The calibration image most similar to the currently captured image can be used as a sufficiently approximate direction for the eye orientation of the currently captured image, based on the known fixed locations of the points of interest.

[0037] Figure 2 yes Figure 1A The top cross-sectional view of the corner of the eye-wearing device 100 depicts the right visible light camera 114, head movement tracker 109, and microphone 116. Except for the connection and coupling located on the left side 170A, the structure and arrangement of the left visible light camera are substantially similar to those of the right visible light camera 114.

[0038] The right corner portion 110B includes a corner body and a corner cover. Figure 2The corner cap is omitted in the cross-section. As shown, the eye-wearing device 100 includes a circuit board, which may be a flexible printed circuit board (PCB) 240, having controller circuitry for the right visible light camera 114, a microphone, and low-power wireless circuitry (e.g., for use via Bluetooth). TM The device uses short-range wireless network communication and high-speed wireless circuitry (e.g., for wireless local area network communication via WiFi). The right hinge 126B connects the right corner 110B to the right temple 125C of the eye-wearing device 100. In some examples, components such as the right visible light camera 114, flexible PCB 140, or other electrical connectors or contacts may be located on the right temple 125C or the right hinge 126B.

[0039] The head movement tracker 109 includes, for example, an inertial measurement unit (IMU). An IMU is an electronic device that uses a combination of accelerometers and gyroscopes to measure and report the body's specific force, angular rate, and sometimes a magnetometer to measure and report the magnetic field around the body. An IMU operates by detecting linear acceleration using one or more accelerometers and rotational rate using one or more gyroscopes. A typical IMU configuration includes one accelerometer, one gyroscope, and one magnetometer for each of the following three axes: a horizontal axis (X) for left-right movement, a vertical axis (Y) for top-bottom movement, and a depth or distance axis (Z) for up-down movement. The gyroscope detects the gravity vector. The magnetometer defines rotation in a magnetic field (e.g., facing south, north, etc.), much like a compass generating a heading reference. The three accelerometers are used to detect acceleration along the horizontal, vertical, and depth axes defined above, which can be defined relative to the ground, the eye-wearing device 100, or the user wearing the eye-wearing device 100.

[0040] The eye-wearing device 100 detects user movement by tracking head movement of the user's head via a head motion tracker 109. Head movement includes changes in head orientation relative to an initial head orientation on a horizontal axis, vertical axis, or a combination thereof during the presentation of an initial displayed image on an image display. In one example, tracking user head movement via the head motion tracker 109 includes measuring the initial head orientation on a horizontal axis (e.g., the X-axis), a vertical axis (e.g., the Y-axis), or a combination thereof (e.g., lateral or diagonal movement) via an inertial measurement unit. Tracking user head movement via the head motion tracker 109 also includes measuring subsequent head orientations on the horizontal axis, vertical axis, or a combination thereof via an inertial measurement unit during the presentation of the initial displayed image.

[0041] Figure 3AThis is a high-level functional block diagram of an exemplary image capture, processing, and display system 300. The image capture, processing, and display system 300 includes a mobile device, which in this example is an eye-wearing device 100. The mobile device can communicate with other mobile devices 390 or server systems 398 via one or more wireless networks or wireless links. The image capture, processing, and display system 300 also includes other mobile devices 390 and server systems 398. The mobile device 390 can be a smartphone, tablet, laptop, access point, or other such device capable of connecting to the eye-wearing device 100 using, for example, low-power wireless connection 325 and high-speed wireless connection 337. The mobile device 390 is connected to the server system 398 via a network 395. The network 395 can include any combination of wired and wireless connections.

[0042] The eye-wearing device 100 includes and supports a visible light camera 114, a speaker 115, a microphone 116, a user interface 301, an image display 180 with optical components, an image display driver 342, an image processor 312, an audio processor 313, low-power circuitry 320, and high-speed circuitry 330. Figure 3A The components shown for the eye-wearing device 100 are located on one or more circuit boards (e.g., PCBs or flexible PCBs) in the temples. Alternatively or additionally, the depicted components may be located in the corners, frames, hinges, or nose bridge of the eye-wearing device 100. Memory 334 includes a feature analyzer 344, a feature model 345, and a face alignment program 346 to perform the functions described herein for image capture, processing, and display. Memory 334 also includes a rendering engine 348 for rendering overlaid images on displays 180A and 180B using image processor 312 and image display driver 342.

[0043] Feature analyzer 344 executes instructions to cause eye-wearing device 100 to process objects and aspects identified from a scene viewed through eye-wearing device 100. Feature model 345 is a machine learning model trained to identify objects (such as faces, glasses, covered facial areas, uncovered facial areas, etc.) and aspects (such as movement, lines, curves, materials). Face alignment programming 346 executes instructions to cause eye-wearing device 100 to identify and correct boundary mismatches between facial areas covered by eye-wearing lenses and facial areas not covered by eye-wearing lenses.

[0044] like Figure 3AAs shown, the high-speed circuit 330 includes a high-speed processor 343, a memory 334, and a high-speed wireless circuit 336. In the example, the image display driver 342 is operated by the high-speed processor 343 to drive the image display of the optical components 180. The high-speed processor 343 can be any processor capable of managing the high-speed communication and operation of any general-purpose computing system required by the eye-wear device 100. The high-speed processor 343 includes the processing resources required to manage high-speed data transmission from the high-speed wireless connection 337 to the wireless local area network (WLAN) using the high-speed wireless circuit 336. In some examples, the high-speed processor 343 executes an operating system, such as the LINUX operating system or other such operating system of the eye-wear device 100, and the operating system is stored in the memory 334 for execution. Among other duties, the high-speed processor 343 executes the software architecture of the eye-wear device 100 to manage data transmission utilizing the high-speed wireless circuit 336. In some examples, the high-speed wireless circuit 336 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as Wi-Fi. In other examples, the high-speed wireless circuit 336 implements other high-speed communication standards.

[0045] The low-power wireless circuit 324 and high-speed wireless circuit 336 of the eye-wearing device 100 may include a short-range transceiver (Bluetooth). TM The device 390 includes a transceiver for wireless wide area networks, local area networks, or wide area networks (e.g., cellular or WiFi). The mobile device 390, including transceivers communicating via low-power wireless connection 325 and high-speed wireless connection 337, can be implemented using the architectural details of the eye-wearing device 100, just like other components of the network 395.

[0046] Memory 334 includes a storage device capable of storing various data and applications, including, among other things, camera data generated by visible light camera 114 and image processor 312, images generated for display on an image display of optical assembly 180 by image display driver 342, and audio data generated by microphone 116 and audio processor 313. While memory 334 is shown as integrated with high-speed circuitry 330, in other examples, memory 334 may be a separate, independent component of eye-wearing device 100. In some examples, electrical wiring may provide a connection from image processor 312 / audio processor 313 or low-power processor 324 to memory 334 via a chip including high-speed processor 343. In other examples, high-speed processor 343 may manage addressing of memory 334 such that low-power processor 324 will activate high-speed processor 343 whenever a read or write operation involving memory 334 is required.

[0047] The eye-worn device 100 also includes a Global Positioning System (GPS) 331, a compass 332, and an inertial measurement unit (IMU) 333. The GPS 331 is a receiver for a satellite-based radio navigation system that receives geographic location and time information from GPS satellites. The compass 332 provides orientation relative to a geographic fundamental direction (or point). The IMU 333 is an electronic device that uses a combination of accelerometers, gyroscopes, and magnetometers to measure and report force, angular rate, orientation, or combinations thereof.

[0048] The eye-wearing device 100 can be connected to a host computer. For example, the eye-wearing device 100 can be paired with a mobile device 390 via a high-speed wireless connection 337, or connected to a server system 398 via a network 395. In one example, the eye-wearing device 100 captures an image of a scene via a camera 114 and sends the image to the host computer for forwarding to the server system 398 for training a feature model 364. In another example, the eye-wearing device 100 receives images and instructions from the host computer.

[0049] The eye-wear device 100 also includes other output and input components. Other output components include acoustic components (e.g., speaker 115), haptic components (e.g., vibration motor), and other signal generators. Input components for the eye-wear device 100, mobile device 390, and server system 398 may include alphanumeric input components (e.g., keyboard, touchscreen configured to receive alphanumeric input, photographic optical keyboard, or other alphanumeric input components), point-based input components (e.g., mouse, touchpad, trackball, joystick, motion sensor, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens or other haptic input components that provide touch location and touch force or touch gestures), audio input components (e.g., microphone 116), etc.

[0050] The image capture, processing, and display system 300 may optionally include additional peripheral device elements 319. Such peripheral device elements 319 may include biometric sensors, additional sensors, or display elements integrated with the eye-wearing device 100. For example, peripheral device elements 319 may include any I / O components, including output components, motion components, positioning components, or any other such components described herein.

[0051] For example, the biometric components of the image capture, processing, and display system 300 include components that detect facial expressions (e.g., gestures, facial expressions, voice expressions, body postures, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), and identify people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Motion components include accelerometer components (e.g., accelerometers), gravity sensor components, rotation sensor components (e.g., gyroscopes), etc. Positioning components include position sensor components (e.g., Global Positioning System (GPS) receiver components) that generate position coordinates, and WiFi or Bluetooth that generates positioning system coordinates. TM Transceivers, altitude sensor components (e.g., altimeters or barometers that detect air pressure and derive altitude from air pressure), orientation sensor components (e.g., magnetometers), etc. Coordinates of such positioning systems can also be received from mobile devices 390 via wireless connections 325 and 337 via low-power wireless circuit 324 or high-speed wireless circuit 336.

[0052] In one example, the image processor 312 includes a microprocessor integrated circuit (IC) customized for processing image sensor data from the visible light camera 114, and volatile memory used by the microprocessor for operation. To reduce the amount of time the image processor 312 spends when powered on to process data, a non-volatile read-only memory (ROM) may be integrated on the IC along with instructions for running or starting the image processor 312. This ROM may be minimized to match the minimum size required to provide the basic functionality for collecting sensor data from the visible light camera 114, so that no additional functionality would cause a startup time delay. The ROM may be configured with direct memory access (DMA) to the volatile memory of the microprocessor of the image processor 312. DMA allows data transfer from the ROM to the system memory of the image processor 312 independently of the operation of the image processor 312's main controller. Providing DMA to this startup ROM further reduces the amount of time from the power-on of the image processor 312 until the sensor data from the visible light camera 114 can be processed and stored. In some examples, minimal processing of the camera signal from the visible light camera 114 is performed by the image processor 312, and additional processing may be performed by an application running on the mobile device 390 or the server system 398.

[0053] Low-power circuitry 320 includes a low-power processor 322 and a low-power wireless circuitry 324. These components of low-power circuitry 320 may be implemented as separate components or as part of a single IC as part of a single-chip system. Low-power processor 324 includes logic for managing other components of the eye-wearing device 100. Low-power processor 324 is configured to receive input signals or command communications from mobile device 390 via low-power wireless connection 325. Additional details relating to such commands will be further described below. Low-power wireless circuitry 324 includes circuitry elements for implementing a low-power wireless communication system via a short-range network. Bluetooth TM Smart, also known as Bluetooth TM Low power consumption is a standard implementation of low-power wireless communication systems that can be used to implement low-power wireless circuitry 324. Other low-power communication systems may be used in other examples.

[0054] The components of mobile device 390 and network 395, low-power wireless connectivity 325, and high-speed wireless architecture 337 can be implemented using detailed components of the mobile device 390's architecture, for example, by utilizing... Figure 4 The short-range XCVR and WWAN XCVR of the mobile device 390 described herein are implemented.

[0055] like Figure 3B As shown, server system 398 may be one or more computing devices as part of a service or network computing system, such as a computing device including processor 360, memory 362, and network communication interface 361 for communicating with mobile device 390 and eye-wearing device 100 via network 395. Memory 362 includes feature model 364 and neural network programming 365. Execution of neural network programming 365 by processor 360 configures server system 398 to perform some of the functions described herein.

[0056] In one example, server system 398 receives images of a scene via network 395 from eye-wearing device 100, mobile device 390, or other devices via neural network programming 365 for training feature model 364. Server system 398 sends the trained feature model to eye-wearing device 100 or mobile device 390 for recognizing facial features, eye-wearing devices, covered facial regions, and uncovered facial regions. Suitable neural networks are convolutional neural networks (CNNs) based on one of the following architectures: VGG16, VGG19, ResNet50, Inception V3, and Xception, or other CNN architectures.

[0057] In one example, machine learning techniques (e.g., deep learning) are used to locate objects in an image. Deep learning is a subset of machine learning that uses a set of algorithms and depth maps with multiple processing layers, including linear and nonlinear transformations, to model high-level abstractions in data. While many machine learning systems are embedded with initial features and network weights that will be modified through the learning and updating of the machine learning network, deep learning networks train themselves to identify “good” features for analysis. Using a multi-layered architecture, machines employing deep learning techniques can process raw data better than those using conventional machine learning techniques. Different evaluation or abstraction layers can be used to facilitate the examination of multiple sets of highly correlated values ​​or unique topics in data.

[0058] CNNs are biologically inspired networks of interconnected data used in deep learning for the detection, segmentation, and recognition of related objects and regions in a dataset. CNNs evaluate the raw data as multiple arrays, break down the data in a series of stages, and examine the learned features of the data.

[0059] In one example, a CNN is used to perform image analysis. The CNN receives an input image and abstracts it in convolutional layers to identify learned features. In a second convolutional layer, the image is transformed into multiple images, where the learned features are each emphasized in their respective sub-images. These images are further processed to focus on the features of interest within the image. The resulting images are then processed by pooling layers, which reduce the image size to separate the image portions containing the features of interest. The output of the convolutional neural network receives values ​​from the final non-output layer and classifies the image based on the data received from this final non-output layer.

[0060] The feature model 345 of the eye-wearing device 100 may be a mirror image of the feature model 364 of the server system 398. The feature model 345 of the eye-wearing device 100 is locally stored in the read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory of the high-speed circuit 330.

[0061] Figure 4 It provides for Figure 3AA high-level functional block diagram of an example mobile device 390 for image capture, processing, and display system 300 is shown. The diagram illustrates the components of a touchscreen type mobile device 390, which includes a feature analyzer 344, a feature model 345, a face alignment programmable 346, and a rendering engine 348, loaded along with other applications such as chat applications. Examples of usable touchscreen mobile devices include (but are not limited to) smartphones, personal digital assistants (PDAs), tablet computers, laptops, or other portable devices. However, the structure and operation of touchscreen devices are provided by way of example; and the subject matter described herein is not intended to be limited thereto. For the purposes of this discussion, Figure 4 A block diagram illustration of an exemplary mobile device 390 is provided, which has a touchscreen display for displaying content and receiving user input as a user interface (or as part of a user interface). The mobile device 390 also includes a camera 470, such as a visible light camera, and a microphone 471.

[0062] like Figure 4 As shown, the mobile device 390 includes at least one digital transceiver (XCVR) 410 for digital wireless communication via a wide-area wireless mobile communication network, shown as a WWAN XCVR. The mobile device 390 also includes additional digital or analog transceivers, such as those for communication via NFC, VLC, DECT, ZigBee, Bluetooth, etc. TM The short-range XCVR 420 can be used for short-range network communication via WiFi. For example, the short-range XCVR 420 can take the form of any available bidirectional wireless local area network (WLAN) transceiver compatible with one or more standard communication protocols implemented in a wireless local area network, such as the Wi-Fi standard compliant with IEEE 802.11 and WiMAX.

[0063] To generate location coordinates for locating mobile device 390, mobile device 390 may include a Global Positioning System (GPS) receiver 331. Alternatively or additionally, mobile device 390 may utilize either or both of a short-range XCVR 420 and a WWAN XCVR 410 to generate location coordinates for positioning. For example, based on cellular networks, WiFi, or Bluetooth. TM The positioning system can generate very accurate position coordinates, especially when used in combination. These position coordinates can be transmitted to the eye-wearing device 100 via one or more network connections through the XCVR 420. Additionally, the mobile device 390 may include a compass 332 and an inertial measurement unit 333 for determining orientation information.

[0064] Transceivers 410 and 420 (network communication interfaces) conform to one or more of the various digital wireless communication standards utilized by modern mobile networks. Examples of WWAN transceivers 410 include (but are not limited to) transceivers configured to operate according to Code Division Multiple Access (CDMA) and 3rd Generation Partnership Project (3GPP) network technologies, including, for example, but not limited to, 3GPP Type 2 (or 3GPP2) and LTE, sometimes referred to as "4G". For example, transceivers 410 and 420 provide bidirectional wireless communication of information including digitized audio signals, still images and video signals, web page information for display and web-related input, and various types of mobile messaging communications to / from mobile device 390 based on user authorization policies.

[0065] Mobile device 390 also includes a microprocessor, shown as CPU 430. A processor is a circuit having elements constructed and arranged to perform one or more processing functions, typically various data processing functions. Although discrete logic components can be used, these examples utilize components that form a programmable CPU. A microprocessor includes, for example, one or more integrated circuit (IC) chips that incorporate electronic elements that perform the functions of the CPU. For example, processor 430 may be based on any known or available microprocessor architecture, such as Reduced Instruction Set Computing (RISC) using the ARM architecture, as is commonly used today in mobile devices and other portable electronic devices. Other processor circuitry may be used to form CPU 430 or processor hardware in smartphones, laptops, and tablets.

[0066] By configuring the mobile device 390 to perform various operations, for example, according to instructions or programs executable by the processor 430, the microprocessor 430 acts as a programmable host controller for the mobile device 390. Such operations may include, for example, various general operations of the mobile device, as well as operations related to determining the device's position when capturing an image and determining the device's position and orientation when generating and presenting image overlays. Although the processor can be configured using hardwired logic, a typical processor in a mobile device is a general-purpose processing circuit configured by executing programs.

[0067] Mobile device 390 includes a memory or storage device system for storing data and programs. In this example, the memory system may include flash memory 440A and random access memory (RAM) 440B. RAM 440B serves as a short-term storage device for instructions and data processed by processor 430, for example, as working data processing memory. Flash memory 440A typically provides long-term storage.

[0068] Depending on the device type, mobile device 390 stores and runs a mobile operating system for executing specific applications. This mobile operating system may include a feature analyzer 344, a feature model 345, facial alignment programming 346, and a rendering engine 348. However, in some implementations, programming may be implemented in firmware or a combination of firmware and application layers. For example, instructions for acquiring images, recognizing features, analyzing features, aligning facial features, and generating overlays may reside in firmware (e.g., with a dedicated GPU or VPU SOC). Instructions for generating visible output to the user may reside in the application. Applications such as feature analyzer 344, facial alignment programming 346, and other applications may be native applications, hybrid applications, or web applications running on mobile device 390 (e.g., dynamic web pages executed by a web browser). Examples of mobile operating systems include Google Android, Apple iOS (for iPhones or iPads), Windows Mobile, Amazon Fire OS, RIM BlackBerry OS, and others.

[0069] Figure 5A , Figure 5B , Figure 5C and Figure 5D Flowcharts 500, 520, 540, and 560 illustrate exemplary operation of an image processing device (e.g., eye-wearing device 100, mobile device 390, or another electronic device capable of image processing such as a personal computer) for facial distortion in an image caused by the lens of the eye-wearing device. Facial distortion is caused by refraction due to the lens of the eye-wearing device, which will be described with reference to Figures 6A and 6B. In Figures 7A to... Figure 7D The diagram illustrates the initial image and image correction using the method. Although shown as occurring sequentially, one or more boxes in flowcharts 500, 520, 540, or 560 may be reordered or parallelized, depending on the specific implementation.

[0070] At box 502, the image processing device acquires an image including the face 700 (FIG. 7A) of an object wearing the eye-wearing device 100. The image processing device may acquire the image from the device's camera, from the device's memory, or from another device via a network connection. The image processing device may store the image in memory. In this example, acquiring the image includes capturing the image using a camera system of the eye-wearing device with augmented reality optics. According to this example, the field of view of the camera system overlaps with the field of view of the augmented reality optics.

[0071] The acquired image shows an eye-wearing device 100 including lenses 702 (right lens 702a and left lens 702b) that define a coverage area of ​​the face surrounding the eyes 704 (right eye 704a and left eye 704b). The lenses distort the facial region within the covered area to create a covered facial region 706 (right region 706a and left region 706b) within the lens's coverage area. Figure 7B As shown, the covered face region 706b has a covered face boundary 710c between points 710a (where the edge of the covered face intersects with the upper edge of the lens) and 710b (where the edge of the covered face intersects with the lower edge of the lens), which is not aligned with the uncovered face boundary 712c of the face outside the covered area between points 712a (where the edge of the uncovered face intersects with the upper edge of the frame) and 712b (where the edge of the uncovered face intersects with the lower edge of the frame).

[0072] At box 504, the image processing device segments the acquired image to detect the face 700 and the area covered by the lens. The image processing device can also identify the covered facial region (within the covered area) during segmentation. In one example, a known computer vision or machine learning model (e.g., CNN) is used to perform the segmentation. The model can be trained using unannotated images or using hundreds or thousands or more annotated images. In one example, the annotations include the pixel-precise locations of foreground elements such as hair, glasses (lens and frame), eyes, eyebrows, and skin. The detection of glasses and segmentation of images are described in the master's thesis of Paul Urtahler at Graz University of Technology in Graz, Austria—Glasses Detection and Segmentation from Face Portrait Image (December 2008), the full text of which is incorporated herein by reference.

[0073] At frame 506, the image processing device modifies the covered face region 706 within the detected coverage area of ​​lens 702 to match the covered face boundary 710C with the uncovered face boundary 712C. In the corresponding... Figures 7B to 7D The document describes three examples of how to change the covered facial area.

[0074] At box 508, the image processing device displays the altered covered facial area. In one example, the processor 403 of the mobile device 390 displays the original image with the altered covered facial area on a touchscreen display via a display driver. In another example, the processor 343 of the eye-wearing device 100 displays the altered covered facial area on an image display 180 via an image processor 312 and an image display driver 342. The altered covered facial area can be presented on the face of an eye-wearer viewed through the eye-wearing device 100. According to this example, the image processor 312 generates an overlay image from the altered covered facial area and presents the overlay image on the augmented reality optics within the detected covered area of ​​the lens.

[0075] In the example, the obtained image is one of a series of consecutive images, and the process described in reference boxes 502-508 above is repeated for each subsequent image in the series of consecutive images.

[0076] Figure 5B A flowchart 520 depicts exemplary steps for changing the detected coverage area. At box 522, server system 398 trains a machine learning model. In the example, the machine learning model is trained using multiple images of other faces wearing eyewear with lenses that distort the boundaries of the covered face, and corresponding multiple images of other faces wearing eyewear that does not distort the boundaries of the covered face. The images used for training can be annotated or unannotated.

[0077] At box 524, the image processing device applies a machine learning model. In the example, the detected covered face region and the detected face are provided as input to the machine learning model, which produces a modified covered face region as output. The modified covered face region is undistorted (see [link to example]). Figure 7D ).

[0078] At box 526, the image processing device replaces the covered face area with the modified covered face area. In the example, the processor 430 of the mobile device 390 replaces the covered face area with the modified covered face area via a driver of the touchscreen display (see [link]). Figure 7D In another example, the processor 343 of the eye-wearing device 100 replaces the covered facial area with an image processor 312 and an image display driver 343.

[0079] Figure 5C Flowchart 540 depicts additional exemplary steps for changing the detected coverage area.

[0080] At box 542, the image processing device applies an anti-refraction algorithm to the detected covered face region to produce a modified covered face region. The anti-refraction algorithm makes reference to Figures 6A to 6B... Figure 6C The refraction is reversed. Those skilled in the art will understand suitable anti-refraction algorithms from the description herein.

[0081] At box 544, the image processing device replaces the covered face area with the modified covered face area. In the example, the processor 430 of the mobile device 390 replaces the covered face area with the modified covered face area via a driver of the touchscreen display. In another example, the processor 343 of the eye-wearing device 100 replaces the covered face area using an image processor 312 and an image display driver 343.

[0082] Figure 5D Flowchart 560 depicts additional exemplary steps for changing the detected coverage area.

[0083] At box 562, the image processing device identifies the inner upper edge 710a of the covered face boundary 710c, which intersects with the upper edge of the lens at this inner upper edge. At box 564, the image processing device identifies the inner lower edge of the covered face boundary, which intersects with the lower edge of the lens at this inner lower edge. At box 566, the image processing device identifies the outer upper edge of the uncovered face boundary, which intersects with the upper edge of the lens at this outer upper edge. At box 568, the image processing device identifies the outer lower edge of the uncovered face boundary, which intersects with the lower edge of the lens at this outer lower edge. In one example, the image processing device identifies edges by applying a computer vision algorithm configured to identify edges. In another example, the image processing device identifies edges by applying a convolutional neural network configured to identify the eyewear, the covered face boundary, and the uncovered face boundary.

[0084] At box 570, the image processing device scales the covered face boundary in the horizontal direction until at least one of the inner upper edge matches the outer upper edge or the inner lower edge matches the outer lower edge. In the example, the image processing device scales the image by applying linear or other polynomial-based level stretching to the covered face region (see [link to image processing device]). Figure 7C ).

[0085] Referring now to Figure 6A, lens 702 is a thin lens made of transparent material, defined by two spherical surfaces. Because there are two spherical surfaces, there are two centers of curvature C1 and C2, and corresponding two radii of curvature R1 and R2. The line connecting the centers of curvature is called the principal axis of the lens. The center P of the thin lens lies on the principal axis and is called the optical center.

[0086] Consider a thin lens made of a medium with a refractive index of n2 placed in a medium with a refractive index of n1 (e.g., air). Let R1 and R2 be the radii of curvature at which the light enters and leaves the surface, respectively, and P be the optical center.

[0087] Consider a point object O on the principal axis. Ray OP typically falls on the surface of the sphere and passes through the lens without deviation. Ray OA falls at point A, very close to P. Rays refract at the points where they enter and exit the lens surface, forming a distorted version of the image at point I.

[0088] The general equation for refraction at a spherical surface is given by Equation 1.

[0089] (n2 / v)-(n1 / u)=(n2-n1) / R (1)

[0090] For lenses used in air, Equation 1 can be used to derive the lens maker's formula shown in Equation 2.

[0091]

[0092] Figure 6B shows two lenses, A and B. Lenses A and B, with focal lengths f1 and f2 respectively, are placed in contact with each other. The object is placed at point O, outside the focal point of the first lens A, on its common principal axis.

[0093] Lens A produces an image at I1. This image I1 serves as the object of the second lens B. The final image, as shown in Figure 6B, is produced at I. Because the lenses are very thin, a common optical center P is chosen. In terms of magnification, the magnification of the contacting lens assembly is the algebraic sum of the magnifications of the individual lenses.

[0094] Figure 6C An image processor 602 is depicted as replacing the second lens. The image processor 602 is configured to implement an anti-refraction algorithm that reverses the refraction introduced by the first lens. The anti-refraction algorithm may be the inverse of a lens manufacturer's formula or another suitable algorithm for eliminating refraction introduced by the lens. The image processor 602 may be implemented using a processor from an eyewear 100, a mobile device 390, or a server system 398.

[0095] Any methods described herein, such as feature analyzer 344, feature model 345, face alignment programming 346, and programming of rendering engine 348 for eye-wearing device 100, mobile device 390, and server system 398, may be embodied as method steps in one or more methods or in one or more applications as described above. According to some examples, an "application," "multiple applications," or "firmware" is a program that performs functions defined in a program, such as logic embodied in software or hardware instructions. Various programming languages ​​can be used to generate one or more applications structured in various ways, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a particular example, a third-party application (e.g., an entity other than a platform-specific vendor using Android) may be used. TM or iOS TM Applications developed using a Software Development Kit (SDK) can run on mobile operating systems such as iOS. TM ANDROID TM , Mobile software running on a telephone or another mobile operating system. In this example, a third-party application may invoke application programming interface (API) calls provided by the operating system to facilitate the functionality described herein. The application may be stored on any type of computer-readable medium or computer storage device and may be executed by one or more general-purpose computers. Alternatively, the methods and processes disclosed herein may be embodied in special-purpose computer hardware or application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or complex programmable logic devices (CPLDs).

[0096] The program aspect of this technology can be considered as a "product" or "article of manufacture" typically carried on or embodied in a machine-readable medium and in the form of executable code and associated data. For example, programming code may include code for navigation, eye tracking, or other functions described herein. "Storage" media include any or all tangible memory or associated modules of a computer, processor, etc., that can provide non-transitory storage for software programming at any time, such as various semiconductor memories, tape drives, disk drives, etc. All or part of the software can sometimes be transmitted via the Internet or various other telecommunications networks. Such communication, for example, enables the loading of software from one computer or processor to another, such as from server system 398 or the host computer of a service provider to the computer platform of eye-wearing device 100 and mobile device 390. Therefore, another type of medium that can carry programming, media content, or metadata files includes optical, electrical, and electromagnetic waves, such as those used via wired and optical ground networks and physical interfaces between local devices via various air links. Physical elements carrying such waves, such as wired or wireless links, optical links, etc., can also be considered as media carrying software. As used herein, unless limited to “non-transitory,” “tangible,” or “storage” media, the term “readable medium” for a computer or machine refers to any medium that participates in providing instructions or data to a processor for execution.

[0097] Therefore, machine-readable media can take many forms of tangible storage media. Non-volatile storage media include, for example, optical discs or magnetic disks, any storage device such as any computer, such as client devices, media gateways, code converters, etc., that can be used to implement the figures shown. Volatile storage media include dynamic memory, such as the main memory of computer platforms. Tangible transmission media include coaxial cables; copper wires and optical fibers, including wires that form buses within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals, or sound or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Therefore, common forms of computer-readable media include, for example: floppy disks, floppy disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punched card tapes, any other physical storage media with a perforated pattern, RAM, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cartridges, carrier waves for transmitting data or instructions, cables or links for transmitting such carrier waves, or any other media from which a computer can read program code or data. Many of these forms of computer-readable media can be used to carry one or more sequences of one or more instructions to a processor for execution.

[0098] The scope of protection is defined solely by the appended claims. When interpreted in consideration of this specification and subsequent application history, this scope is intended and should be interpreted as a broad range consistent with the ordinary meaning of the language used in the claims, and encompasses all structural and functional equivalents. Nevertheless, none of the claims is intended to include subject matter that does not meet the requirements of Sections 101, 102, or 103 of the Patent Act, nor should they be interpreted in this manner. Therefore, any unintentional inclusion of such subject matter is waived.

[0099] In addition to what has just been stated above, whether or not it is stated in the claims, the stated or described content is not intended or should not be construed as causing any part, step, feature, object, benefit, advantage or equivalent to be offered to the public.

[0100] It should be understood that, unless otherwise specified herein, the terms and expressions used herein have the general meaning consistent with those in the corresponding fields of investigation and research. Relational terms such as “first” and “second” are used only to distinguish one entity or action from another, and do not necessarily require or imply any actual such relationship or order between these entities or actions. The terms “comprising,” “including,” “containing,” “having,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes or comprises a list of elements or steps includes not only those elements or steps, but may also include other elements or steps not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element prefixed with “a” or “an” does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes that element.

[0101] Unless otherwise stated, any and all measurements, values, ratings, positions, quantities, dimensions, and other specifications set forth in this specification, including those in the appended claims, are approximate, not precise. Such quantities are intended to have a reasonable range consistent with the functions they relate to and the conventions in the fields to which they pertain. For example, unless otherwise expressly stated, parameter values, etc., can vary from said quantities by up to ±10%.

[0102] Furthermore, in the foregoing specific embodiments, various features have been combined in various examples for the purpose of simplifying this disclosure. The disclosed method should not be construed as reflecting an intention to require more features than expressly recited in each claim. Rather, as reflected in the following claims, the subject matter of the claims lies in fewer features than any single disclosed example. Therefore, the following claims are hereby incorporated into the specific embodiments, wherein each claim exists independently as a separately claimed subject matter.

[0103] While examples considered to be best practices and other examples have been described above, it should be understood that various modifications may be made therein, and the subject matter disclosed herein can be implemented in various forms and examples, and is applicable to many applications, of which only some have been described herein. The appended claims are intended to claim protection for any and all modifications and variations falling within the true scope of the inventive concept.

Claims

1. A method for resolving facial distortion in an image caused by an eye-worn device, the method comprising: Obtain an image of the face of an object wearing an eye-mount, the eye-mount including a lens defining a coverage area that covers a region of the face, the lens distorting the region of the face in the coverage area to produce a covered face region in the coverage area of ​​the lens, the covered face region having a covered face boundary that is not aligned with an uncovered face boundary outside the coverage area; The acquired image is segmented to detect the face and the coverage area of ​​the lens; The covered face region within the detected coverage area of ​​the lens is modified to match the boundary of the covered face with the boundary of the uncovered face; as well as Display the obtained image with the altered covered facial area; The changes mentioned above include: A machine learning model is applied to the detected covered face region and the detected face to generate the altered covered face region, wherein the altered covered face region is undistorted, and wherein the machine learning model is trained using multiple images of other faces wearing eyewear including lenses that distort the boundaries of the covered face and corresponding multiple images of the other faces wearing eyewear that does not distort the boundaries of the covered face. as well as Replace the covered face area with the modified covered face area.

2. The method of claim 1, wherein the segmentation further includes identifying the covered facial region.

3. The method of claim 1, wherein obtaining comprises acquiring the image from a camera system of an eye-wearing device having augmented reality optics, the field of view of the camera system overlapping with the field of view through the augmented reality optics, and wherein the method further comprises: Generate at least one overlay image from the altered covered facial area; as well as The at least one overlay image is presented on the augmented reality optical component within the detected coverage area of ​​the lens.

4. The method of claim 1, wherein the image is one of a series of consecutive images, and the method further comprises: For each subsequent image in the series of consecutive images: The subsequent image is segmented to detect the face and the coverage area of ​​the lens; Modify the covered face region within the detected coverage area of ​​the lens to match the boundary of the covered face with the boundary of the uncovered face; and The subsequent image shows the altered, covered facial area.

5. A system for resolving facial distortion in images caused by eye-worn devices, the system comprising: An image capturing device configured to acquire an image including the face of an object wearing an eye-mount, the eye-mount including a lens defining a coverage area covering a region of the face, the lens distorting the region of the face in the coverage area to produce a covered face region in the coverage area of ​​the lens, the covered face region having a covered face boundary not aligned with an uncovered face boundary outside the coverage area; A processor coupled to the image capture device, the processor being configured to segment the acquired image to detect the face and the coverage area of ​​the lens, and to modify the covered face area within the detected coverage area of ​​the lens to match the covered face boundary with the uncovered face boundary; as well as A display, coupled to the processor, is configured to display an acquired image with altered covered facial regions; Also includes: A machine learning model, which is trained using multiple images of other faces wearing eyewear including lenses that distort the boundaries of the covered face, and corresponding multiple images of the other faces wearing eyewear that does not distort the boundaries of the covered face. In order to modify the covered facial area, the processor is configured to: The machine learning model is applied to the detected covered facial region and the detected face to generate the modified covered facial region, and Replace the covered face area with the modified covered face area.

6. The system of claim 5, wherein the processor is further configured to segment the acquired image to identify the covered facial region.

7. The system according to claim 5, further comprising: An eye-wearing device having a camera system for acquiring the image and an augmented reality optics component, the field of view of the camera system overlapping the field of view of the augmented reality optics component; The processor is further configured to generate at least one overlay image from the altered covered facial area and to present the at least one overlay image on the augmented reality optical assembly within the detected covered area of ​​the lens.

8. The system of claim 5, wherein the image is one of a series of consecutive images; The processor is configured to: for each subsequent image in the series of consecutive images, segment the subsequent image to detect the face and the coverage area of ​​the lens, and modify the covered face region within the detected coverage area of ​​the lens to match the boundary of the covered face with the boundary of the uncovered face; and The display is configured to show, for each subsequent image in the series of consecutive images, the subsequent image having the changed covered facial area.

9. A non-transitory computer-readable medium for resolving facial distortion in images caused by an eye-worn device, the non-transitory computer-readable medium comprising instructions that, when executed by a processor, configure the processor to perform a function including the function of performing the following operations: Obtain an image of the face of an object wearing an eye-mount, the eye-mount including a lens defining a coverage area that covers a region of the face, the lens distorting the region of the face in the coverage area to produce a covered face region in the coverage area of ​​the lens, the covered face region having a covered face boundary that is not aligned with an uncovered face boundary outside the coverage area; The acquired image is segmented to detect the face and the coverage area of ​​the lens; Modify the covered face area within the detected coverage area of ​​the lens to match the boundary of the covered face with the boundary of the uncovered face; and Display the obtained image with the altered covered facial area; The instructions also configure the processor to perform functions that carry out the following operations: A machine learning model is applied to the detected covered face region and the detected face to generate the altered covered face region, wherein the altered covered face region is undistorted, and wherein the machine learning model is trained using multiple images of other faces wearing eyewear including lenses that distort the boundaries of the covered face and corresponding multiple images of the other faces wearing eyewear that does not distort the boundaries of the covered face. as well as Replace the covered face area with the modified covered face area.

Images