Method for providing an image and electronic device for performing the method

Abstract

A method includes: acquiring one or more first image frames associated with a first diffraction pattern and one or more second image frames associated with a second diffraction pattern using at least one sensor located below at least one display. The first and second diffraction patterns are correlated by a transformation. The method further includes: generating a first deblurred image using the one or more first image frames, and generating a second deblurred image using the one or more second image frames. The method further includes: combining the first and second deblurred images while utilizing image artifacts of complementary types generated by the first and second diffraction patterns to generate an image of a scene.

Description

Technical Field

[0001] This disclosure generally relates to imaging systems. More specifically, this disclosure relates to reducing light diffraction effects in under-display camera (UDC) systems. Background Technology

[0002] Various electronic devices represent "full-screen" devices, where most or all of the user-facing surface is a display panel. For example, many mobile electronic devices, such as smartphones and tablets, include a front surface almost entirely occupied by a display panel. Under-display cameras (UDCs) are a key enabling factor for full-screen and other electronic devices. In a UDC system, the digital camera is located beneath a semi-transparent organic light-emitting diode (OLED) display panel or other display panel. This typically hides the digital camera from the user's view, allowing the user to see only the display panel. Summary of the Invention

[0003] Technical solution This disclosure relates to reducing light diffraction effects in under-display camera (UDC) systems.

[0004] In one embodiment, a method may include: acquiring one or more first image frames associated with a first diffraction pattern and one or more second image frames associated with a second diffraction pattern using at least one sensor located below at least one display. The first and second diffraction patterns may be correlated by transformation. The method may include: generating a first deblurred image using the one or more first image frames and generating a second deblurred image using the one or more second image frames. The method may also include: combining the first and second deblurred images while utilizing image artifacts of complementary types generated by the first and second diffraction patterns to generate an image of the scene.

[0005] In an embodiment, an electronic device may include at least one display and at least one sensor located below the at least one display. The electronic device may include at least one processor, which includes processing circuitry. The electronic device may include a memory storing instructions that, when executed individually or jointly by the at least one processor, cause the electronic device to: acquire, using the at least one sensor, one or more first image frames associated with a first diffraction pattern and one or more second image frames associated with a second diffraction pattern. The first and second diffraction patterns may be correlated by transformation. When executed individually or jointly by the at least one processor, the instructions cause the electronic device to: generate a first deblurred image using the one or more first image frames and generate a second deblurred image using the one or more second image frames. When executed individually or jointly by the at least one processor, the instructions cause the electronic device to: combine the first and second deblurred images to generate an image of a scene while utilizing image artifacts of complementary types generated by the first and second diffraction patterns.

[0006] In one embodiment, a computer-readable medium instruction (e.g., a non-transitory computer-readable medium) may store instructions that, when run by at least one processor, cause at least one processor to perform the method.

[0007] Other technical features will be obvious to those skilled in the art based on the following figures, description and claims. Attached Figure Description

[0008] To gain a more complete understanding of this disclosure and its advantages, reference is now made to the following description in conjunction with the accompanying drawings, wherein: Figure 1 An example network configuration including electronic devices is shown according to this disclosure; Figure 2 An example architecture for reducing optical diffraction effects in an under-display camera (UDC) system is shown according to this disclosure; Figure 3 Example results are shown of reducing optical diffraction effects in a UDC system by using different diffraction patterns, according to this disclosure; Figure 4A and Figure 4B Example combinations of modulation transfer functions associated with different diffraction patterns used with UDC systems according to this disclosure are shown; Figure 5 The following is shown in accordance with this disclosure: Figure 2 Example merge operations in the architecture; Figures 6 to 8 The following is shown in accordance with this disclosure: Figure 5 Example alignment functionality in the merge operation; and Figure 9 An example method for reducing optical diffraction effects in a UDC system, according to this disclosure, is shown. Detailed Implementation

[0009] Before proceeding with the detailed embodiments below, it may be advantageous to define the specific words and phrases used in this patent document. The terms “send,” “receive,” and “communicate,” and their derivatives, include both direct and indirect communication. The terms “comprise,” “include,” and their derivatives, mean including but not limited to. The term “or” is inclusive, indicating and / or. The phrase “associated with,” and its derivatives, mean including, being included within, interconnected with, containing, contained within, connected to or connected with, combined with or combined with, communicable with, cooperating with, interleaved, juxtaposed, proximate, bound to or bound with, having, possessing the properties of, having a relation to or related to, etc. It should be understood that, unless the context explicitly states otherwise, the singular form includes plural references. Thus, for example, a reference to “component surface” includes a reference to one or more such surfaces. It should be understood that the boxes in each flowchart and combinations of flowcharts can be executed by one or more computer programs including computer-executable instructions. One or more computer programs may be stored in a single memory, or one or more computer programs may be divided into different parts stored in different multiple memories.

[0010] Furthermore, the various functions described below may be implemented or supported by one or more computer programs, each of which is formed by computer-readable program code and embodied in a computer-readable medium. The terms "application" and "program" refer to one or more computer programs, software components, instruction sets, procedures, functions, objects, classes, instances, associated data, or portions thereof suitable for implementation in suitable computer-readable program code. The phrase "computer-readable program code" includes any type of computer code, including source code, object code, and executable code. The phrase "computer-readable program code" includes any type of media accessible by a computer, such as read-only memory (ROM), random access memory (RAM), hard disk drive, optical disc (CD), digital video disc (DVD), or any other type of storage. "Non-transitory" computer-readable media excludes wired, wireless, optical, or other communication links that transmit transient electrical or other signals. Non-transitory computer-readable media includes media that permanently store data and media that store data and can be rewritten later, such as rewritable optical discs or erasable memory devices.

[0011] As used herein, terms and phrases such as “having,” “may have,” “comprising,” or “may include” indicate the presence of a feature (such as a number, function, operation, or component, like a part) and do not exclude the presence of other features. Furthermore, as used herein, the phrases “A or B,” “at least one of A and / or B,” or “one or more of A and / or B” can include all possible combinations of A and B. For example, “A or B,” “at least one of A and B,” and “at least one of A or B” can indicate all of the following: (1) including at least one A, (2) including at least one B, or (3) including at least one A and at least one B. Furthermore, as used herein, the terms “first” and “second” can modify various components regardless of importance and do not limit the components. These terms are used only to distinguish one component from another. For example, a first user device and a second user device can indicate user devices that are different from each other, regardless of the order or importance of the devices. A first component can be referred to as a second component without departing from the scope of this disclosure, and vice versa.

[0012] It will be understood that when an element (such as a first element) is referred to as being (operably or communicatively) "combined" with / "attached to" another element (such as a second element) or "connected" with / "connected to" another element (such as a second element), it may be directly combined or connected to / attached to or connected to that other element, or combined or connected to / attached to / connected to that other element via a third element. Conversely, it will be understood that when an element (such as a first element) is referred to as being "directly combined" with / "directly attached to" another element (such as a second element) or "directly connected" with / "directly connected to" another element (such as a second element), no other element (such as a third element) is between that element and that other element.

[0013] As used herein, the phrase “configured (or set) to” may be used interchangeably with the phrases “suitable for,” “capable of,” “designed to,” “adapted to,” “manufactured as,” or “capable of”, depending on the context. The phrase “configured (or set) to” does not inherently mean “specifically designed to” in hardware. Rather, the phrase “configured to” may indicate that a device can perform operations in conjunction with another device or component. For example, the phrase “processor configured (or set) to perform A, B, and C” may refer to a general-purpose processor (such as a CPU or application processor) that can perform operations by running one or more software programs stored in a memory device, or a special-purpose processor (such as an embedded processor) for performing operations.

[0014] The terms and phrases used herein are provided only to describe some embodiments of this disclosure and are not intended to limit the scope of other embodiments of this disclosure. It should be understood that, unless the context clearly indicates otherwise, the singular form includes plural references. All terms and phrases used herein (including technical and scientific terms and phrases) have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of this disclosure pertain. It will also be understood that terms and phrases (such as those defined in common dictionaries) should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly defined herein. In some cases, the terms and phrases defined herein may be interpreted as excluding embodiments of this disclosure.

[0015] Examples of "electronic devices" according to embodiments of this disclosure may include at least one of the following: smartphones, tablet PCs, mobile phones, video phones, e-book readers, desktop PCs, laptop computers, netbooks, workstations, personal digital assistants (PDAs), portable multimedia players (PMPs), MP3 players, mobile medical devices, cameras, or wearable devices (such as smart glasses, head-mounted displays (HMDs), electronic clothing, electronic bracelets, electronic necklaces, electronic accessories, electronic tattoos, smart mirrors, or smartwatches). Other examples of electronic devices include smart home appliances. Examples of smart home appliances may include at least one of the following: televisions, digital video disc (DVD) players, audio players, refrigerators, air conditioners, vacuum cleaners, ovens, microwave ovens, washing machines, dryers, air purifiers, set-top boxes, home automation control panels, security control panels, TV boxes, game consoles, smart speakers or speakers with integrated digital assistants, electronic dictionaries, electronic keys, cameras, or electronic photo frames.

[0016] Other examples of electronic devices include at least one of the following: various medical devices (such as various portable medical measurement devices (e.g., blood glucose measuring devices, heart rate measuring devices, or body temperature measuring devices), magnetic resonance angiography (MRA) devices, magnetic resonance imaging (MRI) devices, computed tomography (CT) devices, imaging devices, or ultrasound devices), navigation devices, global positioning system (GPS) receivers, event data loggers (EDR), flight data loggers (FDR), automotive infotainment devices, navigation electronics (such as navigation devices or gyrocompasses), avionics devices, safety devices, vehicle head units, industrial or household robots, automated teller machines (ATMs), point-of-sale (POS) devices, or Internet of Things (IoT) devices (such as light bulbs, various sensors, electricity or gas meters, sprinklers, fire alarms, thermostats, streetlights, toasters, fitness equipment, hot water tanks, heaters, or boilers). Other examples of electronic devices include at least a part of a piece of furniture or building / structure, electronic boards, electronic signature receivers, projectors, or various measuring devices (such as devices for measuring water, electricity, gas, or electromagnetic waves). It should be noted that, according to various embodiments of this disclosure, the electronic device may be one or a combination of the devices listed above. According to embodiments of this disclosure, the electronic device may be a flexible electronic device. The electronic devices disclosed herein are not limited to the devices listed above and may include new electronic devices as the technology develops.

[0017] In the following description, electronic devices are described with reference to the accompanying drawings according to various embodiments of the present disclosure. As used herein, the term "user" may refer to a person using the electronic device or another device (such as an artificial intelligence electronic device).

[0018] Definitions for other specific words and phrases are provided throughout this patent document. It will be understood by those skilled in the art that, in many cases (and even most cases), these definitions apply to the prior and future use of the words and phrases defined in this way.

[0019] The descriptions in this application should not be construed as implying that any particular element, step, or function is an essential element that must be included within the scope of the claims. The scope of the patent subject matter is defined solely by the claims. Furthermore, unless the exact phrase "method for..." is followed by a participle, none of the claims are intended to invoke 35 USC. 112(f). Any other term used in the claims (including, but not limited to, “mechanism,” “module,” “device,” “unit,” “component,” “element,” “building,” “device,” “machine,” “system,” “processor,” or “controller”) is understood by the applicant to refer to a structure known to a person skilled in the art and is not intended to invoke 35 USC. 112 (f).

[0020] The following discussion is described with reference to the accompanying drawings. Figures 1 to 9 Various embodiments of this disclosure are also described. However, it should be understood that this disclosure is not limited to these embodiments, and all changes and / or equivalents or substitutions thereof are also within the scope of this disclosure. Throughout the specification and drawings, the same or similar reference numerals may be used to refer to the same or similar elements.

[0021] As mentioned above, various electronic devices represent "full-screen" devices, where most or all of the user-facing surface is a display panel. For example, many mobile electronic devices, such as smartphones and tablets, include a front surface almost entirely occupied by a display panel. Under-display cameras (UDCs) are a key enabling factor for full-screen and other electronic devices. In a UDC system, the digital camera is located beneath a semi-transparent organic light-emitting diode (OLED) display panel or other display panel. This typically hides the digital camera from the user's view, allowing the user to see only the display panel.

[0022] Unfortunately, placing a digital camera under a display panel often introduces numerous problems into images captured using that camera. For example, the periodic patterns of OLED or other display panels, along with the limited amount of light that can transmit through them, can produce various artifacts (such as noise and light diffraction artifacts) in images captured using a UDC system. This can significantly degrade the overall image quality (IQ) of the resulting image, potentially reducing user satisfaction. Some approaches have undertaken considerable work to tune the display panel to balance the trade-off between image quality and visibility from an under-display camera. However, these approaches may still suffer from various issues, such as low signal-to-noise ratio (SNR), significant light diffraction artifacts, and low resolution.

[0023] This disclosure provides various techniques for reducing optical diffraction effects in UDC systems. As described in more detail below, one or more first image frames associated with a first diffraction pattern and one or more second image frames associated with a second diffraction pattern can be obtained using at least one under-display camera. The first and second diffraction patterns may be associated with a periodic pattern of an OLED display panel or other display panel located above at least one under-display camera. The first and second diffraction patterns may be correlated with each other by transformation, such as when the second diffraction pattern represents a rotated version of the first diffraction pattern. A first deblurred image and a second deblurred image can be generated using the first and second image frames, such as by applying multi-frame processing to a sequence of first image frames and applying multi-frame processing to a sequence of second image frames to produce a mixed or otherwise processed image, and applying the inverse of a point spread function to the processed image to produce a deblurred image. The first and second deblurred images can be combined to generate an image of a scene while utilizing image artifacts of complementary types generated by the first and second diffraction patterns. For example, the deblurred images can be merged by: (i) performing an alignment operation to generate an aligned version of the deblurred image; and (ii) merging the aligned versions of the deblurred images.

[0024] In this way, the UDC system can generate deblurred images of a scene using different image frames, where different image frames can be captured using different diffraction patterns associated with the display. For example, different image frames can also be captured from different viewpoints. This results in complementary types of image artifacts within the deblurred image. The different diffraction patterns used to obtain the image frames can be implemented in various ways, such as by rotating at least one under-display camera, rotating at least a portion of the display relative to at least one under-display camera, integrating different grid patterns into the display, and / or rotating electronics including the display and at least one under-display camera. The complementary nature of the artifacts within the deblurred image can be utilized during the merging process to improve the final image quality of the resulting image. Since different deblurred images can have different light responses when using different diffraction patterns (and possibly when facing the light source from different angles), image quality can be improved (often significantly) by combining deblurred images with different light responses. As a result, the overall image generated using these techniques can have an improved signal-to-noise ratio, fewer light source diffraction artifacts, and / or improved resolution.

[0025] It should be noted that although the embodiments discussed below are described in the context of use in a specific type of consumer electronic device (such as a smartphone), this is merely an example. It will be understood that the principles of this disclosure can be implemented in any number of other suitable scenarios and with any suitable apparatus. It should also be noted that although the various embodiments discussed below are described based on the assumption that a single device receives and processes image frames to generate an image of a scene, this is also merely an example. It will be understood that the principles of this disclosure can be implemented with any number of devices (including one device for capturing image frames and another device for processing image frames). Generally, this disclosure is not limited to use with any particular type of device.

[0026] Figure 1 An example network configuration 100 including electronic devices according to this disclosure is shown. Figure 1 The embodiment of network configuration 100 shown is for illustrative purposes only. Other embodiments of network configuration 100 may be used without departing from the scope of this disclosure.

[0027] According to embodiments of this disclosure, electronic device 101 may be included in network configuration 100. Electronic device 101 may include at least one of bus 110, processor 120, memory 130, input / output (I / O) interface 150, display 160, communication interface 170, and sensor 180. In embodiments, electronic device 101 may not include at least one of these components, or at least one other component may be added. Bus 110 includes circuitry for connecting components 120-180 to each other and for transmitting communications (such as control messages and / or data) between components 120-180.

[0028] Processor 120 may include one or more processing devices, such as one or more microprocessors, microcontrollers, digital signal processors (DSPs), application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). In embodiments, processor 120 may include one or more of a central processing unit (CPU), application processor (AP), communication processor (CP), or graphics processing unit (GPU). Processor 120 is capable of performing control and / or performing operations or data processing related to communication or other functions on at least one of the other components of electronic device 101. As described below, processor 120 may be used to acquire multiple image frames from one or more sensors 180 located below display 160 and process the image frames to generate a final image of the scene while reducing light diffraction effects. Processor may include various processing circuitry and / or multiple processors. For example, as used herein (including the claims), the term "processor" may include various processing circuitry including at least one processor, wherein one or more of the at least one processor may be configured individually and / or in a distributed manner to perform the various functions described herein. As used herein, when “processor,” “at least one processor,” and “one or more processors” are described as being configured to perform a number of functions, these terms cover, for example, a situation where one processor performs some of the functions and another processor performs other functions, as well as a situation where a single processor can perform all of the functions. Additionally, at least one processor may comprise, for example, a combination of processors performing various stated / disclosed functions in a distributed manner. At least one processor can execute program instructions to implement or perform various functions.

[0029] Memory 130 may include volatile memory and / or non-volatile memory. For example, memory 130 may store commands or data associated with at least one other component of electronic device 101. In embodiments of this disclosure, memory 130 may store software and / or program 140. Program 140 may include, for example, kernel 141, middleware 143, application programming interface (API) 145, and / or application program (or “application”) 147. At least a portion of kernel 141, middleware 143, or API 145 may be represented as an operating system (OS).

[0030] Kernel 141 can control or manage system resources (such as bus 110, processor 120, or memory 130) used to perform operations or functions implemented in other programs (such as middleware 143, API 145, or application 147). Kernel 141 can provide an interface that allows middleware 143, API 145, or application 147 to access various components of electronic device 101 to control or manage system resources. Application 147 may include one or more applications for receiving and processing image frames while reducing optical diffraction effects. Functions associated with receiving and processing image frames while reducing optical diffraction effects may be performed by a single application or by multiple applications, each performing one or more of these functions. For example, middleware 143 may act as a repeater to allow API 145 or application 147 to communicate data with kernel 141. Multiple applications 147 may be provided. Middleware 143 is capable of controlling work requests received from application 147, such as by prioritizing system resources (such as bus 110, processor 120, or memory 130) of electronic device 101 for at least one of the applications 147. API 145 may be an interface that allows application 147 to control functionality provided from kernel 141 or middleware 143. For example, API 145 may include at least one interface or function (such as commands) for file control, window control, image processing, or text control.

[0031] I / O interface 150 can be used as an interface, for example, to transmit commands or data input from a user or other external device to other components of electronic device 101. I / O interface 150 can also output commands or data received from other components of electronic device 101 to the user or other external device.

[0032] Display 160 may include, for example, a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, a quantum dot light-emitting diode (QLED) display, a microelectromechanical system (MEMS) display, or an electronic paper display. Display 160 may also be a depth-sensing display (such as a multi-focal display). Display 160 is capable of displaying various content to a user, such as text, images, videos, icons, or symbols. Display 160 may include a touchscreen and can receive input such as touch, gestures, proximity, or hover input using an electronic pen or a user's body part.

[0033] Communication interface 170, for example, enables communication between electronic device 101 and external electronic devices (such as first external electronic device 102, second electronic device 104, or server 106). For instance, communication interface 170 can be connected to network 162 or 164 via wireless or wired communication to communicate with external electronic devices. Communication interface 170 can be a wired transceiver or a wireless transceiver or any other component for transmitting and receiving signals (such as images).

[0034] Wireless communication can use at least one of the following as a communication protocol: WiFi, Long Term Evolution (LTE), LTE-A Advanced, 5G, millimeter wave or 60 GHz wireless communication, wireless USB, Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Universal Mobile Telecommunications System (UMTS), Wireless Broadband (WiBro), or Global System for Mobile Communications (GSM). Wired connections may include at least one of the following: Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), Recommended Standard 232 (RS-232), or Common Older Telephone Service (POTS). Networks 162 or 164 may include at least one communication network, such as a computer network (e.g., a Local Area Network (LAN) or Wide Area Network (WAN)), the Internet, or a telephone network.

[0035] Electronic device 101 may also include one or more sensors 180 that can measure physical quantities or detect the activation state of electronic device 101 and convert the measured or detected information into electrical signals. For example, one or more sensors 180 may include one or more cameras or other imaging sensors that can be used to capture images of a scene. One or more cameras or other imaging sensors may include one or more under-display cameras that may represent one or more imaging sensors located below at least one display 160 and capturing illumination passing through at least one display 160. Sensor 180 may also include one or more buttons for touch input, one or more microphones, depth sensors, gesture sensors, gyroscopes or gyroscope sensors, atmospheric pressure sensors, magnetic sensors or magnetometers, accelerometers or accelerometers, grip sensors, proximity sensors, color sensors (such as RGB sensors), biophysical sensors, temperature sensors, humidity sensors, illuminance sensors, ultraviolet (UV) sensors, electromyography (EMG) sensors, electroencephalography (EEG) sensors, electrocardiography (ECG) sensors, infrared (IR) sensors, ultrasound sensors, iris sensors, or fingerprint sensors. Sensor 180 may also include an inertial measurement unit, which may include one or more accelerometers, gyroscopes, and other components. Sensor 180 may include control circuitry for controlling at least one of the sensors. Any one of the sensors 180 may be located within electronic device 101.

[0036] In this embodiment, the first external electronic device 102 or the second external electronic device 104 may be a wearable device or an electronically mountable wearable device (such as an HMD). When electronic device 101 is mounted in the first external electronic device 102 (such as an HMD), electronic device 101 can communicate with the first external electronic device 102 via communication interface 170. Electronic device 101 can be directly connected to the first external electronic device 102 to communicate with it without involving a separate network. Electronic device 101 may also be an augmented reality wearable device (such as glasses) including one or more imaging sensors.

[0037] The first external electronic device 102, the second external electronic device 104, and the server 106 may be devices of the same or different types as electronic device 101. According to embodiments of this disclosure, server 106 may include a combination of one or more servers. Furthermore, according to embodiments of this disclosure, all or some of the operations performed on electronic device 101 may be performed on another electronic device or multiple other electronic devices (such as the first external electronic device 102, the second external electronic device 104, or the server 106). Furthermore, according to embodiments of this disclosure, when electronic device 101 automatically or on request performs some functions or services, electronic device 101 may request another device (such as the first external electronic device 102, the second external electronic device 104, or the server 106) to perform at least some of its associated functions, rather than performing the function or service alone; or electronic device 101 may request another device (such as the first external electronic device 102, the second external electronic device 104, or the server 106) to perform at least some of its associated functions in addition to running the function or service. Another electronic device (such as a first external electronic device 102 and a second external electronic device 104 or a server 106) is capable of performing the requested function or additional function and transmitting the result of the performance to electronic device 101. Electronic device 101 can provide the requested function or service by processing the received result as is or additionally. For example, cloud computing, distributed computing, or client-server computing technologies can be used. Although Figure 1 The electronic device 101 is shown to include a communication interface 170 for communicating with a second external electronic device 104 or a server 106 via a network 162 or 164, but according to embodiments of this disclosure, the electronic device 101 can operate independently without separate communication functionality.

[0038] Server 106 may include components 110-180 (or suitable subsets thereof) that are the same as or similar to those in electronic device 101. Server 106 may support driving electronic device 101 by performing at least one of the operations (or functions) implemented on electronic device 101. For example, server 106 may include a processing module or processor that can support processor 120 implemented in electronic device 101.

[0039] although Figure 1 An example of a network configuration 100 including electronic device 101 is shown, but it is possible to modify it. Figure 1 Various changes can be made. For example, network configuration 100 can include any number of each component in any suitable arrangement. Typically, computing and communication systems can have a wide variety of configurations, and Figure 1 This disclosure is not intended to limit the scope to any particular configuration. Furthermore, although... Figure 1 An operating environment is shown in which the various features disclosed in this patent document can be used, but these features can be used in any other suitable system.

[0040] Figure 2 An example architecture 200 for reducing optical diffraction effects in a UDC system according to this disclosure is shown. For ease of explanation, Figure 2 The architecture 200 shown is described as being in Figure 1 Implemented on or by electronic device 101 in network configuration 100 Figure 1 The electronic device 101 in the network configuration 100 is supported. However, Figure 2 At least a portion of the architecture 200 shown can be used with any other suitable device and in any other suitable system, such as when at least a portion of the architecture 200 is implemented on or supported by the server 106.

[0041] like Figure 2As shown, architecture 200 can receive and process different image frames 202 and 204. In one embodiment, a single first image frame 202 and a single second image frame 204 can be acquired and processed. In another embodiment, a sequence of two or more first image frames 202 can be acquired and processed, and / or a sequence of two or more second image frames 204 can be acquired and processed. Each of the first image frame 202 and the second image frame 204 can be acquired during image capture operation using at least one sensor 180 of electronic device 101 (such as at least one camera or at least one imaging sensor). At least one sensor 180 may include one or more under-display cameras. At least one sensor 180 (such as at least one camera or at least one imaging sensor) can acquire the first image frame 202 and the second image frame 204. Depending on the implementation, the same under-display camera can be used to acquire different image frames 202 and 204, or different under-display cameras can be used to acquire different image frames 202 and 204. Depending on the implementation, the same under-display camera can acquire different image frames 202 and 204, or different under-display cameras can acquire different image frames 202 and 204. Each of the different image frames 202 or 204 can have any suitable resolution and format, such as Bayer image format or other raw image formats, red-green-blue (RGB) image format, or luminance-chrominance (YUV) image format. In the following discussion, it may be assumed that multiple first image frames 202 in the sequence can be acquired and processed, and multiple second image frames 204 in the sequence can be acquired and processed; however, this assumption is for illustration and explanation only.

[0042] In embodiments, each sequence of the first image frame 202 and the second image frame 204 may include two or more image frames captured using different capture conditions. Capture conditions may represent any suitable settings of the electronic device 101 or other means for capturing the first image frame 202 and the second image frame 204 or any suitable content of the scene being imaged. For example, capture conditions may represent different exposure settings (such as different exposure times or ISO settings) of the imaging sensor used to capture image frames 202 or 204. In a multi-frame processing pipeline, for example, different exposure settings may be used to capture multiple image frames 202 or 204, such that portions of different image frames 202 or 204 can be combined to produce a high dynamic range (HDR) output image or other mixed images. When capturing a dynamic scene, image frames 202 or 204 may have different image content.

[0043] In an embodiment, a first diffraction pattern 206 may be used to obtain (e.g., capture) one or more first image frames 202, and a second diffraction pattern 208 may be used to obtain (e.g., capture) one or more second image frames 204. The first diffraction pattern 206 may obtain (e.g., capture) one or more first image frames 202, and the second diffraction pattern 208 may obtain (e.g., capture) one or more second image frames 204. The first diffraction pattern 206 and the second diffraction pattern 208 may represent patterns contained in a display 160 located above a camera or other imaging sensor (such as an under-display camera) used to capture image frames 202 and 204. The first diffraction pattern 206 may be associated with a first pattern of the display 160, with the camera or other imaging sensor (such as an under-display camera) located below the display 160. The second diffraction pattern 208 may be associated with a second pattern of the display 160, with the camera or other imaging sensor (such as an under-display camera) located below the display 160. Each of the first diffraction pattern 206 and the second diffraction pattern 208 can be formed by elements of an OLED display panel or other display located above a camera or other imaging sensor (such as an under-display camera) used to capture image frames. For example, the display panel (such as an OLED display panel or other display) may include wires, repeating or periodic structures (such as light-emitting diodes), and other components for generating illumination. Components included in the display panel can partially block some of the light passing through the display panel and can produce diffraction in the light passing through the display panel. Diffraction in the light passing through the display panel can be very difficult to avoid using optical techniques.

[0044] As described below, the processing operations can be used to reduce diffraction artifacts in images generated using a mixture or other processing methods, such as first image frame 202 and second image frame 204. For example, by capturing different image frames 202 and 204 using different diffraction patterns 206 and 208, artifacts caused by diffraction in architecture 200 can be significantly reduced or minimized. The first image frame 202 can be captured using the first diffraction pattern 206, and the second image frame 204 can be captured using the second diffraction pattern 208. The first diffraction pattern 206 and the second diffraction pattern 208 are different from each other but related via a specified transformation. Known relationships or transformations can be used to define how the second diffraction pattern 208 is related to the first diffraction pattern 206 (and vice versa). For example, the second diffraction pattern 208 may represent a rotated version of the first diffraction pattern 206, in which case the specified transformation may be a rotation of the first diffraction pattern 206. As described below, using different diffraction patterns 206 and 208 when capturing different image frames 202 and 204 allows different image frames 202 and 204 (and mixed or otherwise processed images generated using different image frames 202 and 204) to have complementary types of image artifacts. As an example, spikes generated around a light source may extend in a specific direction in the first image frame 202 captured using the first diffraction pattern 206, and may extend in different directions in the second image frame 204 captured using the second diffraction pattern 208. Complementary types of image artifacts can be used as described below to reduce or minimize the presence of these image artifacts in the final image of the scene generated using different image frames 202 and 204.

[0045] In one embodiment, a sequence of first image frames 202 may be provided to a multi-frame processing (MFP) operation 210. The MFP operation 210 is operable to mix or otherwise combine the contents of the first image frames 202. In another embodiment, a sequence of second image frames 204 may be provided to a MFP operation 212. The MFP operation 212 is operable to mix or otherwise combine the contents of the second image frames 204. For example, each of the MFP operations 210, 212 may process the associated sequence of image frames 202 or 204 to determine how the image frames 202 or 204 can be mixed to produce a mixed or otherwise processed image 214 or 216. The MFP operation 210 may generate the processed image 214 by combining the sequence of the first image frames 202, and the MFP operation 212 may generate the processed image 216 by combining the sequence of the second image frames 204. Each processed image 214 or 216 may represent an HDR image or other images with improved characteristics relative to the respective image frames 202 or 204. Each multi-frame processing operation 210, 212 may include any suitable logic for mixing or otherwise combining multiple image frames. If a single image frame 202 or 204 is obtained, the multi-frame processing operation 210 or 212 (or both) may be replaced by other image processing functions that can process image frame 202 or 204 and generate processed image 214 or 216.

[0046] Processed images 214 and 216 from multi-frame processing operations 210 and 212 can be provided to point spread function inversion (PSFI) operations 218 and 220, respectively. Each of PSFI operations 218 and 220 is operable to partially remove diffraction artifacts from the associated processed image 214 or 216. Therefore, each PSFI operation 218 and 220 can receive the processed image 214 or 216 and operate to produce a sharper or deblurred image 222 or 224. Each of PSFI operations 218 and 220 may include any suitable logic for generating a deblurred (less blurred) image based on the incoming image. In embodiments, each of PSFI operations 218 and 220 may be implemented using a neural network that has been trained to receive a blurred image and generate a sharper image.

[0047] Deblurred images 222 and 224 can be provided to merging operation 226. Merging operation 226 is operable to combine the contents of deblurred images 222 and 224 to generate image 228 of the scene. For example, merging operation 226 can process one or more of deblurred images 222 and 224 to align deblurred images 222 and 224 and generate aligned versions of deblurred images 222 and 224. Merging operation 226 can combine or merge the contents of aligned versions of deblurred images 222 and 224. Various techniques can be used to merge the contents of aligned versions of deblurred images 222 and 224, and one or more specific techniques used may be based on one or more specific factors or criteria. For example, when merging the contents of aligned versions of deblurred images 222 and 224, averaging, weighted averaging, or minimization may be used, and details of these example techniques are provided below. Depending on the implementation, the same merging technique may be used to combine all portions of deblurred images 222 and 224, or different merging techniques may be used to combine different portions of deblurred images 222 and 224. For example, the specific merging technique applied may be based on one or more statistics associated with deblurred images 222 and 224 or portions thereof. As described below, due to the complementary nature of the image artifacts generated when capturing the first image frame 202 and the second image frame 204 using the first diffraction pattern 206 and the second diffraction pattern 208, merging the deblurred images 222 and 224 can help reduce or minimize the presence of diffraction-based image artifacts in image 228.

[0048] Any desired post-processing of image 228 can be performed to generate a final image of the scene. For example, image 228 can be provided to tone mapping operation 230. Tone mapping operation 230 can be used to adjust the colors in image 228 to generate output image 232. Tone mapping operation 230 can be useful or important in various applications, such as when generating HDR images. For example, since generating an HDR image may involve capturing multiple images of a scene using different exposures and combining the captured images to produce an HDR image, this type of processing often results in unnatural tones within the HDR image. Therefore, tone mapping operation 230 can use one or more color maps to adjust the colors contained in image 228. For example, tone mapping operation 230 can be performed when the HDR image will be presented on a display device with a smaller dynamic range than the HDR image itself, such that output image 232 can be displayed or otherwise presented in a form with a smaller dynamic range. Tone mapping operation 230 may include any suitable logic for adjusting the image tones, such as converting various colors in image 228 to other colors in output image 232 using at least one lookup table.

[0049] although Figure 2An example of an architecture 200 for reducing optical diffraction effects in a UDC system is shown, but it is applicable to other systems. Figure 2 Make various changes. For example, Figure 2 The various components and functions can be combined, further subdivided, copied, or rearranged as needed. Furthermore, one or more additional components and functions may be included if required or desired. Additionally, image frames 202 and 204 may undergo any desired preprocessing operations before processing, and the generated image 228 may undergo any desired post-processing operations before generating the output image 232. Figure 2 Various pairs of operations that occur in parallel are illustrated (such as multi-frame processing operations 210 and 212 or PSFI operations 218 and 220). In embodiments, architecture 200 may perform common operations sequentially multiple times to achieve the same or similar results.

[0050] Figure 3 Example results are shown of reducing optical diffraction effects in a UDC system by using different diffraction patterns, according to this disclosure. More specifically, Figure 3 It shows Figure 2 The architecture 200 can be used to generate image frames 202 and 204 using different diffraction patterns 206 and 208, and image frames 202 and 204 can be used to generate output image 232 with reduced or minimized diffraction artifacts.

[0051] like Figure 3 As shown, using the first diffraction pattern 206 can result in the generation of specific diffraction artifacts (such as the diffraction artifacts shown in image data 302). For example, a display 160 located above a sensor 180 (such as an under-display camera) can have a pattern that serves as a binary aperture, and the binary aperture can produce a point spread function (PSF) containing multiple peaks. Figure 3 The generated point spread function (PSF) containing multiple peaks is clearly visible in image data 302 shown. Due to the wires, light-emitting diodes (LEDs), or other components of display 160, the binary aperture contains multiple straight edges, and the peaks in the point spread function may be approximately perpendicular to those edges. Similarly, using a second diffraction pattern 208 can result in the generation of specific diffraction artifacts (such as the diffraction artifacts shown in image data 304). Again, display 160, located above sensor 180 (such as an under-display camera), may have a pattern serving as a binary aperture, and the binary aperture can produce a point spread function containing multiple peaks. Due to the wires, LEDs, or other components of display 160, the binary aperture contains multiple straight edges, and the peaks in the point spread function may be approximately perpendicular to those edges.

[0052] However, in Figure 3In the example shown, the first diffraction pattern 206 and the second diffraction pattern 208 may not be identical, but may be related in some way (such as by rotation in this particular example). The second diffraction pattern 208 can be formed by rotating the first diffraction pattern 206. Therefore, the peaks of diffraction artifacts in image data 302 and 304 are misaligned with each other and are generally offset from each other angularly. As a particular example, if the second diffraction pattern 208 is rotated about 45° relative to the first diffraction pattern 206, the peaks of diffraction artifacts in image data 304 can also be rotated about 45° relative to the peaks of diffraction artifacts in image data 302. When the deblurred images 222 and 224 containing image data 302 and 304 are merged by merging operation 226, the peaks of misaligned diffraction artifacts can be reduced or minimized in image 228. Figure 3 An example of merging operation 226 is shown, in which merging operation 226 combines deblurred images 222 and 224 containing diffraction artifacts of image data 302 and 304, and produces image 228 containing image data 306. It can be seen here that image data 306 is much clearer and may contain very few or no diffraction artifacts. In other words, by preventing the peaks of diffraction artifacts in deblurred images 222 and 224 from aligning with each other, the peaks in image 228 can be significantly reduced or minimized by merging operation 226.

[0053] When capturing different image frames 202 and 204, there are various ways to obtain different diffraction patterns 206 and 208. As an example, one or more imaging sensors can be used to capture image frames 202 and 204, and the one or more imaging sensors can be rotatable. An imaging sensor can capture one or more image frames 202 using a first diffraction pattern 206, and the imaging sensor can be rotated; an imaging sensor can also capture one or more image frames 204 using a second diffraction pattern 208. As an example, at least a portion of one or more displays 160 located above one or more imaging sensors can be rotatable. An imaging sensor can capture one or more image frames 202 using the first diffraction pattern 206, and the display 160 can be rotated; an imaging sensor can also capture one or more image frames 204 using the second diffraction pattern 208. As an example, different imaging sensors can be used to capture image frames 202 and 204, and different imaging sensors can be located below different portions of one or more displays 160 with different diffraction patterns. Different imaging sensors 180 can capture different image frames 202 and 204 using different diffraction patterns 206 and 208. As an example, one or more imaging sensors may be used to capture image frames 202, 204, and the electronics 101 itself may be rotated (e.g., by a user). The imaging sensor may capture one or more image frames 202 using a first diffraction pattern 206, and the electronics 101 may be rotated; the imaging sensor may also capture one or more image frames 204 using a second diffraction pattern 208. It should be noted that these examples are for illustrative purposes only, and any other or additional mechanisms may be used to obtain different diffraction patterns 206, 208. It should also be noted that any combination of these and / or other methods may be used. Generally, this disclosure is not limited to any particular technique for obtaining different diffraction patterns 206, 208.

[0054] These examples demonstrate that various techniques or combinations thereof can be used to achieve the desired diffraction patterns 206 and 208 used when capturing different image frames 202 and 204. Regardless of how the different diffraction patterns 206 and 208 are actually implemented, the end result is the ideal generation of multiple deblurred images 222 and 224 with complementary or substantially non-overlapping diffraction artifacts. The multiple deblurred images 222 and 224 can be merged in a manner that reduces or minimizes the occurrence of those diffraction artifacts in the generated image 228. As a result, image 228 can have various desired properties, such as improved signal-to-noise ratio, fewer light source diffraction artifacts, and / or improved resolution.

[0055] although Figure 3 An example of the results obtained by reducing optical diffraction effects in a UDC system by using different diffraction patterns is shown, but it is possible to modify the results for other systems. Figure 3Various modifications can be made. For example, specific diffraction artifacts and specific reductions in diffraction artifacts provided through merging are merely examples. The actual diffraction artifacts present in an image can vary widely based on (but not limited to) the specific scene being imaged and the specific diffraction pattern used on one or more displays. Furthermore, specific reductions in diffraction artifacts can vary widely based on (but not limited to) the specific techniques used to merge image data.

[0056] Figure 4A and Figure 4B Example combinations of modulation transfer functions associated with different diffraction patterns used with the UDC system according to this disclosure are shown. Modulation transfer function (in Figure 4A and Figure 4B The MTF curve (represented in the original text) can be used to characterize the imaging system for different spatial frequencies (in the original text). Figure 4A and Figure 4B The resolution is expressed in "pairs per millimeter" (pairs per millimeter), where a higher value indicates stronger resolving power or higher resolution. Higher resolution may be preferred because it improves the quality of the image produced by the imaging system.

[0057] like Figure 4A As shown, graph 400 may include curves 402 and 404 associated with different modulation transfer functions, where those modulation transfer functions may be associated with different diffraction patterns 206 and 208. When considering a single camera or other imaging sensor (such as an under-display camera), the resolution of a single camera or imaging sensor may follow a single MTF curve. However, because the diffraction pattern is located above the imaging sensor, the imaging sensor may have different spatial resolutions at different orientations of the diffraction pattern. Figure 4A For example, when the imaging sensor is used with the first diffraction pattern 206, curve 402 can be associated with the modulation transfer function of the imaging sensor, and when the imaging sensor is used with the second diffraction pattern 208, curve 404 can be associated with the modulation transfer function of the imaging sensor. It can be seen from this that the imaging sensor can have higher spatial resolution at a specific frequency when used with the first diffraction pattern 206, and higher spatial resolution at other frequencies when used with the second diffraction pattern 208.

[0058] By using architecture 200 (including merging operation 226), complementary portions of the modulation transfer function defined by curves 402 and 404 can be combined. Figure 4B For example, graph 420 may include curve 422 having two parts 424 and 426. More specifically, part 424 of curve 422 may represent Figure 4A The portion of curve 402 that is higher than curve 404, while portion 426 of curve 422 can be represented. Figure 4AThe portion of curve 404 above curve 402. The modulation transfer function defined by curve 422 may include the higher resolution portions of both curves 402 and 404. The merging operation 226 may utilize the complementary properties of the different diffraction patterns 206 and 208, such as by combining (i) image data with higher spatial resolution associated with the first diffraction pattern 206 and (ii) image data with higher spatial resolution associated with the second diffraction pattern 208. When generating image 228, the merging operation 226 may preserve the higher resolution portions of the different deblurred images 222 and 224. Thus, an image 228 with reduced or minimized diffraction artifacts can be generated.

[0059] although Figure 4A and Figure 4B An example of a combination of modulation transfer functions associated with different diffraction patterns used with the UDC system is shown, but it is possible to apply to... Figure 4A and Figure 4B Various modifications can be made. For example, the modulation transfer function shown here is merely an example and only illustrates how different higher-resolution portions of the deblurred images 222 and 224 can be combined. The actual modulation transfer function associated with architecture 200 can vary widely depending on the implementation of architecture 200.

[0060] Figure 5 The following is shown in accordance with this disclosure: Figure 2 Example merging operation 226 in architecture 200. As described above, merging operation 226 can be used to combine image data from deblurred images 222 and 224 to generate image 228 of the scene. Deblurred images 222 and 224 may contain complementary diffraction artifacts associated with different diffraction patterns 206 and 208, and merging operation 226 can combine deblurred images 222 and 224 such that the resulting image 228 has reduced or minimized diffraction artifacts.

[0061] like Figure 5As shown, deblurred images 222 and 224 can be provided to alignment function 502. Alignment function 502 is operable to modify at least one of deblurred images 222 and 224 to generate aligned versions of deblurred images 222 and 224. For example, alignment function 502 can modify deblurred image 222 to generate a modified deblurred image 504. The modified deblurred images 504 and 224 can represent aligned versions of deblurred images 222 and 224. The modified deblurred image 504 can be spatially aligned substantially with corresponding points in deblurred image 224. It should be noted that although alignment function 502 is shown here as modifying only deblurred image 222, alignment function 502 can modify only deblurred image 222 or both deblurred images 222 and 224 to generate aligned versions of deblurred images 222 and 224. Alignment function 502 can use various methods (such as global alignment and / or local alignment) to generate aligned versions of the deblurred images 222 and 224. Figures 6 to 8 Examples of various alignment techniques are shown below.

[0062] Aligned versions of deblurred images 222 and 224 (modified deblurred image 504 and deblurred image 224 in this example) can be provided to merging function 506. Merging function 506 is operable to combine image data contained in the aligned versions of deblurred images 222 and 224. Merging function 506 can combine image data from the aligned versions of deblurred images 222 and 224 to generate image 228 based on deblurred images 222 and 224. Merging function 506 can use any suitable technique to merge image data from multiple deblurred images. As described above, for example, averaging, weighted averaging, and / or minimization techniques can be used to combine image data.

[0063] Averaging the image content in the aligned versions of the deblurred images 222 and 224 may involve determining an average value for each pixel location within image 228 based on the pixel values ​​at the same pixel location in the aligned versions of the deblurred images 222 and 224. In an embodiment, the output of the merging function 506 may be defined as follows for averaging.

[0064]

[0065] Here, Output represents the pixel values ​​in image 228, and L and R represent the pixel values ​​in the aligned versions of the deblurred images 222 and 224, respectively. For example, when a higher signal-to-noise ratio is desired, the average of the image content in the aligned versions of the deblurred images 222 and 224 can be used.

[0066] Weighting the image content in aligned versions of deblurred images 222 and 224 may involve determining a weighted average for each pixel location in image 228 based on the pixel values ​​at the same pixel location in the aligned versions of deblurred images 222 and 224. Different pixel values ​​in different deblurred images may be weighted differently. For example, pixel values ​​in different aligned versions of deblurred images 222 and 224 may be pre-multiplied by weights before averaging. Weights may be selected to reflect the amount of information from each deblurred image 222 and 224, such that the resulting image 228 may have less distortion compared to the individual deblurred images 222 and 224. As an example, in the two deblurred images 222 and 224, if one deblurred image contains more artifacts than the other, a higher weight may be applied to the image with less distortion. In embodiments, local contrast may be used as a weighting function. Deblurred images 222 or 224 with higher contrast in a specified region may have their pixel values ​​in that region more heavily weighted compared to pixel values ​​in deblurred images 224 or 222 with lower contrast. Local contrast can be determined in any suitable manner, such as by using a Laplace filter. In an embodiment, the output of the merging function 506 may be defined as follows for weighted averaging.

[0067]

[0068] Here, Output represents the pixel values ​​in image 228, and L and R represent the pixel values ​​in the aligned versions of the deblurred images 222 and 224, respectively. and These represent the weights of the pixel values ​​applied to the aligned versions of the deblurred images 222 and 224, respectively. For example, and It can be defined as making In this embodiment, a weighted average can be used when lower distortion is desired.

[0069] Minimizing the image content in the aligned versions of the deblurred images 222 and 224 may involve selecting a value for each pixel location in image 228 based on the minimum pixel value at the same pixel location in the aligned versions of the deblurred images 222 and 224. In an embodiment, the output of the merging function 506 may be defined as follows for minimization.

[0070]

[0071] Here, Output represents the pixel value in image 228, L and R represent the pixel values ​​in the aligned versions of deblurred images 222 and 224, respectively, and min() represents a function that selects the smaller of L and R. The output of the defined merging function 506 is designed to minimize the image intensity of light source diffraction artifacts, thus minimizing the pixel value at each location in the aligned versions of deblurred images 222 and 224 effectively removes diffraction artifacts. In this embodiment, minimization of the image content in the aligned versions of deblurred images 222 and 224 can be used when it is desired to remove diffraction artifacts.

[0072] In embodiments, multiple merging options may be provided in architecture 200, and merging function 506 may apply different merging options as needed. For example, merging function 506 may select one of the merging options based on at least one image quality criterion for merging deblurred images 222 and 224, or one or more portions of deblurred images 222 and 224. In embodiments, merging function 506 may adaptively select different merging options from the multiple merging options on an ad-hoc basis, such as based on image content. In embodiments, merging function 506 may execute various merging options using software or firmware code, thus incurring minimal hardware and memory overhead when multiple merging options are available. However, architecture 200 may support a single merging option, support one or more other or additional merging options, or support any other suitable mechanism for switching between merging options.

[0073] although Figure 5 It shows Figure 2 An example of merge operation 226 in architecture 200, but applicable to... Figure 5 Make various changes. For example, Figure 5 The various components and functions within can be combined, further subdivided, copied, or rearranged as needed. Furthermore, one or more additional components and functions may be included if required or desired. Additionally, while two deblurred images 222 and 224 are shown here, architecture 200 can generate more than two deblurred images, and merging operation 226 can combine more than two deblurred images.

[0074] Figures 6 to 8 The following is shown in accordance with this disclosure: Figure 5 Example alignment function 502 in the merging operation 226. As described above, alignment function 502 is operable to modify at least one of a plurality of deblurred images 222 and 224 to generate aligned versions of deblurred images 222 and 224. The aligned versions of deblurred images 222 and 224 can then be merged and used to generate image 228 of the scene (ideally with reduced or minimized diffraction artifacts as described above).

[0075] like Figure 6 As shown, an embodiment of alignment function 502 can perform global registration of deblurred images 222 and 224 to generate aligned versions of deblurred images 222 and 224. Here, deblurred images 222 and 224 can be provided to feature extraction function 602. Feature extraction function 602 is operable to detect and describe salient features within each of deblurred images 222 and 224. The identified features of deblurred images 222 and 224 can be associated with, for example, specific objects captured in deblurred images 222 and 224, such as features associated with people, vehicles, buildings, natural landmarks, or other content in deblurred images 222 and 224. Feature extraction function 602 can use any suitable technique to perform feature recognition and extraction using multiple images. In an embodiment, for example, feature extraction function 602 may use one or more methods described in “ORB: An Efficient Alternative to SIFT or SURF” published by Rublee et al. at the 2011 International Conference on Computer Vision in November 2011 (the entire contents of which are incorporated herein by reference), to extract features from deblurred images 222 and 224, but other implementations of feature extraction function 602 may also be used.

[0076] The extracted features can be provided to feature matching function 604. Feature matching function 604 is operable to match recognized features from different deblurred images 222 and 224 such that common features captured in the multiple deblurred images 222 and 224 can be identified as the same feature. For example, feature matching function 604 can determine that one or more recognized features associated with the same object in the different deblurred images 222 and 224 represent the same feature, such as when a particular feature corresponds to the same feature of people, vehicles, buildings, natural landmarks, or other content in deblurred images 222 and 224. Feature matching function 604 can use any suitable technique to perform feature matching. In embodiments, feature matching function 604 can be implemented using a "fast approximate nearest neighbor library" or a "FLANN" feature matcher to match features extracted from deblurred images 222 and 224, but other implementations of feature matching function 604 can be used.

[0077] Matching features can be provided to homography estimation function 606. Homography estimation function 606 is operable to estimate how a deblurred image 222 or 224 should be warped or otherwise transformed such that features in the deblurred image are located in the same or substantially the same positions as the same features in another deblurred image 224 or 222. For example, homography estimation function 606 can generate a viewpoint transformation matrix that defines how the viewpoint of a scene captured in one deblurred image 222 or 224 needs to be transformed so as to at least substantially match the viewpoint of the same scene captured in another deblurred image 224 or 222. Homography estimation function 606 can use any suitable technique to generate the viewpoint transformation matrix or other transformations based on the matching features. In embodiments, homography estimation function 606 can use the "Random Sample Consensus" or "RANSAC" algorithm to generate the viewpoint transformation matrix based on the matching features, but other implementations of homography estimation function 606 are also possible.

[0078] The deblurred image 222 and a view transformation matrix or other transformation can be provided to the global warp function 608. The global warp function 608 is operable to warp the deblurred image 222 based on the transformation to generate a modified deblurred image 504. For example, the global warp function 608 can warp the deblurred image 222 based on the transformation such that features in the modified deblurred image 504 are located in the same or substantially the same positions as the same features in the deblurred image 224. For example, the global warp function 608 can warp the deblurred image 222 based on the view transformation matrix such that the viewpoint of the scene defined by the modified deblurred image 504 matches or substantially matches the viewpoint of the scene defined by the deblurred image 224. The global warp function 608 can use any suitable technique to warp at least one image. In an embodiment, the global warp function 608 can perform bicubic interpolation to interpolate each pixel value based on the surrounding pixel values ​​in a 4×4 or other grid, which can result in a warped image that can be highly accurate and exhibit fewer artifacts (i.e., the modified deblurred image 504 in this example). However, the global deformation function 608 can use any other suitable deformation technique.

[0079] like Figure 7As shown, an embodiment of alignment function 502 can perform local alignment of deblurred images 222 and 224 to generate aligned versions of deblurred images 222 and 224. Deblurred images 222 and 224 can be provided to block motion estimation function 702. Block motion estimation function 702 is operable to divide each of deblurred images 222 and 224 into non-overlapping blocks. For example, the blocks of deblurred images 222 and 224 may have a fixed size. Block motion estimation function 702 can identify the most similar block between deblurred images 222 and 224, such as by identifying the block in deblurred image 222 that is most similar to a block in deblurred image 224. In an embodiment, a grid search can be used to identify the most similar block, where a suitable distance metric can be used to evaluate blocks in deblurred images 222 and 224 that are at different displacements. The two blocks with the smallest distance (one in deblurred image 222 and one in deblurred image 224) can be selected as the most similar blocks. Once the most similar block is identified, the local deformation function 704 can be executed to locally deform each block based on the identified displacement (minimum distance).

[0080] like Figure 8 As shown, an embodiment of alignment function 502 can perform global registration and local alignment of deblurred images 222 and 224 to generate aligned versions of deblurred images 222 and 224. Feature extraction function 602, feature matching function 604, homography estimation function 606, and global warp function 608 can be used to process deblurred images 222 and 224. Alignment function 502, which performs global registration and local alignment of deblurred images 222 and 224, can result in generating globally aligned versions of deblurred images 222 and 224 (e.g., a globally warped version of deblurred image 222 and deblurred image 224). The globally aligned versions of deblurred images 222 and 224 can be provided to non-rigid (local) warp function 802. Non-rigid (local) warp function 802 can provide local warps of one or more globally aligned versions of deblurred images 222 and 224. For example, the result of global warp function 608 can be used to initialize non-rigid (local) warp function 802 to accelerate the processing of deblurred images 222 and 224. This type of approach can be useful when lens distortion and three-dimensional (3D) scenes make global registration itself difficult to produce good results. Non-rigid registration provided by non-rigid (local) warp function 802 can address the problem that lens distortion and three-dimensional (3D) scenes make global registration difficult to produce good results in higher-dimensional parameter spaces. In the embodiment, Figure 8 The various operations shown can be implemented as described by Klein et al. in “elastix: A toolbox for intensity-based medical image registration” (published in IEEE Transactions on Medical Imaging, January 2010, Vol. 29, No. 1) (the entire contents of which are incorporated herein by reference).

[0081] although Figures 6 to 8 It shows Figure 5 An example of alignment function 502 in merge operation 226, but it is possible to... Figures 6 to 8 Make various changes. For example, Figures 6 to 8 The various components and functions within each of these components can be combined, further subdivided, replicated, or rearranged as needed. Furthermore, one or more additional components and functions can be included if required or desired. Figures 6 to 8 Each of the elements. Furthermore, the deblurred image can be aligned in any other suitable way, and does not require the use of... Figures 6 to 8 Any of the specific techniques shown in the diagram.

[0082] Figure 9 An example method 900 for reducing optical diffraction effects in a UDC system according to this disclosure is shown. For ease of explanation, Figure 9 The method 900 shown is described as being by Figure 1 The electronic device 101 in the network configuration 100 executes, wherein the electronic device 101 can realize Figure 2 The architecture 200 is shown in the diagram. However, Figure 9 The method 900 shown can be executed by any other suitable device and in any other suitable system, such as when the method 900 is executed using the server 106.

[0083] like Figure 9As shown, in operation 902, one or more first image frames of the scene can be obtained using a first diffraction pattern. In operation 904, the diffraction pattern can be optionally rotated or otherwise modified (or allowed to be rotated or otherwise modified). In operation 906, one or more second image frames of the scene can be obtained using a second diffraction pattern. For example, the processor 120 of electronic device 101 can use one or more cameras or other imaging sensors (such as one or more under-display cameras) and the first diffraction pattern 206 to obtain one or more image frames 202. For example, the processor 120 of electronic device 101 can use one or more cameras or other imaging sensors (such as one or more under-display cameras) and the second diffraction pattern 208 to obtain one or more image frames 204. The second diffraction pattern 208 may be different from the first diffraction pattern 206, but is associated with the first diffraction pattern 206 via a transformation (such as rotation). As described above, the same imaging sensor 180 may or may not be used to obtain image frames 202, 204, and one or more actions may or may not be required to generate different diffraction patterns 206, 208. For example, one or more image frames 202 may be captured by one or more imaging sensors using a first diffraction pattern 206, and a second diffraction pattern 208 may be created before capturing one or more image frames 204 using the same imaging sensor (e.g., via rotation of the imaging sensor, rotation of at least a portion of the display 160 above the imaging sensor, and / or rotation of the electronics 101). For example, image frames 202, 204 may be captured by different imaging sensors using different diffraction patterns 206, 208, such as when the display 160 above the imaging sensor contains different diffraction patterns 206, 208 in different areas of the display 160.

[0084] In operation 908, a first deblurred image and a second deblurred image can be generated using the first image frame and the second image frame. For example, the processor 120 of the electronic device 101 can perform multi-frame processing operations 210, 212 or other image processing operations to process image frames 202, 204 and generate processed images 214, 216. For example, the processor 120 of the electronic device 101 can perform PSFI operations 218, 220 to process processed images 214, 216 and generate deblurred images 222, 224. Each of the deblurred images 222, 224 can represent a less blurred version of the associated processed image 214, 216.

[0085] In operation 910, the first and second deblurred images can be aligned and merged to generate an image of the scene. For example, the processor 120 of the electronic device 101 can perform a merging operation 226 to combine the deblurred images 222, 224 while utilizing complementary types of image artifacts generated by the first diffraction pattern 206 and the second diffraction pattern 208. For example, alignment can be performed to generate aligned versions of the deblurred images 222, 224. In embodiments, alignment can include a global registration operation and / or a local alignment operation, whereby a global registration operation can globally align the deblurred images 222, 224 (e.g., based on feature extraction and feature matching), and a local alignment operation can locally deform at least a portion of a deblurred image 222, 224 and locally align portions of the deblurred images 222, 224 (e.g., based on non-rigid deformation). Merging can be performed to merge the aligned versions of the deblurred images 222, 224. In embodiments, merging can use at least one merging operation that can be selected based on image quality criteria, and the image quality criteria can vary within different portions of the deblurred images 222, 224. For example, the merging operation may include an averaging operation that can be selected when the image quality criterion specifies obtaining a higher signal-to-noise ratio, a weighted averaging operation that can be selected when the image quality criterion specifies obtaining lower distortion, and / or a minimum operation that can be selected when the image quality criterion specifies removing diffraction artifacts. For example, the weighting operation that can be used in the weighted averaging operation may be based on the local contrast within the deblurred images 222, 224.

[0086] In operation 912, one or more post-processing operations may optionally be performed to generate a final image of the scene. For example, the processor 120 of the electronic device 101 may perform a tone mapping operation 230 to adjust the colors in image 228 and generate an output image 232. In operation 914, the final image of the scene may be stored, output, or used in some manner. For example, the output image 232 may be displayed on the display 160 of the electronic device 101, saved to camera film stored in the memory 130 of the electronic device 101, or attached to a text message, email, or other communication sent from the electronic device 101. Of course, the output image 232 may be used in any other or additional manner.

[0087] although Figure 9 An embodiment of a method 900 for reducing optical diffraction effects in a UDC system is shown, but it is possible to modify... Figure 9 Make various changes. For example, although it is shown as a series of operations, Figure 9 The various operations in the process can overlap, occur in parallel, occur in different orders, or occur any number of times (including zero times).

[0088] It should be noted that this can be implemented in any suitable manner in electronic device 101, server 106, or other devices. Figures 2 to 9Shown or about Figures 2 to 9 The described functions. In embodiments, one or more software applications or other software instructions running on the processor 120 of electronic device 101, server 106, or other device may be used to implement or support the functions described. Figures 2 to 9 Shown or about Figures 2 to 9 At least some of the functions described. In an embodiment, one or more dedicated hardware components may be used to implement or support the functions described. Figures 2 to 9 Shown or about Figures 2 to 9 At least some of the functions described. Typically, any suitable hardware or any suitable combination of hardware and software / firmware instructions can be used to perform these functions. Figures 2 to 9 Shown or about Figures 2 to 9 The described function.

[0089] According to embodiments of this disclosure, a method may include: acquiring one or more first image frames associated with a first diffraction pattern and one or more second image frames associated with a second diffraction pattern using at least one sensor located below at least one display, the first and second diffraction patterns being related by a transformation. According to embodiments of this disclosure, the method may include: generating a first deblurred image using the one or more first image frames, and generating a second deblurred image using the one or more second image frames. According to embodiments of this disclosure, the method may include: combining the first and second deblurred images while utilizing image artifacts of complementary types generated by the first and second diffraction patterns to generate an image of a scene.

[0090] According to embodiments of this disclosure, the operation of combining a first deblurred image and a second deblurred image may include: performing an alignment operation to generate an aligned version of the first deblurred image and the second deblurred image, the alignment operation utilizing image artifacts of complementary types generated by the first diffraction pattern and the second diffraction pattern.

[0091] According to embodiments of this disclosure, the operation of combining a first deblurred image and a second deblurred image may include merging aligned versions of the first deblurred image and the second deblurred image.

[0092] According to embodiments of this disclosure, the operation of performing the alignment operation may include: performing a global registration operation to globally align the first deblurred image and the second deblurred image.

[0093] According to embodiments of this disclosure, the operation of performing the alignment operation may include: performing a local alignment operation to locally deform at least a portion of the first deblurred image, and locally aligning portions of the first deblurred image and portions of the second deblurred image.

[0094] According to embodiments of this disclosure, the method may be characterized in that the global registration operation may include a feature extraction operation and a feature matching operation.

[0095] According to embodiments of this disclosure, the method may be characterized in that: the local alignment operation may include a non-rigid deformation operation.

[0096] According to embodiments of this disclosure, merging aligned versions of a first deblurred image and a second deblurred image may include merging the aligned versions of the first deblurred image and the second deblurred image using at least one merging operation selectable based on image quality criteria. According to embodiments of this disclosure, the image quality criteria may vary within different portions of the first deblurred image and the second deblurred image.

[0097] According to embodiments of this disclosure, the method may be characterized in that at least one merging operation may include an averaging operation selected when an image quality criterion specifies that a higher signal-to-noise ratio is obtained.

[0098] According to embodiments of this disclosure, the method may be characterized in that at least one merging operation may include a weighted averaging operation selected when an image quality criterion specifies that lower distortion is desired. According to embodiments of this disclosure, the method may be characterized in that the weighting operation in the weighted averaging operation may be based on local contrast.

[0099] According to embodiments of this disclosure, the method may be characterized in that at least one merging operation may include the minimum operation selected when an image quality criterion specifies the removal of diffraction artifacts.

[0100] According to embodiments of this disclosure, the method may be characterized in that at least one sensor may include at least one under-display camera.

[0101] According to embodiments of the present disclosure, the method may be characterized in that: a first diffraction pattern may be associated with a first pattern of at least one display, wherein at least one under-display camera is located below the at least one display.

[0102] According to embodiments of this disclosure, the method may be characterized in that: the second diffraction pattern may be associated with a second pattern of at least one display, wherein at least one under-display camera is located below the at least one display.

[0103] According to embodiments of the present disclosure, the method is characterized in that a second diffraction pattern can be obtained by rotating at least one under-display camera. According to embodiments of the present disclosure, the method is characterized in that a second diffraction pattern can be obtained by rotating at least a portion of at least one display relative to at least one under-display camera. According to embodiments of the present disclosure, the method is characterized in that a second diffraction pattern can be obtained by integrating different grid patterns into different portions of at least one display. According to embodiments of the present disclosure, the method is characterized in that a second diffraction pattern can be obtained by rotating an electronic device comprising at least one display and at least one under-display camera.

[0104] According to embodiments of the present disclosure, an electronic device may include: at least one display; at least one sensor located below the at least one display; at least one processor including processing circuitry; and a memory storing instructions. According to embodiments of the present disclosure, the instructions, when executed individually or jointly by at least one processor, may cause the electronic device to: acquire, using at least one sensor, one or more first image frames associated with a first diffraction pattern and one or more second image frames associated with a second diffraction pattern, the first and second diffraction patterns being related by transformation. According to embodiments of the present disclosure, the instructions, when executed individually or jointly by at least one processor, may cause the electronic device to: generate a first deblurred image using one or more first image frames, and generate a second deblurred image using one or more second image frames. According to embodiments of the present disclosure, the instructions, when executed individually or jointly by at least one processor, may cause the electronic device to: combine the first and second deblurred images while utilizing image artifacts of complementary types generated by the first and second diffraction patterns to generate an image of a scene.

[0105] According to embodiments of this disclosure, in order to combine a first deblurred image and a second deblurred image, instructions, when executed individually or jointly by at least one processor, may cause an electronic device to perform an alignment operation to generate aligned versions of the first deblurred image and the second deblurred image, the alignment operation utilizing image artifacts of complementary types generated by the first diffraction pattern and the second diffraction pattern.

[0106] According to embodiments of this disclosure, in order to combine a first deblurred image and a second deblurred image, instructions, when executed individually or jointly by at least one processor, can cause an electronic device to perform the following operation: merging aligned versions of the first deblurred image and the second deblurred image.

[0107] According to embodiments of this disclosure, in order to perform an alignment operation, instructions, when executed individually or jointly by at least one processor, can cause an electronic device to perform a global registration operation to globally align a first deblurred image and a second deblurred image.

[0108] According to embodiments of the present disclosure, in order to perform an alignment operation, instructions, when executed individually or jointly by at least one processor, can cause an electronic device to perform a partial alignment operation to locally deform at least a portion of a first deblurred image and locally align portions of the first deblurred image and portions of the second deblurred image.

[0109] According to embodiments of this disclosure, the electronic device may be characterized in that: the global registration operation may include a feature extraction operation and a feature matching operation.

[0110] According to embodiments of this disclosure, the electronic device may be characterized in that: the local alignment operation may include a non-rigid deformation operation.

[0111] According to embodiments of this disclosure, in order to merge aligned versions of a first deblurred image and a second deblurred image, instructions, when executed individually or jointly by at least one processor, may cause an electronic device to perform the following operation: merging aligned versions of the first deblurred image and the second deblurred image using at least one merging operation selectable based on image quality criteria. According to embodiments of this disclosure, the image quality criteria may be able to vary within different portions of the first deblurred image and the second deblurred image.

[0112] According to embodiments of the present disclosure, the electronic device may be characterized in that at least one merging operation may include an averaging operation selected when an image quality criterion specifies that a higher signal-to-noise ratio is obtained.

[0113] According to embodiments of the present disclosure, the electronic device may be characterized in that at least one merging operation may include a weighted averaging operation selected when an image quality criterion specifies that lower distortion is desired. According to embodiments of the present disclosure, the electronic device may be characterized in that the weighting operation in the weighted averaging operation may be based on local contrast.

[0114] According to embodiments of this disclosure, the electronic device may be characterized in that at least one merging operation may include the minimum operation selected when an image quality criterion specifies the removal of diffraction artifacts.

[0115] According to embodiments of this disclosure, the electronic device may be characterized in that at least one sensor may include at least one under-display camera.

[0116] According to embodiments of the present disclosure, the electronic device may be characterized in that: a first diffraction pattern may be associated with a first pattern of at least one display, wherein at least one under-display camera is located below the at least one display.

[0117] According to embodiments of the present disclosure, the electronic device may be characterized in that: a second diffraction pattern may be associated with a second pattern of at least one display, wherein at least one under-display camera is located below the at least one display.

[0118] According to embodiments of the present disclosure, the electronic device is characterized in that: a second diffraction pattern can be obtained by rotating at least one under-display camera. According to embodiments of the present disclosure, the electronic device is characterized in that: a second diffraction pattern can be obtained by rotating at least a portion of at least one display relative to at least one under-display camera. According to embodiments of the present disclosure, the electronic device is characterized in that: a second diffraction pattern can be obtained by integrating different grid patterns into different portions of at least one display. According to embodiments of the present disclosure, the electronic device is characterized in that: a second diffraction pattern can be obtained by rotating the electronic device.

[0119] Although this disclosure has been described with reference to various exemplary embodiments, various changes and modifications may be suggested to those skilled in the art. This disclosure is intended to cover such changes and modifications that fall within the scope of the appended claims.

Claims

1. A method comprising: One or more first image frames associated with a first diffraction pattern and one or more second image frames associated with a second diffraction pattern are obtained using at least one sensor located below at least one display, the first diffraction pattern and the second diffraction pattern being related by transformation; A first deblurred image is generated using the one or more first image frames, and a second deblurred image is generated using the one or more second image frames; as well as The first deblurred image and the second deblurred image are combined while utilizing complementary types of image artifacts generated by the first diffraction pattern and the second diffraction pattern to generate an image of the scene.

2. The method as described in claim 1, wherein, The operation of combining the first deblurred image and the second deblurred image includes: An alignment operation is performed to generate aligned versions of the first and second deblurred images, the alignment operation utilizing complementary image artifacts generated by the first and second diffraction patterns; and The aligned versions of the first deblurred image and the second deblurred image are merged.

3. The method as described in claim 2, wherein, The operation of performing the alignment operation includes at least one of the following operations: Perform a global registration operation to globally align the first deblurred image and the second deblurred image; as well as Perform a local alignment operation to locally deform at least a portion of the first deblurred image and locally align portions of the first deblurred image and portions of the second deblurred image.

4. The method of claim 3, wherein, Includes at least one of the following features: The global registration operation includes feature extraction and feature matching operations; and The local alignment operation includes non-rigid deformation operations.

5. The method according to any one of claims 2 to 4, wherein, The operation of merging the aligned versions of the first deblurred image and the second deblurred image includes: The aligned versions of the first deblurred image and the second deblurred image are merged using at least one merge operation selected based on image quality criteria that can vary within different portions of the first deblurred image and the second deblurred image.

6. The method of claim 5, wherein, The at least one merge operation includes at least one of the following operations: The averaging operation is selected when the image quality criterion specifies that a higher signal-to-noise ratio is obtained. The image quality criterion specifies a weighted averaging operation selected to achieve lower distortion, wherein the weighting operation in the weighted averaging operation is based on local contrast; and The image quality criterion specifies the minimum operation to be selected when removing diffraction artifacts.

7. The method according to any one of claims 1 to 6, wherein: The at least one sensor includes at least one under-display camera. The first diffraction pattern is associated with a first pattern of the at least one display, wherein the at least one under-display camera is located below the at least one display; The second diffraction pattern is associated with a second pattern of the at least one display, wherein the at least one under-display camera is located below the at least one display; and The second diffraction pattern is obtained by at least one of the following operations: Rotate the at least one under-screen camera; Rotate at least a portion of the at least one display relative to the at least one under-display camera; Integrating different grid patterns into different parts of the at least one display; and The rotating electronic device includes the at least one display and the at least one under-screen camera.

8. An electronic device comprising: At least one display; At least one sensor is located below the at least one display; At least one processor, including processing circuitry; as well as The memory stores instructions that, when executed individually or jointly by the at least one processor, cause the electronic device to perform the following operations: The at least one sensor is used to obtain one or more first image frames associated with a first diffraction pattern and one or more second image frames associated with a second diffraction pattern, the first diffraction pattern and the second diffraction pattern being correlated by transformation; A first deblurred image is generated using the one or more first image frames, and a second deblurred image is generated using the one or more second image frames; as well as The first deblurred image and the second deblurred image are combined while utilizing complementary types of image artifacts generated by the first diffraction pattern and the second diffraction pattern to generate an image of the scene.

9. The electronic device as claimed in claim 8, wherein, In order to combine the first deblurred image and the second deblurred image, the instructions, when executed individually or jointly by the at least one processor, cause the electronic device to perform the following operations: An alignment operation is performed to generate aligned versions of the first deblurred image and the second deblurred image, the alignment operation utilizing complementary types of image artifacts generated by the first diffraction pattern and the second diffraction pattern; as well as The aligned versions of the first deblurred image and the second deblurred image are merged.

10. The electronic device of claim 9, wherein, In order to perform the alignment operation, the instructions, when executed individually or jointly by the at least one processor, cause the electronic device to perform at least one of the following operations: Perform a global registration operation to globally align the first deblurred image and the second deblurred image; as well as Perform a local alignment operation to locally deform at least a portion of the first deblurred image and locally align portions of the first deblurred image and portions of the second deblurred image.

11. The electronic device of claim 10, wherein, Includes at least one of the following features: The global registration operation includes feature extraction and feature matching operations; and The local alignment operation includes non-rigid deformation operations.

12. The electronic device according to any one of claims 9 to 11, wherein, In order to merge aligned versions of the first deblurred image and the second deblurred image, the instructions, when executed individually or jointly by the at least one processor, cause the electronic device to perform the following operation: merge aligned versions of the first deblurred image and the second deblurred image using at least one merging operation selected based on image quality criteria, which can vary within different portions of the first deblurred image and the second deblurred image.

13. The electronic device of claim 12, wherein, The at least one merge operation includes at least one of the following operations: The averaging operation is selected when the image quality criterion specifies that a higher signal-to-noise ratio is obtained. The image quality criterion specifies a weighted averaging operation selected to achieve lower distortion, wherein the weighting operation in the weighted averaging operation is based on local contrast; and The image quality criterion specifies the minimum operation to be selected when removing diffraction artifacts.

14. The electronic device according to any one of claims 8 to 13, wherein: The at least one sensor includes at least one under-display camera. The first diffraction pattern is associated with a first pattern of the at least one display, wherein the at least one under-display camera is located below the at least one display; The second diffraction pattern is associated with a second pattern of the at least one display, wherein the at least one under-display camera is located below the at least one display; and The second diffraction pattern is obtained by at least one of the following operations: Rotate the at least one under-screen camera; Rotate at least a portion of the at least one display relative to the at least one under-display camera; Integrating different grid patterns into different parts of the at least one display; and Rotate the electronic device.

15. A computer-readable medium storing instructions, wherein, When the instructions are executed by at least one processor, the at least one processor causes the processor to perform the method as described in any one of claims 1 to 7.

Images