Light field camera, photographing method and apparatus, storage medium, and program product

Abstract

This disclosure relates to a light field camera, imaging method and apparatus, storage medium and program product. The light field camera includes: a lens assembly; a light field imaging component disposed opposite to the lens assembly, for detecting and counting photons transmitted from the lens assembly to generate a light field image; and a driving component for driving the light field imaging component to move relative to the lens assembly. This light field camera improves image quality and reduces costs.

Description

Technical Field

[0001] This disclosure relates to the field of optical imaging technology, and in particular to a light field camera, a photographing method and apparatus, a storage medium and a program product. Background Technology

[0002] Light field cameras can record not only the intensity and color of light, but also the direction of light. However, traditional light field cameras are limited by sensor resolution, resulting in low-resolution light field images that affect image quality and cannot meet the requirements for high-precision imaging. Summary of the Invention

[0003] This disclosure provides a light field camera, a method and apparatus for taking pictures, a storage medium, and a program product.

[0004] According to a first aspect of the present disclosure, a light field camera is provided, comprising:

[0005] Lens assembly;

[0006] A light field imaging component, disposed opposite to the lens assembly, is used to detect and count photons transmitted from the lens assembly to generate a light field image;

[0007] A driving component for driving the light field imaging component to move relative to the lens component.

[0008] In some embodiments, the light field imaging component includes:

[0009] Photon counting sensor;

[0010] A microlens array is fixed relative to the photon counting sensor and located between the lens assembly and the photon counting sensor.

[0011] In some embodiments, the light field camera includes:

[0012] A control component is electrically connected to both the light field imaging component and the driving component, and is used to send displacement commands to the driving component and image acquisition commands to the light field imaging component, so that the driving component drives the light field imaging component to move and acquire multiple light field images.

[0013] According to a second aspect of the present disclosure, a method for taking pictures is provided, including:

[0014] Upon receiving a detected photo-taking command, determine the current photo-taking scene;

[0015] In response to the fact that the shooting scene belongs to a preset scene, the light field camera is controlled to move and acquire multiple light field images; wherein, the light field camera is the light field camera described in any one of the first aspects;

[0016] Image processing is performed on the multiple light field images to obtain a target image and output it.

[0017] In some embodiments, the step of controlling the light field camera to move and acquire multiple light field images in response to the shooting scene belonging to a preset scene includes:

[0018] In response to the fact that the shooting scene is a dark scene, the light field camera is controlled to move and acquire the multiple light field images; and / or,

[0019] In response to the fact that the shooting scene is a moving scene, the light field camera is controlled to move and acquire the multiple light field images.

[0020] In some embodiments, the method further includes:

[0021] Get the specified focal length;

[0022] The step of image processing based on the multiple light field images to obtain and output a target image includes:

[0023] Super-resolution reconstruction is performed based on the multiple light field images to obtain a super-resolution reconstructed light field image.

[0024] Based on the super-resolution reconstructed light field image and the specified focal length, a refocusing process is performed to obtain the target image at the specified focal length.

[0025] In some embodiments, the super-resolution reconstruction based on the multiple light field images to obtain the super-resolution reconstructed light field image includes:

[0026] Register the multiple light field images;

[0027] Super-resolution reconstruction is performed based on multiple registered light field images to obtain the super-resolution reconstructed light field image.

[0028] In some embodiments, the refocusing process based on the super-resolution reconstructed light field image and the specified focal length to obtain the target image at the specified focal length includes:

[0029] The target image is obtained by inputting the super-resolution reconstructed light field image and the specified focal length into a preset refocusing model; wherein the refocusing model is obtained by training a deep learning network.

[0030] According to a third aspect of the present disclosure, a photographing device is provided, comprising:

[0031] The module is configured to determine the current shooting scene in response to a detected photo-taking command.

[0032] The acquisition module is configured to control the light field camera to move and acquire multiple light field images in response to the shooting scene belonging to a preset scene; wherein the light field camera is the light field camera described in any one of the first aspects;

[0033] The processing module is configured to perform image processing based on the multiple light field images to obtain a target image and output it.

[0034] In some embodiments, the acquisition module is further configured to control the light field camera to move and acquire the plurality of light field images in response to the shooting scene being a dark scene; and / or, to control the light field camera to move and acquire the plurality of light field images in response to the shooting scene being a moving scene.

[0035] In some embodiments, the apparatus further includes:

[0036] The acquisition module is configured to acquire a specified focal length.

[0037] The processing module is further configured to perform super-resolution reconstruction based on the multiple light field images to obtain a super-resolution reconstructed light field image; and to perform refocusing processing based on the super-resolution reconstructed light field image and the specified focal length to obtain the target image at the specified focal length.

[0038] In some embodiments, the processing module is further configured to register the plurality of light field images; and to perform super-resolution reconstruction based on the registered plurality of light field images to obtain the super-resolution reconstructed light field image.

[0039] In some embodiments, the processing module is further configured to input the super-resolution reconstructed light field image and the specified focal length into a preset refocusing model to obtain the target image; wherein the refocusing model is obtained based on a deep learning network.

[0040] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising:

[0041] processor;

[0042] Memory used to store computer programs or instructions;

[0043] The processor executes the computer program or instructions to implement the steps of the method described in the second aspect above.

[0044] According to a fifth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, the storage medium storing a computer program or instructions that, when executed by a processor, implement the steps of the method described in the second aspect above.

[0045] According to a sixth aspect of the present disclosure, a computer program product is provided, including a computer program or instructions, which, when executed by a processor, implement the steps of the method described in the second aspect above.

[0046] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0047] In this embodiment of the present disclosure, on the one hand, the use of photon counting imaging in the light field camera greatly improves the imaging performance of the light field camera, and relatively increasing the density of the microlens array or using a higher resolution sensor helps to reduce costs; on the other hand, the use of a driving component to drive the light field imaging component to move also helps to realize the diverse functions of the camera, such as synthesizing a high-resolution light field image by moving and shooting multiple light field images, or assisting in focus adjustment, image blurring, three-dimensional image display, etc. during the shooting process.

[0048] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0049] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0050] Figure 1 This is a structural example diagram of a light field camera shown in an embodiment of this disclosure.

[0051] Figure 2 This is an example diagram of light propagation in a light field camera.

[0052] Figure 3 This is a flowchart of a photographing method provided in an embodiment of this disclosure.

[0053] Figure 4 This is a diagram of a photographing device shown in an embodiment of this disclosure.

[0054] Figure 5 This is a structural block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0055] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0056] Light field cameras typically consist of a lens assembly (primary lens), a microlens array, and a sensor. The sensor is usually a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) sensor. Currently, common methods to improve the resolution of light field cameras include increasing the density of the microlens array or using a higher resolution sensor; however, these methods all face challenges such as complex manufacturing processes, high costs, or reduced signal-to-noise ratio. Therefore, how to improve resolution and reduce cost while maintaining light field imaging capabilities has become a key challenge for the commercialization of light field cameras.

[0057] In response, this disclosure provides a light field camera. Figure 1 This is a structural example diagram of a light field camera shown in an embodiment of this disclosure, consisting of... Figure 1 It can be seen that the light field camera 100 includes:

[0058] Lens assembly 101;

[0059] The light field imaging component 102 is disposed opposite to the lens component 101 and is used to detect and count the photons transmitted from the lens component to generate a light field image.

[0060] The driving component 103 is used to drive the light field imaging component 102 to move relative to the lens component 101.

[0061] In this embodiment, the lens assembly 101, typically composed of multiple lens groups, is responsible for collecting light from the scene. The light collected by the lens assembly 101 is transmitted to the light field imaging assembly 102 to generate a light field image. Photons transmitted to the light field imaging assembly 102 are converted into electrons based on the photoelectric effect. The resulting electron signal is then amplified to a detectable level. The light field imaging assembly 102 simultaneously counts the detected photon events and converts the photon count data into an image, thereby generating a light field image. Each photon event records the arrival time, position, and intensity information of a single photon.

[0062] In this embodiment, the driving component 103 can be directly connected to the light field imaging component 102, thereby driving the light field imaging component 102 to move relative to the lens assembly 101, or the driving component 103 can indirectly drive the light field imaging component 102 to move. The driving component 103 can be a nanostage, an electrostrictive actuator, or an electromagnetic actuator, etc., and this embodiment is not limited thereto. Taking the driving component 103 as an electromagnetic actuator as an example, the electromagnetic actuator can be a motor, such as a voice coil motor or a piezoelectric motor.

[0063] In this embodiment, the driving component 103 can drive the light field imaging component 102 to shift within a small range (e.g., 1 to 8 pixels), including driving the light field imaging component 102 to translate vertically or horizontally relative to the lens component 101, or to translate forward or backward (i.e., changing the distance between them). This allows for the acquisition of multiple light field images to synthesize a high-resolution image by moving the light field imaging component 102 relative to the lens component 101 in any direction, and also enables fine-tuning of the focus point by adjusting the relative distance between the light field imaging component 102 and the lens component 101.

[0064] It should be noted that in this embodiment, the driving component 103 can drive not only the light field imaging component 102 to move, but also the lens component 101 to move, thereby achieving lens focusing and zooming. Taking a motor as an example, a ball motor or a multi-axis motor can simultaneously drive both the lens component 101 and the light field imaging component 102. The driving component 103 can also drive the lens component 101 to move by changing the relative distance between the lens component 101 and the light field imaging component 102, i.e., moving it back and forth to achieve lens focusing and zooming. It is understood that by having the lens component 101 and the light field imaging component 102 share the driving component 103, the size of the light field camera 100 can be reduced.

[0065] In this embodiment of the disclosure, the light field camera 100 can be a single photographing device with a photographing function, or it can be a functional module in a terminal device with multiple functions. For example, the light field camera 100 can be a functional module in a terminal device such as a smartphone or a tablet computer. This embodiment of the disclosure does not limit this.

[0066] In this embodiment, the light field imaging component 102 can detect and count photons, and compared to conventional CMOS or CCD image sensors, it can efficiently capture photon signals in a shorter exposure time (down to nanoseconds). Furthermore, it has lower dark current and readout noise, thus providing a relatively higher resolution light field image. It is understood that, on the one hand, this embodiment, by employing photon counting imaging in the light field camera, greatly improves the camera's imaging performance, and by increasing the density of the microlens array or using a higher resolution sensor, it helps reduce costs. On the other hand, this embodiment, by driving the light field imaging component to move, also helps to realize diverse camera functions, such as synthesizing a high-resolution light field image by capturing multiple light field images during movement, or assisting in focus adjustment, image blurring, or 3D image display during the shooting process.

[0067] In some embodiments, the light field imaging component 102 includes:

[0068] Photon counting sensor 102a;

[0069] The microlens array 102b is fixed relative to the photon counting sensor and is located between the lens assembly 101 and the photon counting sensor 102a.

[0070] In this embodiment, the photon counting sensor 102a can be a sensor based on a single-photon avalanche diode (SPAD), a photomultiplier tube (PMT), or a microchannel plate (MCP), and this embodiment is not limited thereto. The microlens array 102b can be composed of multiple microlenses arranged between the lens assembly 101 and the photon counting sensor 102a, with each microlens corresponding to a small area on the photon counting sensor 102a. The function of the microlenses is to separate light rays transmitted from the lens assembly 101 in different directions and project them onto different pixels of the photon counting sensor 102a.

[0071] like Figure 2 This is an example diagram of light propagation from a light field camera, such as... Figure 2 As shown, for a point P on the subject being photographed, light rays are transmitted through the lens assembly 101 to the microlens array 102b and then to the photon counting sensor 102a. Each microlens corresponds to a region on the sensor, capturing light from the scene from different angles. Each microlens generates a small image (called a sub-image, which records the scene information seen from the perspective of that microlens). All the sub-images of the microlenses are combined to form a complete light field image. Figure 2 The image shows the transmission and imaging of light based on a microlens.

[0072] In this embodiment, the photon counting sensor 102a and the microlens array 102b can be fixed by mechanical fixing or by welding, bonding or other methods. By fixing the photon counting sensor 102a and the microlens array 102b, it is convenient for the driving component 103 to drive the light field imaging component 102 to move as a whole, thereby helping to improve the accuracy of driving control.

[0073] In some embodiments, the light field camera 100 includes:

[0074] The control component 104 is electrically connected to the light field imaging component 102 and the driving component 103, respectively, and is used to send displacement commands to the driving component 103 and image acquisition commands to the light field imaging component 102, so that the driving component 103 drives the light field imaging component 102 to move and acquire multiple light field images.

[0075] In this embodiment of the disclosure, the light field camera 100 further includes a control component 104. The control component 104 may be a central processing unit (CPU) or a microcontroller unit (MCU). The control component 104 may be a processor for independently controlling the light field camera 100, or it may be a processor in a multi-functional terminal device that can be used to control the light field camera 100 and other functional components. This embodiment of the disclosure does not limit this.

[0076] In this embodiment of the disclosure, the control component 104 simultaneously controls the movement of the drive component 103 and the image acquisition of the drive component 103 to generate multiple light field images, which helps to reduce the number of control components in the electronic device, thereby reducing size and cost, and achieving a high degree of integration.

[0077] Figure 3 This is a flowchart of a photographing method provided in an embodiment of this disclosure, such as... Figure 3 As shown, it includes the following steps:

[0078] S301, In response to the detection of a photo-taking command, determine the current photo-taking scene;

[0079] S302. In response to the fact that the shooting scene belongs to a preset scene, control the light field camera to move and acquire multiple light field images; wherein, the light field camera is any of the light field cameras described above;

[0080] S303. Perform image processing based on the multiple light field images to obtain a target image and output it.

[0081] In this embodiment of the disclosure, the photographing method is applied to electronic devices including the aforementioned light field camera, such as mobile phones, tablets, personal digital assistants (PDAs), wearable devices, and other devices with photographing functions. In addition, the electronic device may be a camera device with only photographing functions.

[0082] In step S301, the electronic device can detect a shooting command based on a voice command or a touch command of the shutter button, and determine the current shooting scene. The shooting scene can indicate the shooting environment, or it can indicate the state of the subject being photographed or the electronic device itself. In this embodiment, the electronic device can determine the shooting scene using built-in sensors, or it can determine the shooting scene based on user operation or other information.

[0083] In step S302, the electronic device controls the light field camera to move and acquire multiple light field images only if it determines that the shooting scene belongs to a preset scene. It should be noted that in this embodiment, the light field camera undergoes minute displacement, as described above, it can make minute offsets in multiple directions. In this embodiment, if the shooting scene does not belong to the preset scene, the electronic device can control the light field camera to capture only one light field image; or, if the shooting scene does not belong to the preset scene and the electronic device also includes a camera other than the light field camera, it can control the camera other than the light field camera to generate a two-dimensional image. For example, the preset scene can be a scene indicating a poor shooting environment and / or a scene indicating the presence of motion. A poor shooting environment can refer to conditions such as rain or insufficient lighting, while a scene indicating motion can be, for example, a shaking mobile phone or movement of the subject being photographed.

[0084] In step S303, the electronic device performs image processing based on the acquired multiple light field images to obtain and output a target image. In some embodiments, the electronic device may refocus based on multiple light field images to generate and output a clear two-dimensional target image; in some embodiments, the electronic device may obtain depth information based on multiple light field images and further blur the background to obtain the target image; in other embodiments, the electronic device may also generate a target image with both foreground and background clear based on multiple light field images.

[0085] It is understood that, since the light field camera of this disclosure can provide relatively higher resolution light field images, multiple light field images can be captured in certain preset scenarios. Furthermore, image processing of these multiple light field images allows for the acquisition of a single target image with even higher resolution, improving the imaging quality in the preset scenarios. In addition, the exposure time of a photon-counting-based light field camera is extremely short. Therefore, compared to traditional CMOS or CCD image sensors, the light field camera of this disclosure requires less time to acquire the same number of images, thus reducing the time required to acquire multiple light field images, contributing to faster image capture and improving the user experience.

[0086] In some embodiments, the step of controlling the light field camera to move and acquire multiple light field images in response to the shooting scene belonging to a preset scene includes:

[0087] In response to the fact that the shooting scene is a dark scene, the light field camera is controlled to move and acquire the multiple light field images; and / or,

[0088] In response to the fact that the shooting scene is a moving scene, the light field camera is controlled to move and acquire the multiple light field images.

[0089] In this embodiment, a dark scene refers to a scene where the ambient light is very limited or almost nonexistent, and the image quality is generally poor in dark scenes. In contrast, a bright scene is characterized by moderate or high ambient light intensity, resulting in better image quality. In this embodiment, a dark scene can be a nighttime shooting scene, a daytime shooting scene such as an indoor scene with poor lighting, or any other scene with poor lighting.

[0090] In some embodiments, the electronic device can determine the current shooting scene based on a preview image. For example, if the ratio of the number of pixels with brightness values ​​greater than a preset brightness threshold to the total number of pixels in the preview image is less than a preset ratio threshold, then the current shooting scene is determined to be a dark scene. In other embodiments, the electronic device can determine the current shooting scene based on user instructions. For example, the electronic device may receive a user's instruction to select a night shooting mode, or determine the scene as dark based on a user's voice instruction indicating poor lighting conditions. Furthermore, the electronic device can also determine a dark scene based on the spatiotemporal information of the current shooting time. For example, if the current shooting time is between 8 PM and 4 AM in Beijing, then it can be determined to be a dark scene.

[0091] In the embodiments of this disclosure, the motion scene includes the movement of the electronic device itself or the movement of the object being photographed. In some embodiments, the electronic device can detect whether shaking occurs based on a built-in motion sensor such as an accelerometer or a gyroscope. In some embodiments, after detecting a photo-taking command, the electronic device can acquire multiple preview images and analyze whether there is a moving object based on the multiple preview images. In other embodiments, the electronic device can also determine that it belongs to a motion scene based on the user's settings, such as when the user sets the photo to be taken in "motion mode".

[0092] In this embodiment, the aforementioned light field camera is applied to photography in dark scenes and / or moving scenes. Since the exposure time of a photon-counting-based light field camera is extremely short, motion blur is reduced in moving scenes, thereby improving the quality of the captured single light field image. Furthermore, photon-counting-based light field cameras also have low noise properties, thus improving the imaging quality in dark scenes. In addition, since the quality of a single image captured in dark scenes and / or moving scenes is typically limited, image fusion of multiple images can yield a better quality image. Therefore, the light field camera based on this embodiment helps improve the speed of photography in dark scenes and / or moving scenes, enhancing the user experience.

[0093] In some embodiments, the method further includes:

[0094] Get the specified focal length;

[0095] The step of image processing based on the multiple light field images to obtain and output a target image includes:

[0096] Super-resolution reconstruction is performed based on the multiple light field images to obtain a super-resolution reconstructed light field image.

[0097] Based on the super-resolution reconstructed light field image and the specified focal length, a refocusing process is performed to obtain the target image at the specified focal length.

[0098] In this embodiment of the disclosure, the specified focal length can be any focal length set by the user, or it can be a focal length determined based on the focal length of multiple light field images. For example, if multiple light field images are acquired at the same focal length, then the specified focal length is the focal length of the acquired light field images. If there are slight differences in the focal lengths of the acquired light field images, then the specified focal length can be the average of the multiple focal lengths of the acquired light field images.

[0099] In this embodiment of the disclosure, the electronic device performs super-resolution reconstruction based on multiple light field images. Essentially, it utilizes the sub-pixel displacement information between multiple low-resolution images to synthesize a high-resolution image, i.e., the super-resolution reconstructed light field image. In performing super-resolution reconstruction, this embodiment of the disclosure can employ interpolation methods, frequency domain methods, or deep learning methods to generate the high-resolution light field image. The interpolation method can be bilinear interpolation, bicubic interpolation, etc., and the deep learning method can be based on a Deep Super-Resolution (DSR) model or a Transformer-based super-resolution model; this embodiment of the disclosure does not impose any limitations on these methods.

[0100] In this embodiment of the disclosure, the electronic device performs refocusing processing based on the super-resolution reconstructed light field image and a specified focal length to obtain a clear target image at the specified focal length. During the refocusing process, the specified focal length provides the depth of the target focal plane, and focusing can be performed using methods such as spatial domain light field integration and frequency domain slicing. In some embodiments, the refocusing processing based on the super-resolution reconstructed light field image and the specified focal length to obtain the target image at the specified focal length includes:

[0101] The target image is obtained by inputting the super-resolution reconstructed light field image and the specified focal length into a preset refocusing model; wherein the refocusing model is obtained by training a deep learning network.

[0102] In this embodiment, refocusing can also be performed using deep learning methods. A deep learning model learns the multi-view information and light propagation characteristics of the light field image and is trained to generate a refocusing model. Using the refocusing model, after inputting the super-resolution reconstructed light field image and a specified focal length, the model can estimate the depth information of the scene, thereby determining the imaging effect of the target focal plane, i.e., obtaining the target image. In this embodiment, a refocusing model can be obtained by constructing a light field image dataset containing different scenes and focal planes, training and testing a deep learning network. The deep learning network can be a convolutional neural network (CNN), a generative adversarial network (GAN), or a Transformer, etc.

[0103] It is understood that, in this embodiment of the disclosure, refocusing after super-resolution reconstruction enables higher quality imaging in preset scenarios such as dark scenes and / or moving scenes, thereby improving the user's photography experience. Furthermore, refocusing based on a trained model, which is generated from numerous samples, also helps improve the quality of the refocused target image.

[0104] In some embodiments, the super-resolution reconstruction based on the multiple light field images to obtain the super-resolution reconstructed light field image includes:

[0105] Register the multiple light field images;

[0106] Super-resolution reconstruction is performed based on multiple registered light field images to obtain the super-resolution reconstructed light field image.

[0107] In this embodiment, since multiple light field images are acquired under conditions of minute displacement, super-resolution reconstruction also relies on pixel displacement information between multiple light field images to recover high-resolution details. If the images are not properly aligned, the reconstruction result may exhibit artifacts, blurring, or distortion. Therefore, registration before reconstruction can improve the quality of the reconstructed light field image. In this embodiment, registration can be performed based on methods such as optical flow or phase correlation, and this embodiment does not limit the scope of the application.

[0108] Figure 4 This is a diagram of a photographing device shown in an embodiment of this disclosure. Figure 4 It can be seen that this includes:

[0109] The module 401 is configured to determine the current shooting scene in response to the detection of a shooting command;

[0110] The acquisition module 402 is configured to control the light field camera to move and acquire multiple light field images in response to the shooting scene belonging to a preset scene; wherein the light field camera is any of the light field cameras described above;

[0111] The processing module 403 is configured to perform image processing based on the multiple light field images to obtain a target image and output it.

[0112] In some embodiments, the acquisition module 402 is further configured to control the light field camera to move and acquire the plurality of light field images in response to the shooting scene being a dark scene; and / or, in response to the shooting scene being a moving scene, control the light field camera to move and acquire the plurality of light field images.

[0113] In some embodiments, the apparatus further includes:

[0114] The acquisition module is configured to acquire a specified focal length.

[0115] The processing module 403 is further configured to perform super-resolution reconstruction based on the multiple light field images to obtain a super-resolution reconstructed light field image; and to perform refocusing processing based on the super-resolution reconstructed light field image and the specified focal length to obtain the target image at the specified focal length.

[0116] In some embodiments, the processing module 403 is further configured to register the plurality of light field images; and to perform super-resolution reconstruction based on the registered plurality of light field images to obtain the super-resolution reconstructed light field image.

[0117] In some embodiments, the processing module 403 is further configured to input the super-resolution reconstructed light field image and the specified focal length into a preset refocusing model to obtain the target image; wherein the refocusing model is obtained based on deep learning network training.

[0118] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0119] Figure 5 This is a structural block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device may be a light field camera with only a photo-taking function, or it may be a multifunctional device with a photo-taking function, such as a mobile phone or tablet computer, which can be implemented based on the aforementioned light field camera. The electronic device may be device 500.

[0120] Reference Figure 5 The device 500 may include one or more of the following components: processing component 502, memory 504, power supply component 506, multimedia component 508, audio component 510, input / output (I / O) interface 512, sensor component 514, and communication component 516.

[0121] Processing component 502 typically controls the overall operation of device 500, such as operations associated with at least one of display, telephone call, data communication, camera operation, and recording operation. Processing component 502 may include one or more processors 520 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 502 may include one or more modules to facilitate interaction between processing component 502 and other components. For example, processing component 502 may include a multimedia module to facilitate interaction between multimedia component 508 and processing component 502.

[0122] Memory 504 is configured to store various types of data to support operation on device 500. Examples of such data include at least one of the following: instructions for any application or method operating on device 500, contact data, phonebook data, messages, pictures, and videos. Memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0123] Power supply component 506 provides power to various components of device 500. Power supply component 506 may include at least one of the following: a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 500.

[0124] Multimedia component 508 includes a screen that provides an output interface between device 500 and the user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen may be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 508 includes a front-facing camera and / or a rear-facing camera. When device 500 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0125] Audio component 510 is configured to output and / or input audio signals. For example, audio component 510 includes a microphone (MIC) configured to receive external audio signals when device 500 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 504 or transmitted via communication component 516. In some embodiments, audio component 510 also includes a speaker for outputting audio signals.

[0126] I / O interface 512 provides an interface between processing component 502 and peripheral interface modules, such as keyboards, click wheels, and buttons. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0127] Sensor assembly 514 includes one or more sensors for providing state assessments of various aspects of device 500. For example, sensor assembly 514 may detect the on / off state of device 500, the relative positioning of components such as the display and keypad of device 500, changes in the position of device 500 or one of its components, the presence or absence of user contact with device 500, orientation or acceleration / deceleration of device 500, and temperature changes of device 500. Sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 514 may also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, sensor assembly 514 may also include, but is not limited to, at least one of the following: an accelerometer, a gyroscope, a magnetometer, a pressure sensor, and a temperature sensor.

[0128] Communication component 516 is configured to facilitate wired or wireless communication between device 500 and other devices. Device 500 can access wireless networks based on communication standards, such as Wi-Fi, 4G, 5G, or combinations thereof. In one exemplary embodiment, communication component 516 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 516 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wide Band (UWB), Bluetooth (BT), and other technologies.

[0129] In an exemplary embodiment, device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components.

[0130] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including executable instructions or a computer program, which can be executed by the processor 520 of the device 500 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0131] A non-transitory computer-readable storage medium, wherein instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform any of the photographing methods described above in the embodiments of this disclosure.

[0132] This disclosure provides a computer program product comprising a computer program or executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer program or executable instructions from the computer-readable storage medium and executes the computer program or executable instructions, causing the electronic device to perform any of the above-described photographing methods of this disclosure.

[0133] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the foregoing claims.

[0134] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A light field camera, characterized in that, The light field camera includes: Lens assembly; A light field imaging component, disposed opposite to the lens assembly, is used to detect and count photons transmitted from the lens assembly to generate a light field image; A driving component for driving the light field imaging component to move relative to the lens component.

2. The light field camera according to claim 1, characterized in that, The light field imaging component includes: Photon counting sensor; A microlens array is fixed relative to the photon counting sensor and located between the lens assembly and the photon counting sensor.

3. The light field camera according to claim 1, characterized in that, The light field camera includes: A control component is electrically connected to both the light field imaging component and the driving component, and is used to send displacement commands to the driving component and image acquisition commands to the light field imaging component, so that the driving component drives the light field imaging component to move and acquire multiple light field images.

4. A method for taking photos, characterized in that, The method includes: Upon receiving a detected photo-taking command, determine the current photo-taking scene; In response to the fact that the shooting scene belongs to a preset scene, the light field camera is controlled to move and acquire multiple light field images; wherein, the light field camera is the light field camera according to any one of claims 1 to 3; Image processing is performed on the multiple light field images to obtain a target image and output it.

5. The method according to claim 4, characterized in that, The response that the shooting scene belongs to a preset scene, controlling the light field camera to move and acquire multiple light field images includes: In response to the fact that the shooting scene is a dark scene, the light field camera is controlled to move and acquire the multiple light field images; and / or, In response to the fact that the shooting scene is a moving scene, the light field camera is controlled to move and acquire the multiple light field images.

6. The method according to claim 4, characterized in that, The method further includes: Get the specified focal length; The step of image processing based on the multiple light field images to obtain and output a target image includes: Super-resolution reconstruction is performed based on the multiple light field images to obtain a super-resolution reconstructed light field image. Based on the super-resolution reconstructed light field image and the specified focal length, a refocusing process is performed to obtain the target image at the specified focal length.

7. The method according to claim 6, characterized in that, The super-resolution reconstruction based on the multiple light field images to obtain the super-resolution reconstructed light field image includes: Register the multiple light field images; Super-resolution reconstruction is performed based on multiple registered light field images to obtain the super-resolution reconstructed light field image.

8. The method according to claim 6, characterized in that, The refocusing process based on the reconstructed light field image and the specified focal length to obtain the target image at the specified focal length includes: The target image is obtained by inputting the super-resolution reconstructed light field image and the specified focal length into a preset refocusing model; wherein the refocusing model is obtained by training a deep learning network.

9. A photographing device, characterized in that, The device includes: The module is configured to determine the current shooting scene in response to a detected photo-taking command. The acquisition module is configured to control the light field camera to move and acquire multiple light field images in response to the shooting scene belonging to a preset scene; wherein the light field camera is the light field camera according to any one of claims 1 to 3; The processing module is configured to perform image processing based on the multiple light field images to obtain a target image and output it.

10. An electronic device, characterized in that, include: processor; Memory used to store computer programs or instructions; The processor executes the computer program or instructions to implement the steps of the method according to any one of claims 4 to 8.

11. A non-transitory computer-readable storage medium storing a computer program or instructions, characterized in that, When the computer program or instructions in the storage medium are executed by a processor, the steps of the method according to any one of claims 4 to 8 are implemented.

12. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 4 to 8.

Images