Method and apparatus for displaying light field photograph

Abstract

The application relates to the terminal field and provides a method and device for displaying a light field photo, the method comprising the following steps: receiving a first operation of a user, the first operation being used for displaying a first light field photo, the first light field photo comprising N persons, N being a positive integer; in response to the first operation, obtaining a facial feature of the user; determining a target person from the N persons according to the facial feature of the user; performing refocusing on the first light field photo with the target person as a refocusing focus point to obtain a first refocusing photo; and displaying the first refocusing photo. The above method can present a visual effect that the user wants to see without manually selecting a focus point by the user, thereby improving the user experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the terminal field, and in particular to a method and device for displaying a light field photo. BACKGROUND

[0002] A light field photo is a photo containing information such as intensity and direction of light, and contains more abundant information than a normal photo. Therefore, a terminal device can change a focus of the light field photo through refocusing, and present different visual effects to a user.

[0003] When a user opens a light field photo on a terminal device, the terminal device displays the light field photo according to a default focus, which can be located at a focus plane where a person A is located, or at a focus plane where a person B is located, or at a focus plane where a building is located. Different default focuses result in different visual effects seen by the user, and how to present the visual effect that the user wants to see to the user is a problem to be solved at present. SUMMARY

[0004] Embodiments of the present application provide a method and device for displaying a light field photo, a computer readable storage medium and a computer program product, which can present the visual effect that a user wants to see when displaying the light field photo, and improve user experience.

[0005] In a first aspect, a method for displaying a light field photo is provided, including: receiving a first operation of a user, the first operation being used to display a first light field photo, the first light field photo including N persons, N being a positive integer; in response to the first operation, obtaining a facial feature of the user; determining a target person from the N persons according to the facial feature of the user; refocusing the first light field photo with the target person as a refocusing focus point to obtain a first refocusing photo; and displaying the first refocusing photo.

[0006] When a user views a light field photo, the user usually wants to see a clear image of himself or a clear image of a person closely related to himself, therefore, a terminal device can obtain a facial feature of a user currently viewing the light field photo, and determine a target person from N persons according to the facial feature of the user, the target person being the user or a person closely related to the user. Subsequently, the terminal device refocuses the light field photo with a position where the target person is located as a refocusing focus point to obtain a refocusing photo, and displays the refocusing photo to the user. Compared with a method for displaying a light field photo based on a default focus, the method for displaying a light field photo provided in the embodiments of the present application can present the visual effect that the user wants to see without manual selection of the focus point by the user, thereby improving user experience.

[0007] In an implementation, determining the target person from the N persons according to the facial feature of the user includes: matching the facial feature of the user with facial features of the N persons; when there is a facial feature matching the facial feature of the user among the facial features of the N persons, determining that the person corresponding to the facial feature matching the facial feature of the user is the target person; when there is no facial feature matching the facial feature of the user among the facial features of the N persons, obtaining a facial feature of a first relationship person of the user; and determining the target person from the N persons according to the facial feature of the first relationship person.

[0008] Generally, when the user views the photo, the user wants to see himself or herself or a person related to himself or herself (for example, a first relationship person), and therefore, when the user is not found in the first light field photo, the terminal device can search for the first relationship person in the first light field photo. If the first relationship person is found, the first light field photo is refocused with the first relationship person as the refocus focal point. Compared with the method of randomly selecting the refocus focal point, the light field photo displayed in this embodiment is more in line with the user's intention, and the user experience is improved.

[0009] In an implementation, determining the target person from the N persons according to the facial feature of the first relationship person includes: matching the facial feature of the first relationship person with facial features of the N persons; when there is a facial feature matching the facial feature of the first relationship person among the facial features of the N persons, determining that the person corresponding to the facial feature matching the facial feature of the first relationship person is the target person; when there is no facial feature matching the facial feature of the first relationship person among the facial features of the N persons, obtaining a facial feature of a second relationship person of the user, the priority of the second relationship person being lower than the priority of the first relationship person; and determining the target person from the N persons according to the facial feature of the second relationship person.

[0010] The user can set multiple relationship persons in advance and set priorities of the multiple relationship persons. The terminal device first searches for a relationship person with a higher priority, for example, a first relationship person, in the first light field photo. When the terminal device does not find the first relationship person in the first light field photo, the terminal device searches for a relationship person with a lower priority, for example, a second relationship person, in the first light field photo. Compared with the method of randomly searching for a relationship person, the light field photo displayed in this embodiment is more in line with the user's intention, and the user experience is improved.

[0011] In an implementation, the method further includes: determining the target person from the second light field photo according to the facial feature of the target person; refocusing the second light field photo with the target person in the second light field photo as a refocus focal point to obtain a second refocused photo; receiving a second operation of the user, the second operation being used to display the second light field photo; and in response to the second operation, displaying the second refocused photo.

[0012] After the terminal device determines the target person, the terminal device can refocus other light field photos (e.g., a second light field photo) in the photo album according to the target person. When the user performs a second operation (e.g., a sliding operation on the photo album) on the terminal device, the terminal device can immediately display the second refocused photo, thereby reducing the delay in displaying the second refocused photo.

[0013] In an implementation manner, the method further includes: obtaining a visual focus point of the user; and refocusing the first refocused photo with the target region as a refocusing focus point when the visual focus point stays on the target region of the first refocused photo for more than a time threshold.

[0014] When the user views the first refocused photo, the region of interest of the user can change. The visual focus point of the user reflects the region of interest of the user. Therefore, the terminal device can obtain the visual focus point of the user, refocus the first refocused photo with the region where the visual focus point is located (i.e., the target region) as a refocusing focus point, and display the refocused light field photo to the user, thereby displaying a light field photo that is more in line with the intention of the user.

[0015] In an implementation manner, the method further includes: obtaining a visual focus point of the user; obtaining a moving direction and a moving distance of the head of the user; and performing a perspective switching process on the first refocused photo according to the moving direction and the moving distance when the visual focus point stays on the first refocused photo.

[0016] The light field photo can adjust the perspective. When the visual focus point of the user is on the light field photo, the moving direction and the moving distance of the head of the user can reflect the perspective that the user wants to see. The terminal device can perform a perspective switching process on the first refocused photo according to the moving direction and the moving distance of the head of the user, thereby displaying a light field photo that is more in line with the intention of the user.

[0017] In a second aspect, a device for displaying a light field photo is provided, including units for performing any of the methods in the first aspect. The device can be a terminal device or a chip in the terminal device. The device can include a processing unit.

[0018] When the device is a terminal device, the processing unit can be a processor. The terminal device can further include a memory for storing computer program code, which, when executed by the processor, causes the terminal device to perform any of the methods in the first aspect.

[0019] When the device is a chip in a terminal device, the processing unit can be a logic processing unit inside the chip; the chip can further include a memory, which can be a memory (for example, a register, a cache, etc.) inside the chip or a memory (for example, a read-only memory, a random access memory, etc.) outside the chip; the memory is used to store computer program codes, and when the processor executes the computer program codes stored in the memory, the chip executes any one of the methods of the first aspect.

[0020] In a third aspect, a computer readable storage medium is provided, which stores computer program codes, and when the computer program codes are run by a device for displaying a light field photo, the device executes any one of the methods of the first aspect.

[0021] In a fourth aspect, a computer program product is provided, which includes computer program codes, and when the computer program codes are run by a device for displaying a light field photo, the device executes any one of the methods of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 FIG. 1 is a schematic diagram of a hardware system suitable for the device of the present application;

[0023] Figure 2 FIG. 2 is a schematic diagram of a software system suitable for the device of the present application;

[0024] Figure 3 FIG. 3 is a schematic diagram of a principle of a light field camera capturing a light field photo according to the present application;

[0025] Figure 4 FIG. 4 is a schematic diagram of a sub-aperture image according to the present application;

[0026] Figure 5 FIG. 5 is a schematic diagram of a refocusing principle in a two-dimensional case according to the present application;

[0027] Figure 6 FIG. 6 is a schematic diagram of a light field photo according to the present application;

[0028] Figure 7 FIG. 7 is a schematic diagram of a refocused photo according to the present application;

[0029] Figure 8 FIG. 8 is a flow chart of a method for displaying a light field photo according to the present application;

[0030] Figure 9 FIG. 9 is a schematic diagram of a first operation according to the present application;

[0031] Figure 10 FIG. 10 is a schematic diagram of confirming a recognition result according to the present application;

[0032] Figure 11 This is a flowchart of another method for displaying light field photographs provided in this application;

[0033] Figure 12 This is a flowchart of another method for displaying light field photographs provided in this application;

[0034] Figure 13 This is a schematic diagram of a second operation provided in this application;

[0035] Figure 14 This is a schematic diagram of a refocused photograph provided in this application. Detailed Implementation

[0036] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0037] Figure 1 A hardware system suitable for the apparatus of this application is shown.

[0038] Device 100 may be a mobile phone, smart screen, tablet computer, wearable electronic device, in-vehicle electronic device, augmented reality (AR) device, virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), projector and other terminal devices. This application embodiment does not limit the specific type of device 100.

[0039] Device 100 may include a processor 110, internal memory 120, sensor module 130, display screen 140, camera 150, etc. Sensor module 130 may include a pressure sensor 130A, a gyroscope sensor 130B, a proximity sensor 130C, a touch sensor 130D, etc.

[0040] It should be noted that, Figure 1 The structure shown does not constitute a specific limitation on device 100. In other embodiments of this application, device 100 may include a... Figure 1 The components shown may include more or fewer components, or the device 100 may include... Figure 1 The components shown may be a combination of certain components, or the device 100 may include... Figure 1 Sub-components of some of the components shown. Figure 1 The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0041] The processor 110 can include one or more processing units. For example, the processor 110 can include at least one of the following processing units: an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, a neural-network processing unit (NPU). Among them, different processing units can be independent devices or integrated devices.

[0042] The controller can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching and executing instructions.

[0043] The processor 110 can also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can save instructions or data that the processor 110 has just used or repeatedly uses. If the processor 110 needs to use the instructions or data again, it can directly call from the memory. Avoiding repeated access, reducing the waiting time of the processor 110, thus improving the efficiency of the system.

[0044] In some embodiments, the processor 110 can include one or more interfaces. For example, the processor 110 can include at least one of the following interfaces: an inter-integrated circuit (I2C) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface.

[0045] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 can contain multiple sets of I2C buses. The processor 110 can be coupled to the touch sensor 130D, the charger, the flash, the camera 150, etc. through different I2C bus interfaces respectively. For example, the processor 110 can be coupled to the touch sensor 130D through an I2C interface, so that the processor 110 and the touch sensor 130D communicate through the I2C bus interface, and the touch function of the device 100 is realized.

[0046] The MIPI interface can be used to connect the processor 110 and the peripheral devices such as the display screen 140 and the camera 150. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 150 communicate through the CSI interface, and the shooting function of the device 100 is realized. The processor 110 and the display screen 140 communicate through the DSI interface, and the display function of the device 100 is realized.

[0047] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal interface or as a data signal interface. In some embodiments, the GPIO interface can be used to connect the processor 110 and the camera 150, the display screen 140 and the sensor module 130. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface or a MIPI interface.

[0048] Figure 1 The connection relationship between the modules shown is only illustrative and does not constitute a limitation on the connection relationship between the modules of the device 100. Alternatively, the modules of the device 100 can also use a combination of the above-mentioned various connection modes.

[0049] The device 100 can realize the display function through the GPU, the display screen 140 and the application processor. The GPU is a microprocessor for image processing, connected to the display screen 140 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs that execute program instructions to generate or change display information.

[0050] The display screen 140 can be used to display images or videos. The display screen 140 includes a display panel. The display panel can employ a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flex light-emitting diode (FLED), a mini light-emitting diode (Mini LED), a micro light-emitting diode (Micro LED), a micro OLED, or a quantum dot light emitting diode (QLED). In some embodiments, the device 100 can include 1 or N display screens 140, N being a positive integer greater than 1.

[0051] The device 100 can implement the photographing function through an ISP, a camera 150, a video codec, a GPU, a display screen 140, and an application processor, etc.

[0052] The ISP is used to process the data fed back by the camera 150. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye. The ISP can optimize the noise, brightness, and color of the image through algorithms, and can also optimize the exposure and color temperature of the shooting scene, etc. In some embodiments, the ISP can be arranged in the camera 150.

[0053] The camera 150 is used to capture still images or videos. Objects generate optical images through lenses and project them onto photosensitive elements. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into a standard red green blue (RGB), YUV, etc. format image signal. In some embodiments, the device 100 can include 1 or N cameras 150, N being a positive integer greater than 1.

[0054] The digital signal processor is used to process digital signals, which can process digital image signals, but also other digital signals. For example, when the device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0055] The video codec is used to compress or decompress digital video. The device 100 can support one or more video codecs. In this way, the device 100 can play or record videos in multiple encoding formats, such as: moving picture experts group (MPEG) 1, MPEG 2, MPEG 3 and MPEG 4.

[0056] The NPU is a kind of processor that draws on the structure of biological neural network, for example, it draws on the transmission mode between human brain neurons to process input information quickly, and can also learn continuously. Through the NPU, the device 100 can realize intelligent cognition and other functions, such as: image recognition, face recognition, voice recognition and text understanding.

[0057] The internal memory 120 can be used to store computer executable program codes, which include instructions. The internal memory 120 can include a program storage area and a data storage area. The program storage area can store an operating system and application programs required by at least one function (for example, a sound playing function and an image displaying function). The data storage area can store data (for example, a refocus photo) created during the use of the device 100. In addition, the internal memory 120 can include a high-speed random access memory, and can also include a non-volatile memory, such as: at least one magnetic disk storage device, a flash memory device and a universal flash storage (UFS), etc. The processor 110 executes various processing methods of the device 100 by running instructions stored in the internal memory 120 and / or instructions stored in the memory arranged in the processor.

[0058] The pressure sensor 130A is configured to sense a pressure signal and convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 130A can be disposed on the display screen 140. The pressure sensor 130A can be of various types, such as a resistive pressure sensor, an inductive pressure sensor, or a capacitive pressure sensor. The capacitive pressure sensor can include at least two parallel plates of conductive material. When a force is applied to the pressure sensor 130A, the capacitance between the electrodes changes, and the device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation is applied to the display screen 140, the device 100 detects the touch operation based on the pressure sensor 130A. The device 100 can also calculate the location of the touch based on the detection signal of the pressure sensor 130A. In some embodiments, touch operations applied to the same touch location but with different touch operation intensities can correspond to different operation instructions. For example, when a touch operation with an intensity less than a first pressure threshold is applied to a short message application icon, an instruction to view short messages is executed; when a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the short message application icon, an instruction to create a new short message is executed.

[0059] The gyroscope sensor 130B can be configured to determine the motion attitude of the device 100. In some embodiments, the angular velocity of the device 100 around three axes (i.e., the x-axis, the y-axis, and the z-axis) can be determined by the gyroscope sensor 130B. The gyroscope sensor 130B can be used for anti-shake photography. For example, when the shutter is pressed, the gyroscope sensor 130B detects the angle of the device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and lets the lens offset the shaking of the device 100 by moving in the opposite direction, thereby achieving anti-shake. The gyroscope sensor 130B can also be used in scenarios such as navigation and motion sensing games.

[0060] The distance sensor 130C is configured to measure distance. The device 100 can measure distance by infrared or laser. In some embodiments, for example in a shooting scenario, the device 100 can use the distance sensor 130C to measure distance to achieve fast focusing.

[0061] The touch sensor 130D, also referred to as a touch device. The touch sensor 130D can be disposed on the display screen 140, and the touch sensor 130D and the display screen 140 together form a touch screen, also referred to as a touch screen. The touch sensor 130D is configured to detect a touch operation applied thereto or in the vicinity thereof. The touch sensor 130D can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 140. In other embodiments, the touch sensor 130D can also be disposed on the surface of the device 100 and disposed at a different location from the display screen 140.

[0062] The hardware system of the apparatus 100 is described in detail above, and the software system of the apparatus 100 is described below. The software system can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. Embodiments of the present application take the layered architecture as an example to describe the software system of the apparatus 100.

[0063] As shown in Figure 2 , the software system adopting the layered architecture is divided into several layers, each layer has a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the software system can be divided into four layers, from top to bottom, the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0064] The application layer can include camera, gallery, calendar, call, map, navigation, wireless local area network (WLAN), Bluetooth, music, video, short message, and the like.

[0065] The application framework layer provides the application programming interface (API) and programming framework for the application layer application. The application framework layer can include some predefined functions.

[0066] For example, the application framework layer includes a window manager, a content provider, a view system, a phone manager, a resource manager, and a notification manager.

[0067] The window manager is used to manage the window program. The window manager can obtain the size of the display screen, determine whether there is a status bar, a lock screen, and a screen capture.

[0068] The content provider is used to store and obtain data, and make the data accessible to the application. The data can include video, image, audio, dialed and received calls, browsing history and bookmarks, and phonebook.

[0069] The view system includes visual controls, such as controls for displaying text and controls for displaying pictures. The view system can be used to build an application. A display interface can be composed of one or more views, for example, a display interface including a short message notification icon can include a view for displaying text and a view for displaying pictures.

[0070] The phone manager is used to provide the communication function of the apparatus 100, such as the management of the call state (on or off).

[0071] The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, and video files.

[0072] A notification manager enables applications to display notification information in the status bar, which can be used to convey alert-type messages that can automatically disappear after a brief stay without user interaction. For example, the notification manager is used for download completion alerts and message reminders. The notification manager can also manage notifications that appear in the form of a graphic or scrolling text at the top of the system status bar, such as notifications for applications running in the background. The notification manager can also manage notifications that appear in the form of a dialog window on the screen, such as a text message in the status bar, a prompt sound, a vibration of the electronic device, and a blinking of an indicator light.

[0073] The Android Runtime includes a core library and a virtual machine. The Android runtime is responsible for scheduling and managing the Android system.

[0074] The core library includes two parts: one part is a function function that the java language needs to call, and the other part is the core library of Android.

[0075] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the java files of the application layer and the application framework layer into binary files. The virtual machine is used to perform functions such as management of object life cycle, stack management, thread management, security and exception management, and garbage collection.

[0076] The system library can include multiple functional modules, such as: a surface manager, media libraries, a three-dimensional graphics processing library (for example: open graphics library for embedded systems (OpenGL ES) and a 2D graphics engine (for example: skia graphics library (SGL)).

[0077] The surface manager is used to manage the display subsystem and provides a fusion of 2D layers and 3D layers for multiple applications.

[0078] The media library supports playback and recording of various audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, Moving Picture Experts Group Audio Layer III (MP3), Advanced Audio Coding (AAC), Adaptive Multi-rate (AMR), Joint Photographic Experts Group (JPG), and Portable Network Graphics (PNG).

[0079] The 3D graphics processing library can be used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0080] A 2D graphics engine is a graphics engine for 2D drawing.

[0081] The kernel layer is the layer between hardware and software. The kernel layer can include driver modules such as display drivers, camera drivers, and sensor drivers.

[0082] The workflow of the software and hardware systems of device 100 will be illustrated below with reference to the scene of the photo capture.

[0083] When a user touches the touch sensor 130D, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into raw input events, which include information such as touch coordinates and a timestamp of the touch operation. These raw input events are stored in the kernel layer. The application framework layer retrieves the raw input events from the kernel layer, identifies the corresponding control, and notifies the application (APP) that controls the control. For example, if the touch operation is a single click and the APP corresponding to the control is a camera APP, the camera APP, after being activated by the single click, can call the camera driver in the kernel layer via API to control the camera 150 to take a picture.

[0084] The following uses a mobile phone as an example to introduce the method for displaying light field photographs provided in this application.

[0085] First, a brief introduction to the concepts involved in this application will be given.

[0086] A light field photograph is a photograph that contains information such as the intensity and direction of light, providing richer information than a regular photograph. Light field photographs are obtained by taking pictures with a light field camera. Figure 3This is a schematic diagram of a light field camera used in this application to capture a light field photograph.

[0087] Light rays from different directions enter the camera through the main lens and converge onto different microlenses in the microlens array. After passing through the microlenses, they reach the photosensitive unit array. Each microlens can be considered a macropixel, and each photosensitive unit can be considered a metapixel. Each microlens corresponds to n*n photosensitive units, meaning each macropixel contains n*n metapixels, where n is a positive integer greater than 1. The brightness of a macropixel is the sum of the brightness of its corresponding n*n metapixels; that is, the brightness of a macropixel is the integral of the brightness of all its corresponding metapixels.

[0088] The light entering the main lens can be represented by L(u,v,s,t), where L represents the light field, s and t represent the number of rows and columns of the microlens array, and u and v represent the number of rows and columns of the photosensitive unit corresponding to each microlens. Then the brightness of the macropixel can be represented by formula (1):

[0089] I(x,y)=∫∫L(u,v,s,t)dudv (1).

[0090] In formula (1), x and y are coordinate axes, with the x-axis parallel to the paper and the y-axis perpendicular to the paper. Figure 3 (The y-axis is not shown). In the light field L(u,v,s,t), if s and t are fixed (i.e., a macro pixel is selected), traversing u and v will yield an image of n*n meta pixels corresponding to that macro pixel; if u and v are fixed (i.e., a meta pixel is selected), traversing s and t will yield an image of the sub-aperture of the main lens, and a total of n*n sub-aperture images can be obtained.

[0091] Figure 4 A schematic diagram of a sub-aperture image is shown. Photosensitive unit set A and photosensitive unit set B correspond to microlens A and microlens B (not shown in the figure), respectively. After light enters the main lens through a sub-aperture, it is projected onto photosensitive unit set A through microlens A, and onto photosensitive unit set B through microlens B. Photosensitive units A1 and B1 are photosensitive units at the same location, photosensitive units A2 and B2 are photosensitive units at the same location, photosensitive units A3 and B3 are photosensitive units at the same location, and photosensitive units A4 and B4 are photosensitive units at the same location.

[0092] The corresponding sub-pixels of the light sensing unit A1 and the light sensing unit B1 belong to the sub-pixel of the sub-aperture image 1, the corresponding sub-pixels of the light sensing unit A2 and the light sensing unit B2 belong to the sub-pixel of the sub-aperture image 2, the corresponding sub-pixels of the light sensing unit A3 and the light sensing unit B3 belong to the sub-pixel of the sub-aperture image 3, and the corresponding sub-pixels of the light sensing unit A3 and the light sensing unit B3 belong to the sub-pixel of the sub-aperture image 3. The sub-aperture image 1 to the sub-aperture image 4 are images of different viewing angles, and switching different light sensing units can switch the viewing angle of the light field photo.

[0093] The principle of refocusing will be introduced below.

[0094] Refocusing is to project the collected light field to a new image plane for integration, as shown in Figure 5

[0095] Figure 5 is a refocusing schematic diagram in the two-dimensional case, L(u,s) is the collected light field, U represents the plane where the main lens is located, S represents the plane where the microlens array is located, and the distance between the two planes is l. The distance between the new focal plane S' and the plane U is l', l' = a * l, and the image formed on the plane S' is equal to the integral of the light field between U and S', that is:

[0096] I(s') = ∫L'(u,s')du (2).

[0097] For the same light ray, there should be:

[0098] L(u,s) = L'(u,s') (3).

[0099] Meanwhile, according to the coordinates of the intersection of the light ray and each plane, the following relationship can be obtained:

[0100]

[0101] Where s'-u represents the distance between s' and u, and s-u represents the distance between s' and u.

[0102] Substituting l' = a * l into equation (4) can obtain:

[0103]

[0104] Substituting equation (5) into equation (2) can obtain:

[0105]

[0106] Equation (6) is the refocusing formula in the two-dimensional case. By extending the two-dimensional case to the four-dimensional case, the following refocusing formula can be obtained:

[0107] ​

[0108] As can be seen from equation (7), refocusing is a process of translating the light field in the position dimension and then integrating in the direction dimension.

[0109] The method for displaying a light field photo provided in the present application is described below.

[0110] Figure 6 is a schematic diagram of a light field photo suitable for the present application, which contains three persons, namely, person A, person B and person C, wherein person A is located at the frontmost position, person C is located at the last position, and person B is located between person A and person C, i.e., when the light field camera captures the light field photo, person A is closest to the light field camera, and person C is farthest from the light field camera.

[0111] When the user opens the light field photo on the mobile phone, the default focus of the light field photo is located at the focal plane of person A, thus, the user sees a clear image of person A, and sees blurred images of person B and person C. In the present application, a solid line is used to represent a clear image, and a dashed line is used to represent a blurred image.

[0112] If the user is person B and wishes to see a clear image of himself, the user needs to manually operate (e.g., click on the position of person B in the light field photo) to trigger the mobile phone to refocus and display the light field photo (i.e., the refocused photo) shown in Figure 7 It can be seen that the method needs the user to manually operate to display a clear image of himself, and if the number of persons in the light field photo is large, the user may need a long time to find his position in the light field photo, thus leading to a poor experience.

[0113] Figure 8 is a flowchart of a method for displaying a light field photo provided in the present application. The method comprises the following steps.

[0114] S801, receiving a first operation, the first operation being used to display a first light field photo.

[0115] As shown in Figure 9 , the first operation can be a touch operation of the user on a photo in an album, or a voice command, and the present application does not limit the specific type of the first operation.

[0116] S802, in response to the first operation, obtaining facial features of the user through the camera 150.

[0117] After receiving the first operation, the mobile phone can call the kernel layer camera driver through an API, and control the camera 150 to obtain the facial features of the user viewing the first light field photo through the camera driver. The camera 150 can be a visible light camera or a structured light camera (such as an infrared camera), and the specific type of the camera 150 is not limited in the present application.

[0118] S803, determining whether there is a target person in the first light field photo that matches the facial features of the user.

[0119] The mobile phone can recognize the target person in the first light field photo through a face recognition method, and the specific face recognition method is not limited in the present application.

[0120] When the user views the light field photo, it is usually desired to see a clear image of oneself. If the mobile phone recognizes a person matching the facial features of the user from the N persons included in the first light field photo, the mobile phone can take the person as the target person. If the mobile phone does not recognize a person matching the facial features of the user from the N persons included in the first light field photo, it is possible that the facial features obtained by the camera 150 are not clear, and the mobile phone can call the camera 150 again to obtain the facial features of the user, and recognize the target person in the first light field photo based on the new facial features.

[0121] When the mobile phone repeatedly recognizes but still does not find the target person, the mobile phone can display the first light field photo with a default focus, and manually select the refocusing focus by the user.

[0122] Optionally, the mobile phone can display the first light field photo with a default focus, and prompt the user to confirm the recognition result in the first light field photo. As shown in Figure 10 , the mobile phone can perform thickening processing on the outline of the recognition result. If the user confirms that the recognition result is correct, the user can click the recognition result, and the mobile phone takes the recognition result as the target person. If the user thinks that the recognition result is incorrect, the user can click the person who the user wants to see, and the mobile phone takes the person clicked by the user as the target person.

[0123] S804, refocusing the first light field photo with the target person as the focus.

[0124] S805, displaying the refocused photo.

[0125] For example, the target person is person B, and the mobile phone can refocus the first light field photo with person B as the focus to obtain a first refocused photo. The display effect of the first refocused photo is shown in Figure 7 .

[0126] The above method can present the visual effect desired by the user without manual selection of the focus by the user, thereby improving the user experience.

[0127] When the user is watching the first refocused photo, the user's interested region (a region on the first refocused photo) can change. The phone can determine the user's current interested region through S806.

[0128] S806, detect the user's gaze focus and head movement through the camera 150.

[0129] The user's gaze focus reflects the user's interested region, therefore, the terminal device can obtain the user's gaze focus, refocus the first refocused photo with the region where the gaze focus stays (i.e., the target region) as the refocusing focus, and display the refocused light field photo to the user, so that the light field photo that is more in line with the user's intention can be displayed.

[0130] S807, determine whether the time for which the gaze focus stays on the refocused photo is greater than a time threshold.

[0131] For example, the phone determines that the time for which the gaze focus stays on a region (a region different from the region where the current target person stays) of the first refocused photo exceeds a time threshold (e.g., 1s), and then determines that the region is the target region, and subsequently can perform S808.

[0132] S808, refocus the refocused photo with the region (the target region) where the gaze focus stays as the refocusing focus.

[0133] If the phone determines that the time for which the gaze focus stays on a region of the refocused photo (e.g., the first refocused photo) does not exceed the time threshold, the phone can perform S809.

[0134] S809, determine whether the gaze focus stays on the refocused photo and whether the head moves.

[0135] The light field photo can adjust the view angle, when the user's gaze focus is on the light field photo, the moving direction and distance of the user's head can reflect the view angle that the user wants to see, and the terminal device can perform view angle switching processing on the first refocused photo according to the moving direction and distance of the user's head, so that the light field photo that is more in line with the user's intention can be displayed.

[0136] If the gaze focus is not on the refocused photo, or if the head does not move, the phone can return to perform S806 to continue detecting the user's gaze focus and head movement.

[0137] If the gaze focus is on the refocused photo and the head moves, the phone can perform S810.

[0138] S810, determine the view angle change according to the moving direction and distance of the head.

[0139] S811, performing view angle switching processing on the refocused photo according to the view angle change condition.

[0140] For example, the user moves the head by 1 cm, and the phone can rotate the view angle of the refocused photo by 5°; the user moves the head to the left, and the phone can rotate the refocused photo counterclockwise; the user moves the head to the right, and the phone can rotate the refocused photo clockwise. The specific manner in which the phone determines the view angle change condition according to the moving direction and distance of the head is not limited in the present application.

[0141] Figure 11 is a flowchart of another method for displaying a light field photo provided in the present application. The method comprises the following steps.

[0142] S1101, receiving a first operation, the first operation being used to display a first light field photo.

[0143] As shown in Figure 9 , the first operation can be a touch operation of the user on a photo in the album, or a voice command, and the specific type of the first operation is not limited in the present application.

[0144] S1102, obtaining the facial feature of the user through the camera 150 in response to the first operation.

[0145] After receiving the first operation, the phone can call the camera driver in the kernel layer through the API, and control the camera 150 to obtain the facial feature of the user currently viewing the first light field photo through the camera driver. The camera 150 can be a visible light camera or a structured light camera (such as an infrared camera), and the specific type of the camera 150 is not limited in the present application.

[0146] S1103, determining whether there is a target person in the first light field photo that matches the facial feature of the user.

[0147] The phone can recognize the target person in the first light field photo through a face recognition method, and the specific face recognition method is not limited in the present application.

[0148] When the user views the light field photo, it is usually desired to see a clear image of oneself. If the phone recognizes a person that matches the facial feature of the user from the N persons included in the first light field photo, the phone can take the person as the target person. If the phone does not recognize a person that matches the facial feature of the user from the N persons included in the first light field photo, it is possible that the facial feature obtained by the camera 150 is not clear, and the phone can call the camera 150 again to obtain the facial feature of the user, and recognize the target person in the first light field photo based on the new facial feature.

[0149] After the mobile phone determines that the target person matching the facial feature of the user exists in the first light field photo, the mobile phone can perform S1104-S1105.

[0150] S1104, refocus the first light field photo with the target person as the focus.

[0151] S1105, display the refocused photo.

[0152] For example, the target person is person B, the mobile phone can refocus the first light field photo with person B as the focus to obtain a first refocused photo. The display effect of the first refocused photo is shown in FIG. 6B. Figure 7

[0153] When the mobile phone fails to find the target person after multiple times of identification, the mobile phone can perform S1106-S1107.

[0154] S1106, obtain the facial feature of a relation person of the user.

[0155] The relation person can be a person having interpersonal relationship with the user, such as a friend, a relative, a classmate or a colleague. The user can pre-label some photos of the relation person, so that the mobile phone can obtain the facial feature of the relation person.

[0156] For example, the user can add a friend label to a photo containing the avatar of person A, and the mobile phone extracts the facial feature of person A after face recognition of the photo and stores the facial feature in the memory. When performing S1106, the mobile phone reads the facial feature of person A from the memory.

[0157] S1107, determine whether the target person matching the facial feature of the relation person exists in the first light field photo.

[0158] The mobile phone can extract the facial features of all persons in the first light field photo, and match the facial feature of the relation person with the facial features of all persons to determine whether the target person matching the facial feature of the relation person exists in the first light field photo.

[0159] Optionally, the user can set multiple relation persons in advance, and set the priority of the multiple relation persons. The mobile phone first searches for a relation person with higher priority, such as a first relation person, in the first light field photo. When the mobile phone finds a person corresponding to the first relation person in the first light field photo, the mobile phone sets the person as the target person. When the mobile phone fails to find the first relation person in the first light field photo, the mobile phone searches for a relation person with lower priority, such as a second relation person, in the first light field photo. Compared with the method of searching for the relation person randomly, the light field photo displayed in this embodiment is more in line with the intention of the user, and the user experience is improved.

[0160] When the mobile phone determines that the target person matching the facial feature of the relation person exists in the first light field photo, the mobile phone performs S1107.​

[0161] The mobile phone determines that there is no target person in the first light field photo that matches the facial features of the relationship person, and performs S1108.

[0162] S1108, display the first light field photo based on the default focus point.

[0163] After displaying the first light field photo, the mobile phone can determine the refocus focus point through the user's visual focus point, that is, perform S1109-S1111.

[0164] When the user views the photo (the first refocus photo or the first light field photo), the user's area of interest (an area on the first refocus photo or an area on the first light field photo) can change. The mobile phone can determine the user's current area of interest through S1109.

[0165] S1109, detect the user's visual focus point and head movement through the camera 150.

[0166] The user's visual focus point reflects the user's area of interest, so the mobile phone can obtain the user's visual focus point, refocus the photo with the area where the visual focus point stays (that is, the target area) as the refocus focus point, and display the refocused photo to the user, so that a light field photo that is more in line with the user's intention can be displayed.

[0167] S1110, determine whether the visual focus point stays on the refocus photo for more than a time threshold.

[0168] For example, the mobile phone determines that the visual focus point stays on an area (an area different from the area where the current target person is located) of the first refocus photo for more than a time threshold (for example, 1s), and determines that the area is the target area, and then can perform S1111.

[0169] S1111, refocus the photo with the area where the visual focus point stays (the target area) as the refocus focus point.

[0170] If the mobile phone determines that the visual focus point stays on an area of the photo (for example, the first light field photo or the first refocus photo) for less than a time threshold, the mobile phone can perform S1112.

[0171] S1112, determine whether the visual focus point stays on the photo and the head moves.

[0172] The light field photo can adjust the view angle. When the user's visual focus is on the light field photo, the moving direction and distance of the user's head can reflect the view angle that the user wants to see. The mobile phone can perform view angle switching processing on the first refocused photo or the first light field photo according to the moving direction and distance of the user's head, so that the light field photo that is more consistent with the user's intention can be displayed.

[0173] If the visual focus is not on the light field photo, or if the head does not move, the mobile phone can return to perform S1109 and continue to detect the user's visual focus and head movement.

[0174] If the visual focus stays on the photo and the head moves, the mobile phone can perform S1113.

[0175] S1113, determining the view angle change according to the moving direction and distance of the head.

[0176] S1114, performing view angle switching processing on the photo according to the view angle change.

[0177] For example, if the user's head moves 1 cm, the mobile phone can rotate the view angle of the photo by 5°; if the user's head moves to the left, the mobile phone can rotate the photo counterclockwise; if the user's head moves to the right, the mobile phone can rotate the photo clockwise. The specific way in which the mobile phone determines the view angle change according to the moving direction and distance of the head is not limited in the present application.

[0178] In some scenarios, the user can view multiple light field photos, and the mobile phone can perform the method shown in Figure 12 .

[0179] S1201, determining a target person from the first light field photo.

[0180] S1201 can be S803, S1103 or S1107, that is, the target person can be the user currently using the mobile phone, or a relative of the user.

[0181] S1202, determining the target person from the second light field photo according to the face of the target person.

[0182] S1203, refocusing the second light field photo with the target person in the second light field photo as the focus.

[0183] S1204, receiving a second operation of the user, the second operation being used to display the second light field photo.

[0184] As shown in Figure 13 , the user performs a left swipe operation on the first refocused photo, which is an example of the second operation.

[0185] After receiving the second operation, the mobile phone determines that the user needs to display the next light field photo (i.e., the second refocused photo), and then the mobile phone can perform S1205.

[0186] S1205, in response to the second operation, displaying the second refocused photo.

[0187] The mobile phone can directly display the second light field photo after refocusing processing (i.e., the second refocused photo), as shown in FIG. 12B. Figure 14

[0188] In this way, after the user performs the second operation, the user can see the second light field photo with the focus point being the user or the user's relative, improving the user experience. In addition, since the refocusing processing of the second light field photo is performed before the second operation, after the user performs the second operation, the mobile phone can immediately display the second refocused photo, thereby reducing the delay in displaying the second refocused photo.

[0189] The present application also provides a computer program product, which, when executed by a processor, implements the method described in any of the method embodiments of the present application.

[0190] The computer program product can be stored in a memory and finally converted into an executable target file that can be executed by the processor through processes such as preprocessing, compiling, assembling, and linking.

[0191] The computer program product can also be a code that is fixed in a chip. The present application does not limit the specific form of the computer program product.

[0192] The present application also provides a computer readable storage medium, which stores a computer program that, when executed by a computer, implements the method described in any of the method embodiments of the present application. The computer program can be a high-level language program or an executable target program.

[0193] ​The computer-readable storage medium can be volatile memory or non-volatile memory, or it can include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0194] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process and technical effects of the above-described apparatus and equipment can be referred to the corresponding processes and technical effects in the foregoing method embodiments, and will not be repeated here.

[0195] In the several embodiments provided in this application, the systems, apparatuses, and methods disclosed can be implemented in other ways. For example, some features of the method embodiments described above can be ignored or not performed. The apparatus embodiments described above are merely illustrative; the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Multiple units or components can be combined or integrated into another system. Furthermore, the coupling between units or components can be direct coupling or indirect coupling, including electrical, mechanical, or other forms of connection.

[0196] It should be understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0197] In addition, the terms "system" and "network" are often used interchangeably herein. The term "and / or", herein merely describes an associated relationship between associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the following three cases: A exists alone, A and B exist together, and B exists alone. In addition, the character " / " generally represents an "or" relationship between the front and back associated objects.

[0198] In summary, the above only describes the preferred embodiments of the technical solutions of the present application, and is not used to limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method of displaying a light field photograph, characterized by, The method comprises: receiving a first operation of a user, the first operation being used to display a first light field photo, the first light field photo comprising N persons, N being a positive integer; in response to the first operation, acquiring a facial feature of the user; determining whether there is a person matching the facial feature of the user in the N persons; if there is no person matching the facial feature of the user in the N persons, acquiring a facial feature of a relationship person, the relationship person being a relationship person of the user; determining whether there is a facial feature matching the facial feature of the first relationship person in the N persons, the first relationship person being one of the relationship persons; if there is no facial feature matching the facial feature of the first relationship person in the N persons, acquiring a facial feature of a second relationship person, the second relationship person being one of the relationship persons, and the priority of the second relationship person being lower than that of the first relationship person; refocusing the first light field photo with the target person as a refocusing focus point to obtain a first refocused photo; displaying the first refocused photo, the first refocused photo containing intensity and direction information of light rays; if there is no, displaying the first light field photo based on a default focus point; acquiring a visual focus point of the user; when the visual focus point stays on a target area on the first refocused photo for more than a time threshold, refocusing the first refocused photo with the target area as a refocusing focus point; when the visual focus point stays on a target area on the first refocused photo for less than or equal to a time threshold, determining whether the visual focus point stays on the first refocused photo and whether the head of the user moves; when the visual focus point stays on the first refocused photo and the head of the user moves, determining a visual angle change condition according to the moving direction and moving distance of the head of the user, and performing visual angle switching processing on the photo according to the visual angle change condition; determining the target person from a second light field photo according to the facial feature of the target person; refocusing the second light field photo with the target person in the second light field photo as a refocusing focus point to obtain a second refocused photo; after refocusing the second light field photo, receiving a second operation of the user, the second operation being a sliding operation used to display a next light field photo; in response to the second operation, displaying the second refocused photo.

2. The method of claim 1, wherein, The method further comprises: if there is a person matching the facial feature of the user in the N persons, determining the person matching the facial feature of the user in the N persons as the target person.

3. The method of claim 2, wherein, The method further comprises: matching the facial feature of the first relationship person with the facial features of the N persons; when there is a facial feature matching the facial feature of the first relationship person in the facial features of the N persons, determining the person corresponding to the facial feature matching the facial feature of the first relationship person as the target person.

4. An apparatus for displaying a light field photograph, characterized in that An apparatus comprising a processor and a memory coupled to the processor, the memory to store a computer program that when executed by the processor causes the apparatus to perform the method of any one of claims 1 to 3.

5. A computer readable storage medium, characterized in that, The computer readable storage medium stores a computer program that when executed by a processor causes the processor to perform the method of any one of claims 1 to 3.

Images