Automatic photography composition recommendations

Abstract

A device for taking a photo includes: a camera; a display; a processor for capturing multiple temporary images of one or more objects using the camera; extracting a reconstructed scene from the multiple temporary images; capturing a focused image of a user intending to take a photo using the camera; generating an object graph in the focused image from the reconstructed scene; estimating a skeleton from an object selected from the object graph; combining the skeleton with the focused image; verifying whether the composition of the focused image is good; and providing instructions to the user to adjust the composition using the display. The device implements an improved photographic composition.

Description

Technical Field

[0001] The present invention relates to a device and a method for taking a photo. Background Art

[0002] Photography is widely considered a hobby and a profession, and it is difficult for non-professionals to take well-composed photos. Professional photographers often use skeletons as a guide for photographic composition (i.e., photo layout). A skeleton is a set of lines and intersections (e.g., two vertical lines, two horizontal lines, and the 4 intersections of these lines). However, it is not easy for non-professionals to select a suitable skeleton and apply it to the current goal they are taking photos for to improve their photographic composition.

[0003] The prior art can be divided into two categories. One category involves the quality assessment of photography. These prior art techniques apply aesthetic quality assessment techniques to photos taken by users. The assessment information can be used to provide feedback to users so that they can take photos with high aesthetic quality. However, the focus of these prior art techniques is not on the quality of composition.

[0004] Another category involves photographic composition recommendations. There are three examples of these prior arts. The first prior art provides text suggestions for improving photographic composition. However, users are sometimes not clear about how to embody the text suggestions in the photographic composition. The second prior art describes an adaptive skeleton recognition based on image analysis. This prior art automatically identifies a suitable skeleton for a photo preview. However, if the scene is complex, this prior art is sometimes not applicable because the three-dimensional scene representation is not considered. The third prior art analyzes a portrait preview and displays arrow guidance for improving the portrait composition. The arrow guidance is similar to applying a skeleton. Under the arrow guidance, the user is able to improve the quality of the portrait composition without understanding the skeleton. However, this prior art can only handle simple skeletons and compositions, which hinders further improvement of the composition quality. Summary of the invention

[0005] The present invention provides a device for taking photos to automatically recommend a good composition when a user takes a photo, and also correct the composition according to user permission.

[0006] According to a first aspect, a device for taking a photo is provided. The device comprises: a camera; a display; a processor for capturing multiple temporary images of one or more objects using the camera; extracting a reconstructed scene from the multiple temporary images; capturing a focused image that a user intends to take a photo with the camera; generating an object graph in the focused image from the reconstructed scene; estimating a skeleton from an object selected from the object graph; combining the skeleton with the focused image; verifying whether the composition of the focused image is good; and providing instructions to the user to adjust the composition using the display.

[0007] In a possible implementation manner of the first aspect, the device further includes a speaker, and the processor is further configured to provide an instruction to the user to adjust the composition using the speaker.

[0008] In a possible implementation manner of the first aspect, the processor is further configured to: automatically adjust the composition if the user accepts the automatic adjustment.

[0009] In a possible implementation manner of the first aspect, the temporary images are multiple pairs of color images and depth images.

[0010] In a possible implementation manner of the first aspect, the reconstructed scene includes a polygonal mesh or a point cloud.

[0011] According to a second aspect, a method for taking a photo is provided. The method comprises: capturing multiple temporary images of one or more objects using a camera; extracting a reconstructed scene from the multiple temporary images; capturing a focused image that a user intends to take a photo with the camera; generating an object graph in the focused image from the reconstructed scene; estimating a skeleton from an object selected from the object graph; combining the skeleton with the focused image; verifying whether the composition of the focused image is good; and providing instructions to the user to adjust the composition using a display.

[0012] According to a third aspect, a storage medium storing a program is provided. When the program is executed by a processor, the program causes the processor to execute the method according to the second aspect.

[0013] Embodiments of the present invention improve photographic composition. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly describe the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. The drawings in the following description only illustrate some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0015] Figure 1 An example of the user operation provided by the first embodiment is shown.

[0016] Figure 2 is a block diagram of the hardware configuration of the mobile system 100 provided in the first embodiment.

[0017] Figure 3 is a block diagram of the functional configuration of the mobile system 100 provided in the first embodiment.

[0018] Figure 4 It is an overall flow chart provided by the first embodiment.

[0019] Figure 5 Shown in Figure 4 An example of an image to be processed in the steps of .

[0020] Figure 6 is a detailed flowchart of the object selection in step 403.

[0021] Figure 7 4 is a detailed flowchart of the composition recommendation in step 404.

[0022] Figure 8 An example of a predefined skeleton is shown.

[0023] Fig. 9 An example of an instruction of a user operation is shown.

[0024] Fig.10 is a detailed flowchart of the composition adjustment in step 406 .

[0025] Fig.11 An example of composition adjustment is shown. DETAILED DESCRIPTION

[0026] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should belong to the scope of protection of the present invention.

[0027] An example of a series of processes provided by an embodiment of the present invention is as follows. The user activates the camera function in the mobile system and keeps the camera in the system facing the object. The system automatically captures one or more images (these images may be in color format or depth format) and stores these images in a temporary cache. The temporary cache is deleted after the image is captured. The system uses these temporary images to construct scene information. The scene information may include a color image, a depth image, a normal map, a mesh, or a shape represented by an implicit function. According to the scene information, the candidate of the target object on the current focused image is estimated, and a skeleton (i.e., a set of finite two-dimensional lines) is recommended for photographic composition. The user can specify the target object using a graphical user interface (GUI) on the screen of the system. The system combines the focused image with the skeleton and verifies whether the current composition is good by estimating the Euclidean distance between the object on the screen and the specified two-dimensional point that may be located on the skeleton lines or intersections. After verifying the value, the system displays the combined image and instructions for taking a photo using the best composition. With the permission of the user, the system can adjust the current composition of the current focused image to the best composition. The details of these processes are described below.

[0028] A first embodiment of the present invention is implemented in a camera application to automatically recommend and adjust photographic compositions on a mobile system. The mobile system may be a mobile device including a camera and a display. Figure 1 An example of user operation provided by the first embodiment is shown. First, the user 103 touches the touch display screen 101 on the system 100 to start Figure 4 In the main process, the user 103 should keep the camera 102 facing the surrounding of one or more objects 104. For example, the user moves the camera 102 around the object 104, or the user moves with the camera 102 so that the camera 102 faces the object 104 in different directions.

[0029] Then, the object 104 is displayed on the touch display screen 101, and the user 103 can see the displayed object 105 and operate a graphical user interface (GUI) on the touch display screen 101 to take a photo when necessary. In one embodiment, an object diagram of a designated target object is displayed on a focused image where the user 103 intends to take a photo. According to the designated target object, a skeleton and instructions to the user 103 are displayed on the focused image, and the focused image is adjusted manually or automatically.

[0030] Figure 21 is a block diagram of the hardware configuration of the mobile system 100 provided in the first embodiment. The mobile system 100 includes a touch display screen 101, a camera 102, a central processing unit (CPU) 200, a data path 201, a random access memory (RAM) 202, a read only memory (ROM) 203 and a storage system 204.

[0031] The touch display screen 101 receives a touch operation performed by the user 103, and displays a GUI and a picture captured by the camera 102 or stored in the storage system 204. The camera 102 may include a CMOS image sensor, a depth sensor, or other sensors, and captures images or videos. The format of these images or videos is a color image or a depth image. The depth sensor may be a time of flight (ToF) sensor, a structured light 3D scanner, etc. A depth image is an image that visualizes a depth value, and the depth value is defined as the distance to an object.

[0032] The CPU 200 controls each component connected via the data path 201. The data path 201 includes an input / output interface or bus to transfer data to each component. The RAM 202 is used as the main memory of the CPU 200. The ROM 203 stores an operating system (OS), programs, system drivers, etc. The storage system 204 stores most of the data of the mobile system 100 and can be a solid-state drive (SSD), a hard disk drive (HDD), or other disk drives. A camera application that recommends and adjusts the photographic composition can be stored in the storage system 204.

[0033] Figure 3 is a block diagram of the functional configuration of the mobile system 100 provided in the first embodiment. Figure 2 The mobile system 100 includes a user interface control unit 300, a camera control unit 301, a scene reconstruction unit 302, an object selection unit 303, a composition recommendation unit 304, a composition adjustment unit 305, and a storage unit 306.

[0034] The user interface control unit 300 receives user input or operation on the touch display screen 101, such as touch, slide, etc. The user interface control unit 300 displays a GUI on the touch display screen 101 according to the user input and the status of other units.

[0035] The camera control unit 301 controls the camera 102 in the mobile system 100. The camera control unit 301 captures one or more images and stores the images in the storage unit 306. These images can be a set of static independent images or time-sequential images. A variety of formats of images can be used, such as color and depth. The camera control unit 301 also controls the functions of the camera 102, such as zoom, angle adjustment, focus correction, and other affine transformations to change the view of the image.

[0036] The scene reconstruction unit 302 reconstructs three-dimensional (3D) scene information (reconstructed scene) according to the images captured by the camera control unit 301. The scene information may be a set of color images, a depth image or a 3D geometry (represented as a set of polygonal meshes, point clouds or implicit functions).

[0037] The object selection unit 303 generates an object map based on the captured image and the 3D scene information to specify the target object in the focused image. The focused image refers to the image that the user 103 intends to take a photo of, that is, the image that the camera 102 points to when the camera control unit 301 detects a user operation related to taking a photo, for example, when the camera 102 is directed to a certain direction within a predefined duration, when the user performs a half-press operation on the shutter button.

[0038] The composition recommending unit 304 generates a skeleton and an instruction as a recommendation for focusing an image.

[0039] When the user accepts the automatic composition adjustment, the composition adjustment unit 305 generates a control signal to adjust the camera 102 based on the skeleton and the instruction, and sends the control signal to the camera control unit 301 .

[0040] The storage unit 306 reads and writes images, predefined skeletons, models, and parameters used in other units.

[0041] Figure 4 is an overall flow chart provided by the first embodiment, Figure 5 Shown in Figure 4 An example of an image to be processed in the steps of .

[0042] In step 400, the user 103 should be photographing an object 500. The user 103 activates the camera function in the mobile system 100 and keeps the camera in the mobile system 100 facing the surroundings of one or more objects 104, for example, the user 103 moves the camera 102 around the object 104, or the user 103 moves with the camera 102 so that the camera 102 faces the object 104 in different directions; the camera control unit 301 uses the camera 102 to capture multiple temporary images 501. In the first embodiment, the temporary images 501 are multiple pairs of color images and depth images in a temporary sequence, but only color images, only depth images, or other types of images such as infrared images can be used as temporary images. When the user 103 ends the camera application through the user interface control unit 301, the temporary images 501 are deleted.

[0043] In step 401, the scene reconstruction unit 302 extracts a reconstructed scene 502 from the temporary image 501. In the first embodiment, the reconstructed scene 502 may be a 3D mesh structure.

[0044] The 3D mesh reconstruction based on the temporary image 501 can be implemented using well-known methods such as Structure-from-Motion (SfM), Multi-View Stereo (MVS) or KinectFusion. Scene reconstruction can be completed using CPU 200 or other hardware processors such as a Graphical Processor Unit (GPU) or any other hardware specified for matrix calculations (such as convolution). The format of the reconstructed scene 502 is not limited to a grid. For example, other 3D geometric formats (such as point clouds or implicit functions) can be used as the reconstructed scene. In addition, 2D data such as a set of color images or 2.5D data such as a depth image can be used as the reconstructed scene.

[0045] In step 402, the camera control unit 301 detects a user operation related to taking a photo, such as an operation of directing the camera 102 to a certain direction within a predefined duration, an operation of the user half-pressing the shutter button, etc. The camera control unit 301 captures a focused image 503 from the camera 102. The focused image 503 is used to display the skeleton and instructions.

[0046] In step 403, the object selection unit 303 generates an object map 504 to specify an object region (region of a target object) in the focused image 503. Step 403 will be described in detail later.

[0047] In step 404, the composition recommendation unit 304 generates a skeleton 508 and instructions 510 as a recommendation for the focused image to improve the composition quality of the object area. Then, the composition recommendation unit 304 displays an output image 509 on the touch display screen 101. Step 404 is described in detail later.

[0048] In step 405, the composition adjustment unit 305 requests the user 103 whether to allow automatic composition adjustment. If the user accepts the automatic adjustment, jump to step 406. Otherwise, cancel the automatic adjustment. The user 103 can manually adjust the composition. The user 103 should follow the instructions on the skeleton and touch display screen 101 and move the camera 102 to generate the adjusted image 511 by himself.

[0049] In step 406, the composition adjustment unit 305 automatically adjusts the composition of the focused image to generate an adjusted image 511. Step 406 will be described in detail later.

[0050] Next, combine Figure 5 and Figure 6 The object selection in the above step 403 is described in detail. Figure 6 It is a detailed flowchart of object selection.

[0051] In step 600 , the object selection unit 303 obtains the reconstructed scene 502 extracted in step 401 from the scene reconstruction unit 302 .

[0052] In step 601, the object selection unit 303 generates an object map 504 to specify possible object regions in the space of the focused image 503 through the reconstructed scene 502. In the first embodiment, the object map 504 is a visual saliency map, but other types of images can also be used as object maps, such as semantic segmentation images, instance segmentation images, etc. The generation of the object map can be implemented using well-known methods. For example, a method based on machine learning, such as a convolutional neural network (CNN) or a method based on image and geometry processing can be used. The object map 504 may include objects with low reliability, which are not important. In order to improve the accuracy of the object map 504, it is important to add the reconstructed scene 502 as an input for estimating the object map 504. If it is difficult to segment the image or it is difficult to obtain the pixel intensity of the image, the object map 504 generated only from the color image may include incorrect object candidate regions. For example, this may be the case when taking a photo when the scene is dark or crowded. When the format of the scene information is a geometric body represented as a set of polygonal meshes, the object selection unit 306 only uses the geometric bodies in the three-dimensional area that can be projected from the focused image 503.

[0053] In step 602 , the object selection unit 303 restricts the object graph 504 to remove objects with low reliability and retain automatically selected objects 505 that do not reflect the user's preference.

[0054] In step 603, the object selection unit 303 checks whether there is any user input specifying the user's preference for the object in the focused image 503. The user input may be represented as a touch position 507 of the focused image 503. If there is any user input, the process jumps to step 604. Otherwise, the object selection 402 ends and jumps to step 403.

[0055] In step 604, the object selection unit 303 converts the touch position 507 to mask the automatically selected object 505. At this time, the masking is to keep only the object specified by the touch position 507. Finally, the object 508 selected by the user remains in the object map 504.

[0056] Combination Figure 5 , Figure 7 , Figure 8 and Fig. 9 The composition recommendation of the above step 404 is described in detail. Figure 7 4 is a detailed flowchart of the composition recommendation in step 404. Figure 8 An example of a predefined skeleton is shown. Fig. 9 Examples of instructions for user operations are described.

[0057] In step 700, the composition recommendation unit 304 estimates a skeleton 509 based on the object 508 selected by the user. For the estimation, the unit 304 extracts a feature vector from the selected object 508 using a statistical heuristic or a machine learning method such as a convolutional neural network. The output of the estimation can be a probability vector representing the possibility of a predefined skeleton. Each predefined skeleton includes a limited two-dimensional line, such as a rule-of-thirds 800, a quadrant composition 801, a center composition 802, a diagonal composition 803, and a golden composition 804. Multiple skeletons can be combined, as shown in 805. The storage unit 308 stores predefined skeletons encoded in a vectorized format, a run-length format, or other formats that can be easily decoded. All predefined skeletons are loaded from the storage unit 306, and the output skeleton 509 is generated based on the probability vector using a statistical method such as maximum likelihood estimation or weighted summation.

[0058] In step 701 , the composition recommending unit 304 combines the skeleton 509 with the focused image 503 as an output image 510 .

[0059] In step 702 , the composition recommendation unit 304 verifies whether the current composition is good by estimating the Euclidean distance between a point in a region of the selected object 508 having a distance greater than a predefined threshold and a specified point, line or intersection of the skeleton 509 .

[0060] In step 703, according to the verification in step 702, the composition recommendation unit 304 provides instructions 511 for taking a good photo, which can be operated through the GUI on the touch display screen 101, such as zooming, angle adjustment, focus correction and other affine transformations. These instructions can be no instructions, text instructions 901 for displaying text messages 904, symbol instructions 902 for displaying symbols 905, and voice instructions 903 for emitting voice 906. The user 103 can choose whether to display the skeleton 509 and instructions 511 through the touch display screen 101. If the verification result in step 702 is that the current composition is good, the composition recommendation unit 304 does not provide instructions 511.

[0061] Next, combine Fig.10 and Fig.11 The composition adjustment in step 406 is described in detail. Fig.10 is a detailed flowchart of the composition adjustment in step 406 . Fig.11 An example of composition adjustment is shown.

[0062] In step 1000, the composition adjustment unit 305 verifies whether the current composition of the focused image 503 is good in the same manner as step 702. If the current composition is good, step 1000 ends. Otherwise, the process jumps to step 1001.

[0063] In step 1001, the composition adjustment unit 305 automatically improves the composition quality. First, it is assumed that the verification result in step 1000 is that the focused image 1100 before the composition adjustment does not have a good composition. Then, the composition adjustment unit 305 sends a signal to the camera control unit 301 to adjust the camera 102 according to the instruction 511 calculated by the composition recommendation unit 304 to improve the composition quality. The camera control unit 302 controls the camera 102 to make the composition better, and its operations include zooming, angle adjustment, focus correction, and white balance. Since the Euclidean distance between the point of the object and the designated point of the skeleton is small, the intermediate focused image 1101 in the composition adjustment has a better composition.

[0064] In step 1002, the composition adjustment unit 305 sends a signal to the user interface control unit 300 to display the intermediate focus image 1101. Then, the process jumps back to step 1000. In step 1002, the user 103 can confirm the intermediate focus image 1101. However, this step is not mandatory and can be skipped.

[0065] Iterate from step 1000 to step 1002 to repeatedly update the focused image 1101. The focused image 1102 after iteration finally has a good composition.

[0066] According to the embodiments of the present invention, the user can take good photos without additional knowledge of the composition, and with the user's permission, the current composition focused by the user can be adjusted and photos can be taken automatically.

[0067] To obtain scene information around the user, other sensors such as the Global Positioning System (GPS) or Light Detection and Ranging (LiDAR) can be used. Vision-based optimization methods (such as differentiable rendering) can be used to adjust the current composition.

[0068] The above disclosure is only an exemplary embodiment of the present invention, and is certainly not intended to limit the scope of protection of the present invention. It can be understood by those skilled in the art that all or part of the processes for implementing the above embodiments and equivalent modifications made according to the claims of the present invention should fall within the scope of the present invention.

Claims

1. A device for taking photos, characterized in that The device comprises: camera; monitor; Processor for: capturing, using the camera, a plurality of temporary images of one or more objects; extracting a reconstructed scene from the plurality of temporary images; capturing a focused image of a photo that a user intends to take using the camera; generating an object map in the focused image using the reconstructed scene; estimating a skeleton from an object selected from the object graph; combining the skeleton with the focused image; Verifying whether the composition of the focused image is good; Instructions are provided to the user to adjust the composition using the display.

2. The device according to claim 1, characterized in that The device also includes a speaker, and the processor is further configured to provide instructions to the user to adjust the composition using the speaker.

3. The device according to claim 1 or 2, characterized in that The processor is further configured to automatically adjust the composition if the user accepts the automatic adjustment.

4. The device according to any one of claims 1 to 3, characterized in that The temporary images are pairs of color images and depth images.

5. The device according to claim 4, characterized in that The reconstructed scene includes a polygonal mesh or a point cloud.

6. A method for taking a photo, characterized in that The method comprises: capturing multiple temporary images of one or more objects using a camera; extracting a reconstructed scene from the plurality of temporary images; capturing a focused image of a photo that a user intends to take using the camera; generating an object map in the focused image using the reconstructed scene; estimating a skeleton from an object selected from the object graph; combining the skeleton with the focused image; Verifying whether the composition of the focused image is good; Instructions are provided to the user to adjust the composition using the display.

7. A storage medium for storing a program, characterized in that: When the program is executed by a processor, the program causes the processor to perform the method according to claim 6.

Images