Control device, control method, and program

Abstract

The present art pertains to a control device, a control method, and a program for enabling efficient continuous capture of main subjects. This control device assesses main subjects separated from each other by a distance equal to or greater than a threshold distance to be different main subjects on the basis of the distance to each main subject included in a capture range, and performs control of continuous capture of each main subject assessed to be a different main subject. The present art can be applied to devices having a capture function, such as a digital still camera or a smartphone.

Description

[Technical Field]

[0001] The present technology relates to a control device, a control method, and a program, and in particular to a control device, a control method, and a program that enable efficient continuous shooting of a main subject. [Background technology]

[0002] Some devices with photography capabilities, such as digital still cameras and smartphones, are equipped with a focus bracketing function, which is a photography method that takes multiple consecutive shots while changing the focus position.

[0003] By using the focus bracketing function, the user can take multiple images, each focused on a different subject, with a single release operation. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-120949 Summary of the Invention [Problem to be solved by the invention]

[0005] Improvements in autofocus functionality have made it possible to perform focus bracketing photography by shifting the focus position (focal length) in small increments, but it is not efficient to take advantage of the high resolution of the focus position to focus on all distances.

[0006] Furthermore, even if you take multiple images by changing the focus position, it is meaningless if the main subject is not in focus.

[0007] The present technology has been made in view of such circumstances, and makes it possible to efficiently perform continuous shooting of a main subject. [Means for solving the problem]

[0008] The control device according to one aspect of the present technology is a main subject detection unit that generates a main subject likelihood map in which the pixel value of each pixel is the degree of main subject that indicates the degree of main subject likelihood; a main subject likelihood map that defines a main subject likelihood as a pixel value of each pixel; a main subject likelihood map that defines a main subject likelihood as a pixel value of each pixel; a main subject likelihood map that defines a main subject likelihood as a pixel value of each pixel; a main subject likelihood map that defines a main subject likelihood as a pixel value of each pixel; a main subject likelihood map that defines a main subject likelihood as a pixel value of each pixel; a main subject likelihood map that defines a main subject likelihood as a pixel value of each pixel; The image capturing apparatus includes a main subject determination unit that determines, based on the above, that the main subjects that are separated by a distance equal to or greater than a threshold value are different main subjects, and an imaging control unit that controls continuous imaging of the main subjects determined to be different main subjects.

[0009] In one aspect of the present technology, the distance to each main subject included in the shooting range is a main subject likelihood map is generated in which the degree of main subject likelihood representing the degree of main subject likelihood is set as the pixel value of each pixel; an area on the main subject likelihood map consisting of pixels where the degree of main subject likelihood is equal to or greater than a predetermined threshold is defined as a main subject area; the main subject area is divided into small areas of a predetermined size; distances corresponding to each of the small areas are obtained from the depth map; and distances for each of the small areas are calculated. Based on this, the main subjects that are separated by a distance equal to or greater than a threshold value are determined to be different main subjects, and continuous shooting of the main subjects determined to be different main subjects is controlled. [Brief explanation of the drawings]

[0010] [Figure 1] 1A and 1B are diagrams illustrating examples of subjects photographed by a photographing device. [Figure 2] FIG. 10 is a diagram illustrating an example of a main subject determination process. [Figure 3] 1A and 1B are diagrams illustrating an example of focus bracketing photography by an imaging device. [Figure 4] FIG. 2 is a block diagram illustrating an example of the hardware configuration of the imaging device. [Figure 5] FIG. 2 is a block diagram showing an example of a functional configuration of a control unit. [Figure 6] FIG. 1 is a diagram illustrating an example of a shooting scene. [Figure 7] FIG. 10 is a diagram illustrating an example of a main subject likelihood map. [Figure 8] 10 is a flowchart illustrating a photographing process of the photographing device. [Figure 9]9 is a flowchart illustrating the main subject determination process performed in step S3 of FIG. 8. [Figure 10] FIG. 2 is a block diagram illustrating an example of the hardware configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present technology will be described in the following order. 1. Focus bracketing 2. Configuration of the imaging device 3. Operation of the imaging device 4.Other

[0012] <1. Focus bracketing> FIG. 1 is a diagram showing an example of a subject photographed by an imaging device 1 according to an embodiment of the present technology.

[0013] The photographing device 1 is a device with a photographing function, such as a smartphone, digital still camera, or surveillance camera. The photographing device 1 has a focus bracketing function. Focus bracketing is a photographing method in which multiple photographs are taken consecutively while changing the focus position.

[0014] The image capturing device 1 also has a function of detecting a main subject, which is the primary subject, from among the subjects included in the capture range. For example, the image capturing device 1 detects the main subject based on an image captured during preview capture before focus bracketing is performed. Specific types of subjects, such as people and buildings, and subjects that are large in the image, are detected as main subjects. The detection of the main subject will be described later.

[0015] In the example of Fig. 1, of the subjects included in the shooting range of the image capture device 1, subjects #1-1, #1-2, #2, #3, #4-1 to #4-4 are detected as main subjects. Subjects other than those shown in Fig. 1 are also included in the shooting range of the image capture device 1. Subjects #1-1 and #1-2 are people, and subject #2 is a car. Subject #3 is a house, and subjects #4-1 to #4-4 are buildings.

[0016] Subjects #1-1 and #1-2 are located at approximately the same distance from the position of the image capture device 1. Subject #2 is located further away than subjects #1-1 and #1-2 from the position of the image capture device 1. Subject #3 is located further away than subject #2 from the position of the image capture device 1. Subjects #4-1 to #4-4 are located further away than subject #3 from the position of the image capture device 1.

[0017] Focus bracketing is performed with these subjects as the main subjects. Each focus bracketing session captures multiple images with each main subject in focus.

[0018] Here, focus bracketing by the imaging device 1 is performed by referencing depth information (distance in the depth direction) and obtaining images in a state where main subjects that are positioned close to each other are in focus in a single imaging operation.

[0019] That is, when multiple main subjects are close to each other, they are not photographed by focusing on each main subject, but rather by focusing on one of the main subjects. Before focus bracketing is performed, a main subject determination process is performed to determine whether the multiple main subjects should be photographed as different main subjects or as a single main subject.

[0020] FIG. 2 is a diagram illustrating an example of the main subject determination process.

[0021] For example, main subjects that are at a distance greater than a threshold are determined to be different main subjects, and when a certain main subject is focused on, main subjects that are within the depth of field are determined to be the same main subject.

[0022] 2, subjects #1-1 and #1-2 are located within the depth of field when the focus is on subject #1-1. The double-headed arrow A1 indicates the range of the depth of field when the focus is on subject #1-1.

[0023] At this time, the photographing device 1 determines that the subjects #1-1 and #1-2 are the same main subject.

[0024] Furthermore, subjects #4-1 to #4-4 are located within the depth of field when the focus is on subject #4-1, for example. The two-way arrow A2 indicates the range of the depth of field when the focus is on subject #4-1.

[0025] At this time, the photographing device 1 determines that the subjects #4-1 to #4-4 are the same main subject.

[0026] Subjects #2 and #3 are each located at a distance equal to or greater than the threshold distance from the other main subjects, and are therefore determined to be different main subjects.

[0027] Such a determination is made based on a depth map acquired by the image capture device 1. The depth map is map information that records depth information to each position of each subject included in the capture range as a pixel value for each pixel. The image capture device 1 is equipped with a distance sensor such as a ToF (Time-of-Flight) sensor that can measure the distance to each position of each subject.

[0028] FIG. 3 is a diagram showing an example of focus bracketing photography by the photography device 1.

[0029] When the main subject determination process is performed as described with reference to Fig. 2, focus bracketing is performed by sequentially focusing on the main subjects, starting with the closest, as indicated by the upward arrows A11 to A14. Four shots are taken in a series of focus bracketing shots.

[0030] The first shooting is performed with the focus on, for example, subject #1-1. Since subject #1-2 is located within the depth of field when the position of subject #1-1 is used as the reference, the image obtained by the first shooting is an image in which not only subject #1-1 but also subject #1-2 are in focus.

[0031] The second shooting is performed with the focus on subject #2. The image obtained by the second shooting is an image with the focus on subject #2.

[0032] The third shooting is performed with the focus on subject #3. The image obtained by the third shooting is an image with the focus on subject #3.

[0033] The fourth shooting is performed with the focus on, for example, subject #4-1. Since subjects #4-2 to #4-4 are located within the depth of field when the position of subject #4-1 is used as the reference, the image obtained by the fourth shooting is an image in which not only subject #4-1 but also subjects #4-2 to #4-4 are in focus.

[0034] As described above, focus bracketing with the photographing apparatus 1 is performed by simultaneously photographing main subjects that are within a close distance range, such as a distance that fits within the depth of field. This reduces the number of shots taken compared to when all of the main subjects #1-1, #1-2, #2, #3, #4-1 to #4-4 are individually focused and photographed consecutively, enabling efficient focus bracketing.

[0035] The series of operations of the photographing apparatus 1 that performs focus bracketing photographing as described above will be described later with reference to a flowchart.

[0036] <2. Configuration of imaging device> FIG. 4 is a block diagram showing an example of the hardware configuration of the image capturing device 1. As shown in FIG.

[0037] The photographing device 1 is configured by connecting a photographing unit 12, a microphone 13, a sensor 14, a display 15, an operation unit 16, a speaker 17, a storage unit 18, and a communication unit 19 to a control unit 11.

[0038] The control unit 11 is composed of a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The control unit 11 executes a predetermined program and controls the overall operation of the photographing device 1 in response to user operations. The photographing device 1 having the control unit 11 functions as a control device that controls photographing.

[0039] The photographing unit 12 is composed of a lens, an image sensor, etc., and performs photographing under the control of the control unit 11. The photographing unit 12 outputs image data obtained by photographing to the control unit 11.

[0040] The microphone 13 outputs audio data such as collected voice to the control unit 11.

[0041] The sensor 14 is configured by a ToF sensor etc. The sensor 14 measures the distance to each position of the subject included in the shooting range and outputs the sensor data to the control unit 11.

[0042] The display 15 is configured with an LCD or the like, and displays various information such as a menu screen and an image being captured under the control of the control unit 11.

[0043] The operation unit 16 is configured by operation buttons, a touch panel, etc. provided on the surface of the housing of the image capturing device 1. The operation unit 16 outputs to the control unit 11 information indicating the content of the operation by the user.

[0044] The speaker 17 outputs sound based on the audio signal supplied from the control unit 11 .

[0045] The storage unit 18 is configured by a flash memory or a memory card inserted into a card slot provided in the housing. The storage unit 18 stores various data such as image data supplied from the control unit 11.

[0046] The communication unit 19 performs wireless or wired communication with an external device. The communication unit 19 transmits various data such as image data supplied from the control unit 11 to a computer, a smartphone, or the like.

[0047] FIG. 5 is a block diagram showing an example of the functional configuration of the control unit 11.

[0048] 5, the control unit 11 is made up of a main subject detection unit 31, a depth map generation unit 32, a main subject determination unit 33, and an imaging control unit 34. Image data supplied from the imaging unit 12 is input to the main subject detection unit 31, and sensor data supplied from the sensor 14 is input to the depth map generation unit 32.

[0049] The main subject detection unit 31 detects a predetermined subject as the main subject from among the subjects included in the shooting range.

[0050] Here, a main subject detection method and a main subject likelihood map will be described with reference to FIGS.

[0051] FIG. 6 is a diagram showing an example of a scene photographed by the photographing device 1. In FIG.

[0052] In the example of Fig. 6, subjects #1-1, #1-2, #2, #3, and #4-1 to #4-4 are included within the shooting range of the image capturing device 1. In addition, subjects not shown in Fig. 1 etc. are also included within the shooting range.

[0053] Based on the image obtained by photographing such a scene, the main subject detection unit 31 generates a main subject likelihood map as shown in Fig. 7 as the detection result of the main subject. The main subject likelihood map is map information in which the pixel value of each pixel is the main subject likelihood, which is a value that represents the "likelihood of the main subject" of a subject within the photographed range. For example, a brightly colored pixel is a pixel in which the content depicted in that pixel has a high likelihood of being the main subject.

[0054] In the example of Figure 7, pixels that include subjects #1-1, #1-2, #2, #3, and #4-1 through #4-4 are detected as pixels with a high likelihood of being the main subject, while pixels that include other subjects are detected as pixels with a low likelihood of being the main subject.

[0055] Such a main subject likelihood map is generated using an inference model acquired by machine learning, for example. For example, an inference model for the main subject likelihood map is generated by performing machine learning using, as training data, a plurality of images that have been labeled with information indicating which subject is the main subject.

[0056] The main subject detection unit 31 inputs the image obtained by shooting into an inference model for a main subject likelihood map, and obtains a main subject likelihood map based on the output of the inference model. The main subject detection unit 31 detects a predetermined subject as a main subject based on the main subject likelihood map.

[0057] The main subject likelihood map may be generated by analyzing the image captured by the image capturing unit 12. For example, the image captured by the image capturing unit 12 is analyzed, and a specific type of subject or a subject that appears large is detected as the main subject.

[0058] Information on the main subject detected by the main subject detection unit 31 in the above manner is output to the main subject determination unit 33.

[0059] The depth map generator 32 generates a depth map based on sensor data that indicates the distance to each position of each subject included in the shooting range.

[0060] The depth map may be generated using the output of a ToF sensor or may be generated using AI (Artificial Intelligence). When the depth map is generated using AI, for example, an inference model is prepared in the depth map generation unit 32, which inputs an image captured by the image capture unit 12 and outputs the distance to each position of each subject included in the capture range. The depth map generation unit 32 inputs the image captured by the image capture unit 12 into the inference model generated by machine learning, and acquires the distance to each position of each subject.

[0061] When the image capturing unit 12 is configured with a stereo camera, the distance to each position of each subject may be acquired based on the images captured by the stereo camera. The depth map information generated by the depth map generating unit 32 is output to the main subject determining unit 33.

[0062] The main subject determination unit 33 performs the above-described main subject determination process based on the supplied main subject information and depth map information. Information indicating the determination result is output to the shooting control unit 34. Note that the determination of whether to photograph different main subjects or the same main subject may be performed by other methods. Methods for determining main subjects are disclosed in, for example, JP 2013-120949 A.

[0063] The shooting control unit 34 controls the shooting unit 12 to perform focus bracket shooting. That is, the shooting control unit 34 performs continuous shooting by focusing on each of the main subjects determined as different main subjects based on the information supplied from the main subject determination unit 33.

[0064] <3. Operation of the imaging device> The photographing process of the photographing device 1 will be described with reference to the flowchart of FIG.

[0065] In step S1, the main subject detection unit 31 detects a main subject based on a captured image.

[0066] In step S2, the depth map generator 32 generates a depth map based on sensor data representing the measurement results obtained by the ToF sensor.

[0067] The processing in steps S1 and S2 is the processing during preview shooting, and the processing from step S3 onwards is the processing during focus bracketing shooting.

[0068] In step S3, the main subject determination unit 33 performs a main subject determination process based on the information on the main subject and the information on the depth map. The main subject determination process will be described later with reference to the flowchart of FIG.

[0069] In step S4, the shooting control unit 34 determines, based on the determination result by the main subject determination unit 33, whether or not there are two or more main subjects at different distances.

[0070] If it is determined in step S4 that there are two or more main subjects at different distances, then in step S5, the shooting control unit 34 focuses on each of the main subjects at different distances and performs continuous shooting. Main subjects that are close to each other, such as those within the depth of field, are shot together.

[0071] On the other hand, if it is determined in step S4 that there are no two or more main subjects at different distances, then in step S6, the shooting control unit 34 controls shooting using normal autofocus processing, with one shooting session being performed with the focus on one main subject.

[0072] Through the above processing, the photographing device 1 can efficiently perform focus bracketing photographing of the main subject.

[0073] The main subject determination process performed in step S3 of FIG. 8 will be described with reference to the flowchart of FIG.

[0074] In step S11, the main subject determination unit 33 divides the main subject region, which is the region of the main subject on the main subject likelihood map, into small regions of a predetermined size. For example, a region consisting of pixels whose main subject likelihood is equal to or greater than a predetermined threshold is determined to be the main subject region, and such a main subject region is divided into small regions.

[0075] In step S12, the main subject determination unit 33 acquires depth information corresponding to each small region based on the depth map.

[0076] In step S13, the main subject determination unit 33 determines that main subjects with a depth difference are different main subjects. Main subjects that are separated from each other by a distance equal to or greater than a predetermined threshold value are determined to be different main subjects.

[0077] In step S14, the main subject determination unit 33 determines that, when a certain main subject is focused on, the main subject that is located within the depth of field is the same main subject.

[0078] Through the above processing, for example, subjects #1-1 and #2 in Fig. 1 are determined to be different main subjects. Also, subjects #1-1 and #1-2 are determined to be the same main subject. After that, the process returns to step S3 in Fig. 8, and subsequent processing is performed.

[0079] Through the above series of processes, the photographing device 1 can efficiently perform focus bracketing photographing of the main subject.

[0080] <4.Other> Although the case of performing focus bracketing based on the results of the main subject determination process has been described, the above-mentioned technology can also be applied to exposure bracketing, in which continuous shooting is performed while adjusting the exposure. That is, when continuous shooting is performed while adjusting the exposure according to the brightness of each main subject, main subjects that are within a close distance, such as a distance that fits within the depth of field, are all photographed together using the same exposure adjustment value.

[0081] The above-described focus bracketing in a device having camera functions may be performed under the control of an external device. In this case, the external device that controls the focus bracketing functions as a control device. The external control device has the same configuration as the control unit 11 described with reference to FIG. 5.

[0082] About the program The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware or a general-purpose personal computer.

[0083] FIG. 10 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0084] A CPU (Central Processing Unit) 51, a ROM (Read Only Memory) 52, and a RAM (Random Access Memory) 53 are interconnected by a bus 54.

[0085] An input / output interface 55 is also connected to the bus 54. An input unit 56, an output unit 57, a storage unit 58, a communication unit 59, and a drive 60 are connected to the input / output interface 55. The drive 60 drives removable media 61 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0086] In a computer configured as described above, the CPU 51 performs the above-described series of processes by, for example, loading a program stored in the storage unit 58 into the RAM 53 via the input / output interface 55 and the bus 54 and executing the program.

[0087] The program executed by the CPU 51 is installed in the storage unit 58, for example, by being recorded on a removable medium 61, or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting.

[0088] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0089] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0090] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0091] For example, this technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0092] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by multiple devices.

[0093] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0094] Configuration combination examples The present technology can also be configured as follows.

[0095] (1) a main subject determination unit that determines, based on the distance to each main subject included in the shooting range, that the main subjects that are separated by a distance equal to or greater than a threshold value are different main subjects; an imaging control unit that controls continuous imaging of each of the main subjects determined to be different main subjects; A control device comprising: (2) the main subject determination unit determines another main subject that falls within a depth of field when the main subject is focused on, as the same main subject; The photographing control unit performs photographing once for a plurality of the main subjects determined as the same main subject. The control device described in (1) above. (3) The camera further includes a depth map generating unit that generates a depth map that indicates the distance to each of the main subjects included in the shooting range. The control device according to (1) or (2). (4) The depth map generator generates the depth map based on information obtained from the ToF sensor. The control device described in (3) above. (5) The image processing device further includes a main subject detection unit that generates a main subject likelihood map in which the main subject likelihood representing the degree of main subject likelihood is set as the pixel value of each pixel. The control device according to (3) or (4). (6) The main subject determination unit divides the area of ​​the main subject on the main subject likelihood map into small areas of a predetermined size, and determines the main subject based on the distance for each of the small areas acquired from the depth map. The control device according to (5) above. (7) The main subject detection unit receives an image as an input and generates the main subject likelihood map using an inference model that outputs the main subject likelihood map. The control device according to (5) or (6). (8) The photographing control unit controls focus bracketing photographing for continuously photographing the main subjects by focusing on each of the main subjects. The control device according to any one of (1) to (7). (9) The photographing control unit controls exposure bracket photographing, which performs continuous photographing by adjusting the exposure of each of the main subjects. The control device according to any one of (1) to (7). (10) The control device Based on the distance to each main subject included in the photographing range, the main subjects that are separated by a distance equal to or greater than a threshold value are determined to be different main subjects; Controlling continuous photography of each of the main subjects determined to be different from one another Control method. (11) On the computer, Based on the distance to each main subject included in the photographing range, the main subjects that are separated by a distance equal to or greater than a threshold value are determined to be different main subjects; Controlling continuous photography of each of the main subjects determined to be different from one another A program for executing a process. [Explanation of symbols]

[0096] 1 Photographing device, 11 Control unit, 31 Main subject detection unit, 32 Depth map generation unit, 33 Main subject determination unit, 34 Photography control unit

Claims

1. a depth map generating unit that generates a depth map that indicates the distance to each main subject included in the shooting range; a main subject detection unit that generates a main subject likelihood map in which each pixel has a main subject likelihood representing the degree of the main subject likelihood as a pixel value; a main subject determination unit that defines an area on the main subject degree map consisting of pixels where the main subject degree is equal to or greater than a predetermined threshold as a main subject area, divides the main subject area into small areas of a predetermined size, acquires distances corresponding to each of the small areas from the depth map, and determines, based on the distances for each of the small areas, that main subjects whose distance between them is equal to or greater than a threshold are different main subjects; an imaging control unit that controls continuous imaging of each of the main subjects determined to be different main subjects; A control device comprising:

2. the main subject determination unit determines another main subject that falls within a depth of field when the main subject is focused on, as the same main subject; The photographing control unit performs photographing once with a plurality of the main subjects determined to be the same main subject. The control device according to claim 1 .

3. The depth map generator generates the depth map based on information obtained from the ToF sensor. The control device according to claim 1 .

4. The main subject detection unit receives an image as an input and generates the main subject likelihood map using an inference model that outputs the main subject likelihood map. The control device according to claim 1 .

5. The photographing control unit controls focus bracketing photographing for continuously photographing the main subjects by focusing on each of the main subjects. The control device according to claim 1 .

6. The photographing control unit controls exposure bracket photographing, which performs continuous photographing by adjusting the exposure of each of the main subjects. The control device according to claim 1 .

7. The control device A depth map is generated that shows the distance to each main subject within the shooting range. generating a main subject likelihood map in which the main subject likelihood representing the degree of the main subject likelihood is set as a pixel value for each pixel; an area on the main subject degree map consisting of pixels where the main subject degree is equal to or greater than a predetermined threshold is defined as a main subject area, the main subject area is divided into small areas of a predetermined size, a distance corresponding to each of the small areas is obtained from the depth map, and based on the distance for each of the small areas, the main subjects where the distance between the main subjects is equal to or greater than a threshold are determined to be different main subjects; Controlling continuous photography of each of the main subjects determined to be different from one another Control method.

8. On the computer, A depth map is generated that shows the distance to each main subject within the shooting range. generating a main subject likelihood map in which the main subject likelihood representing the degree of the main subject likelihood is set as a pixel value for each pixel; an area on the main subject degree map consisting of pixels where the main subject degree is equal to or greater than a predetermined threshold is defined as a main subject area, the main subject area is divided into small areas of a predetermined size, a distance corresponding to each of the small areas is obtained from the depth map, and based on the distance for each of the small areas, the main subjects where the distance between the main subjects is equal to or greater than a threshold are determined to be different main subjects; Controlling continuous photography of each of the main subjects determined to be different from one another A program for executing a process.

Images