Information processing device, information processing method, and program

Abstract

An information processing device includes a control unit that performs output image selection mode processing of acquiring a first captured image captured at a first field of view and a second captured image that is an image obtained by capturing a common subject simultaneously with the first captured image, the second captured image including the subject at the first field of view and being captured at a second field of view that is wider, and selecting an output image from a plurality of images including at least the first captured image, a crop image from the second captured image, and the second captured image on the basis of a positional relationship between a determination target in the subject and the first field of view or the second field of view.

Description

INFORMATION PROCESSING DEVICE, INFORMATION PROCESSING METHOD, AND PROGRAMCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of Japanese Priority Patent Application JP 2023-207210 filed on December 7, 2023, the entire contents of which are incorporated herein by reference.

[0002] The present technology relates to an information processing device, an information processing method, and a program, and particularly relates to processing of a captured image.

[0003] In recent years, an information processing device such as a smartphone or a tablet terminal is generally equipped with a camera function. PTL 1 below discloses a technique in a case where a camera application is activated and used as a camera in a smartphone.

[0004] WO2020 / 016966Summary

[0005] By the way, in general, in a case where image capturing is performed by a camera, it is difficult for a user to keep a moving subject within a frame field of view successfully. It is even more difficult when capturing an image with a telephoto lens. For example, even if the user wants to capture an image of a pet dog running around at telephoto as much as possible, it is difficult for the user to perform image capturing without frame-out at telephoto as much as possible. Similarly, in a case where an image of a child is captured in an athletic meet or the like, it is difficult to prevent the child from being framed out at telephoto from a distance. Furthermore, in a case where the user captures an image of a fast moving subject with a handheld smartphone, the difficulty increases.

[0006] Therefore, the present disclosure proposes a suitable function in a case where a user wants to capture an image of a moving subject or the like as beautiful as possible at telephoto.

[0007] An information processing device according to the present technology includes a control unit that performs output image selection mode processing of acquiring a first captured image captured at a first field of view and a second captured image that is an image obtained by capturing a common subject simultaneously with the first captured image, the second captured image including the subject at the first field of view and being captured at a second field of view that is wider, and selecting an output image from a plurality of images including at least the first captured image, a crop image from the second captured image, and the second captured image on the basis of a positional relationship between a determination target in the subject and the first field of view or the second field of view. The first captured image and the second captured image are, for example, a telephoto image and a wide-angle image, and are captured images simultaneously including the common subject. The information processing device selects an image to be output from such the first captured image and second captured image according to a relationship between the subject and the first field of view or the second field of view. The selection includes the first captured image, the crop image from the second captured image, and the second captured image.

[0008] Fig. 1 is an explanatory diagram of a first surface of a smartphone that is an information processing device according to an embodiment of the present technology.Fig. 2 is an explanatory diagram of a second surface of the smartphone according to the embodiment.Fig. 3 is an explanatory diagram of an imaging situation.Fig. 4 is an explanatory diagram of camera switching according to the embodiment.Fig. 5 is an explanatory diagram of a display example of a monitor image according to the embodiment.Figs. 6A to 6C are explanatory diagrams of crop processing according to the embodiment.Figs. 7A and 7B are explanatory diagrams of a display example at the time of crop processing according to the embodiment.Figs. 8A and 8B are explanatory diagrams of correction of a determination target according to the embodiment.Fig. 9 is an explanatory diagram of designation of a determination target according to the embodiment.Fig. 10 is a block diagram of the smartphone according to the embodiment.Fig. 11 is a block diagram of a functional configuration of a smartphone according to first and second embodiments.Fig. 12 is a flowchart of switching processing of an output image selection mode according to the first embodiment.Fig. 13 is a flowchart of a modification of the switching processing of the output image selection mode according to the first embodiment.Fig. 14 is a flowchart of output image selection mode processing according to the first embodiment.Fig. 15 is a flowchart of switching processing of an output image selection mode according to the second embodiment.Fig. 16 is a flowchart of a second mode in output image selection mode processing according to the second embodiment.Fig. 17 is a flowchart of a third mode in the output image selection mode processing according to the second embodiment.Fig. 18 is a block diagram of a functional configuration of a smartphone according to a third embodiment.Fig. 19 is a flowchart of output image selection mode processing according to the third embodiment.Figs. 20A and 20B are explanatory diagrams of a processing example according to a telephoto field of view of the embodiment.

[0009] Hereinafter, an embodiment will be described in the following order. <1. Outline of processing on captured image> <2. Device configuration> <3. First embodiment> <4. Second embodiment> <5. Third embodiment> <6. Other processing examples> <7. Summary and modifications>

[0010] Note that, in the present disclosure, an “image” refers to both a still image and a moving image. For example, image capturing includes both still image capturing and moving image capturing.

[0011] <1. Outline of processing on captured image> Hereinafter, the configuration and operation of an embodiment will be described using a smartphone 1 as an example of an information processing device. That is, the description will be given by exemplifying that the smartphone 1 activates a camera application to capture an image.

[0012] Note that the smartphone 1 is an example of the information processing device that realizes functions of the present disclosure, and various examples such as a tablet terminal, a personal computer, a wristwatch type terminal, and a head mount type terminal are assumed. Furthermore, although the smartphone 1 having a camera function will be described below, the information processing device may not have the camera function. For example, the information processing device may be an information processing device in which a captured image is input from a separate camera.

[0013] Fig. 1 illustrates a display example on a first surface 100 of a housing in the smartphone 1 according to the embodiment. In particular, this case is a display example in a case where the camera application is activated in the smartphone 1.

[0014] Almost the entire first surface 100 of the smartphone 1 is used as a display 2. A monitor region 20 is provided in a part of the display 2, and a monitor image (preview image) on a subject side is displayed thereon. Furthermore, a still image or moving image record button 21, a zoom magnification button 22, a menu button 23, and other various buttons and icons are displayed.

[0015] A user can perform an operation of capturing a still image or starting / ending recording of a moving image using the record button 21. Note that a shutter button 3 is provided on a side surface of the housing. The user can also capture a still image or the like by the shutter button 3.

[0016] Furthermore, the user can perform an operation of changing a zoom magnification by tapping or sliding the zoom magnification button 22. An operation of the zoom magnification also includes a lens switching operation. For example, by selecting the “×0.7” button, an ultra wide-angle lens 6U is selected. Furthermore, for example, the zoom magnification is changed by sliding a button of “×0.7”, “×3.5 ×5.2” or the like. In some cases, the lens is automatically switched in the middle of sliding.

[0017] Various operation menus are displayed by the menu button 23. The user can perform various settings through the operation menu.

[0018] Fig. 2 illustrates a side of a second surface 200 of the housing of the smartphone 1. The second surface 200 is a back surface of the smartphone 1, and a lens unit 5 is disposed on the second surface 200. The lens unit 5 includes three lenses 6 as an example. For example, the ultra wide-engle lens 6U, a wide-angle lens 6W, and a telephoto lens 6T are used. Furthermore, the lens unit 5 is provided with a time of flight (TOF) sensor 9 at a position between the wide-angle lens 6W and the telephoto lens 6T. As illustrated in the drawing, a light emission unit and a light reception unit of the TOF sensor 9 are exposed. Furthermore, a flash unit 8 and a photometry unit 7 that performs photometry of ambient light are provided adjacent to the lens unit 5.

[0019] In this case, the smartphone 1 is provided with three camera units (a telephoto camera unit 31T, a wide-angle camera unit 31W, and an ultra wide-angle camera unit 31U to be described later with reference to Figs. 11 and 18.) including the telelens 6T, the wide-angle lens 6W, and the ultra wide-angle lens 6U. These camera units are usually used by being switched, but in the present embodiment, processing of an output image selection mode is performed in which a plurality of camera units simultaneously capture images of a common subject and selectively output the images.

[0020] The processing of the output image selection mode by the smartphone 1 will be described with reference to Figs. 3 to 9. Fig. 3 illustrates a situation where the smartphone 1 is capturing an image of a dog running around. Note that, as the imaging, it is assumed that the user captures a moving image while aiming at the dog as a subject, or the user captures a still image by performing a shutter operation at a certain timing while aiming at the dog to include it in a monitor image (preview image). Here, a case where an output image is recorded by capturing a moving image will be described as an example, but a case where an output image is used as a monitor image at the time of capturing a still image is similar to the above case.

[0021] Here, it is assumed that the user wants to capture an image at telephoto as much as possible. Although Fig. 3 illustrates a telephoto field of view 50 by the telephoto lens 6T, it is difficult to maintain a subject moving around within the telephoto field of view 50. On the other hand, since a field of view 52 by the wide-anlge lens 6W is wider than the telephoto field of view 50, it is relatively easy to capture the moving subject. Note that the field of view 52 includes the telephoto field of view 50 and is in a wider field of view.

[0022] Therefore, in the output image selection mode, for example, imaging by the telephoto lens 6T (telephoto camera unit 31T) and imaging by the wide-angle lens 6W (wide-angle camera unit 31W) are simultaneously performed. In the case of the smartphone 1, since the plurality of camera units is mounted, the plurality of camera units has the same imaging direction, and simultaneously perform imaging on a common subject.

[0023] Furthermore, a determination target in the subject is set. For example, an entire body of the dog, a head (face) of the dog, and the like are set as the determination target. The determination target is set to a main subject that the user wants to capture an image at telephoto or a subject that is estimated to want to capture an image.

[0024] Then, on the basis of a positional relationship between the determination target and the telephoto field of view 50 (first field of view) or the wide-angle field of view 52 (second field of view), a telephoto image 55 captured on a side of the telephoto lens 6T and a wide-angle image 56 captured on a side of the wide-angle lens 6W are selected as output images. This is illustrated in Fig. 4.

[0025] In Fig. 4, the wide-angle image 56 at each of time points t1, t2, and t3 is illustrated in the upper part, and the telephoto image 55 is illustrated in the lower part. At time point t1, it is assumed that the face of the dog as the determination target is included in the telephoto field of view 50. In this case, both the telephoto image 55 and the wide-angle image 56 include the determination target. In this case, the smartphone 1 selects the telephoto image 55 as an output image as indicated by a thick frame.

[0026] Thereafter, it is assumed that the determination target falls outside the telephoto field of view 50 at time point t2. In this case, the smartphone 1 switches the output image to the wide-angle image 56. As a result, the determination target is included in the output image as much as possible. Moreover, thereafter, it is assumed that the determination target is included in the telephoto field of view 50 at time point t3. In this case, the smartphone 1 switches the output image to the telephoto field of view 50.

[0027] By sequentially selecting the telephoto image 55 and the wide-angle image 56 as output images in this manner, it is possible to record a moving image including the determination target in the wide-angle image 56 even in a case where the determination target is out of the telephoto field of view 50 while setting the telephoto image 55 including the determination target as much as possible.

[0028] Furthermore, the monitor region 20 of the first surface 100 of the smartphone 1 displays as illustrated in Fig. 5, for example. The monitor region 20 is divided to provide a main monitor region 20A and a sub monitor region 20B. An output image is displayed in the main monitor region 20A. Therefore, in the main monitor region 20A, the telephoto image 55 and the wide-angle image 56 are displayed while being switched. On the other hand, the wide-angle image 56 is typically displayed in the sub monitor region 20B. Furthermore, the telephoto field of view 50 is displayed in the wide-angle image 56.

[0029] In this way, the user can grasp the positional relationship between the determination target and the telephoto field of view 50 and the wide-angle field of view 52 (the entire sub monitor region 20B) by the sub monitor region 20B while confirming the output image by the main monitor region 20A. Furthermore, even when the telephoto image 55 is output, the user can capture an image while grasping the overall composition by the wide-angle image 56.

[0030] The switching between the telephoto image 55 and the wide-angle image 56 has been described above, but in the present embodiment, switching to a crop image is also further performed. The crop image is an image obtained by cutting out and enlarging a part of the wide-angle image 56. For example, a crop frame 51 having the same size as a size of the telephoto field of view 50 is sequentially set in the wide-angle image 56, and the crop image is generated by performing the crop processing.

[0031] For example, Fig. 6A illustrates the wide-angle image 56. Furthermore, the telephoto field of view 50 is illustrated in the wide-angle image 56. In this case, when the determination target is the face of the dog, since the determination target is out of the telephoto field of view 50, the wide-angle image 56 is set as the output image in the case of the example of Fig. 4. Here, as illustrated in Fig. 6B, in the wide-angle image 56, the crop frame 51 having the same size as the telephoto field of view 50 is set as a position of the determination target. Then, the crop frame 51 is cut out and enlarged, and a crop image 57 in Fig. 6C is generated. Such a crop image 57 can be selected as an output image.

[0032] Since the crop frame 51 has the same size as the telephoto field of view 50, the crop image 57 becomes an image having the same field of view as the telephoto image 55. Since the crop processing is performed by electronic enlargement processing, the crop image 57 is inferior in image quality to the telephoto image 55, but tracking imaging at telephoto can be performed in a pseudo manner by using the crop image 57, and an output image suitable for the user's desire to capture an image at telephoto as much as possible can be obtained.

[0033] For example, even if the determination target deviates from the telephoto field of view 50, if the crop image 57 is output as long as it is within the wide-angle field of view 52, an image with the telephoto field of view 50 can be set as the output image as much as possible. For example, the telephoto image 55 can be selected when the determination target is in the telephoto field of view 50, the crop image 57 can be selected when the determination target is outside the telephoto field of view 50 and inside the wide-angle field of view 52, and the wide-angle image 56 can be selected when the determination target is outside the wide-angle field of view 52. In this way, while an image with the telephoto field of view 50 (the telephoto image 55 or the crop image 57) is output as much as possible, the wide-angle image 56 can make it easier to make the determination target enter an imaging range again when this is not possible.

[0034] Fig. 7A illustrates a display example of the monitor region 20 in a case where the crop processing is not performed. An output image is displayed in the main monitor region 20A, and here, a state in which the output image is the wide-angle image 56 is illustrated. As can be seen from the display in the sub monitor region 20B, this is because the face of the dog as the determination target is out of the telephoto field of view 50.

[0035] On the other hand, Fig. 7B illustrates a case where the crop processing is enabled. The sub monitor region 20B is similar to that in Fig. 7A. However, by performing the crop processing, the crop image 57 can be output, and the crop image 57 having the same field of view as the telephoto field of view 50 is displayed in the main monitor region 20A.

[0036] By the way, in the smartphone 1, in order to switch the output image in the output image selection mode as described above, whether or not the determination target is within the telephoto field of view 50 is determined from the wide-angle image 56. In order to improve the accuracy of this determination, it is preferable to perform the determination in a state after correcting an optical axis deviation due to a difference in the coordinate system of the image sensors on the side of the telephoto lens 6T and the side of the wide-angle lens 6W or the parallax due to the mounting accuracy.

[0037] A subject to be selected as the determination target can be automatically selected by a so-called priority face selection function or an object tracking function by image analysis. Figs. 8A and 8B illustrate the case of the telephoto image 55 and the case of the wide-angle image 56 as the output images, but it is assumed that a face frame 58 is set by image analysis on the wide-angle image 56. In the telephoto image 55 of Fig. 8A, the face frame 58 is slightly shifted, but this is the face frame detected by the wide-angle image 56, and is affected by an optical axis deviation and the like between the side of the telephoto lens 6T and the side of the wide-angle lens 6W. Since this difference is a fixed difference in the smartphone 1, it can be corrected. Therefore, correction is performed as in a face frame 58c indicated by a broken line.

[0038] Then, in the case that the telephoto image 55 is used as the output image as illustrated in Fig. 8A, when a certain portion of the corrected face frame 58c goes out from the telephoto field of view 50, the output image is switched to the wide-angle image 56 (or the crop image 57) as illustrated in Fig. 8B. The certain portion means that the certain portion is not the entire face frame 58c, and is, for example, a predetermined percentage of an area of the face frame 58c. That is, when a part of the determination target is missing in the telephoto image 55, switching to the wide-angle image 56 (or the crop image 57) is performed.

[0039] Furthermore, in a case where the wide-angle image 56 is output as illustrated in Fig. 8B (including a case where the crop image 57 is output although not illustrated), the face frame 58 set in the wide-angle image 56 is similarly corrected, and in a case where the entire corrected face frame 58c enters the telephoto field of view 50, the output image is switched to the telephoto image 55. That is, when the entire determination target can be captured in the telephoto image 55, switching to the telephoto image 55 is performed.

[0040] As described above, the output image selection is determined so that the appropriate telephoto image 55 can be obtained by eliminating the influence of the optical axis deviation and the like.

[0041] Furthermore, in order to reduce flickering of a screen due to frequent switching of the output image, it is preferable to make a gap in the switching condition between the telephoto image 55 and the wide-angle image 56 and perform hysteresis determination. For example, it is conceivable to provide a small entry detection frame and a large exit detection frame with respect to the telephoto field of view 50. Furthermore, as a temporal condition, at the time of outputting the telephoto image 55, switching to the wide-angle image 56 is considered when a certain portion of the face frame 58c continues to be outside the telephoto field of view 50 for a predetermined time or more, and at the time of outputting the wide-angle image 56, switching to the telephoto image 55 is considered when the entire portion of the face frame 58c continues to be within the telephoto field of view 50 for a predetermined time or more.

[0042] By the way, it has been described that the determination target is automatically set according to the face recognition of the subject, but there is a case where a plurality of faces is recognized and a plurality of the face frames 58 are set as illustrated in Fig. 9. In such a case, it is conceivable that one face frame 58 to be determined is automatically selected according to a predetermined priority or selected according to a user operation. For example, when the user touches one of the face frames 58 displayed as illustrated in Fig. 9, the touched face frame 58 is set as a determination target. Alternatively, the face frame 58 having the highest priority may be automatically set as the determination target according to a positional priority condition such as giving priority to a center of the image or a size condition such as giving priority to a size of the face frame.

[0043] Although the outline of the processing of the output image selection mode has been described above, the function of capturing an image at telephoto as much as possible only with the smartphone 1 is realized by the processing of the output image selection mode.

[0044] <2. Device configuration> Fig. 10 illustrates a configuration example of the smartphone 1 that performs the processing of the output image selection mode as described above. The smartphone 1 includes a control unit 30, a telephoto camera unit 31T, a wide-angle camera unit 31W, an ultra wide-angle camera unit 31U, a display unit 32, an operation unit 33, a sensor unit 34, a distance measurement unit 35, a storage control unit 36, a memory unit 37, and a communication unit 38. Note that the configuration of Fig. 10 is merely an example illustrating the main part, and a configuration (not illustrated) may be included, and it is not necessary to necessarily include all the configurations illustrated. For example, there may be a configuration that does not include the ultra wide-angle camera unit 31U.

[0045] As illustrated in Fig. 11, the telephoto camera unit 31T includes a telephoto lens unit 40T constituting an optical system as the telephoto lens 6T and an image sensor 41T. Furthermore, similarly, the wide-angle camera unit 31W includes a wide-angle lens unit 40W constituting an optical system as the wide-angle lens 6W and an image sensor 41W. As illustrated in Fig. 18, similarly, the ultra wide-angle camera unit 31U includes an ultra wide-angle lens unit 40U constituting an optical system as the ultra wide-angle lens 6U and an image sensor 41U. The image sensors 41T, 41W, and 41U include a light reception element, a gain processing circuit of a signal received by the light reception element and photoelectrically converted, and an A / D conversion circuit that converts the signal into digital data.

[0046] The light reception elements in the image sensors 41T, 41W, and 41U are configured as, for example, charge coupled device (CCD) type, complementary metal oxide semiconductor (CMOS) type, or the like. The image sensors 41T, 41W, and 41U perform, for example, correlated double sampling (CDS) processing, automatic gain control (AGC) processing, and the like on an electrical signal obtained by photoelectrically converting received light, and further perform analog / digital (A / D) conversion processing. Then, the captured image signal as digital data is output to the control unit 30.

[0047] In Fig. 10, the display unit 32 includes the above-described display 2 and its display drive circuit, and performs various displays for the user. The display 2 includes a liquid crystal display (LCD) panel, an organic electro-luminescence (EL) panel, or the like.

[0048] The display unit 32 executes various displays on a display screen on the basis of an instruction from the control unit 30. For example, the display unit 32 displays a monitor image at the time of imaging, various icons and operators as a user interface, and the like. That is, display as a graphical user interface (GUI) is executed on the screen.

[0049] The operation unit 33 collectively indicates input devices for the user to perform various operation inputs. Specifically, it is a physical operator such as the shutter button 3, a touch panel that is provided on the display 2 and detects an operation on an operator (buttons, icons, sliders, or the like) displayed on the display 2, or the like. The operation of the user is detected by the operation unit 33, and a signal according to the input operation is transmitted to the control unit 30.

[0050] The sensor unit 34 indicates various sensors mounted on the smartphone 1. For example, an inertial measurement unit (IMU) is mounted as the sensor unit 34. For example, an angular velocity (gyro) sensor of three axes of pitch, yaw, and roll can detect an angular velocity, and an acceleration sensor can detect an acceleration. Furthermore, as the sensor unit 34, an illuminance sensor constituting the photometry unit 7 is mounted. Various information detected by the sensor unit 34, for example, illuminance information, IMU data, and the like are supplied to the control unit 30.

[0051] The distance measurement unit 35 includes the light emission unit and the light reception unit that constitute the TOF sensor 9, and a calculation control unit that controls light emission and performs calculation based on a light reception signal, and performs processing of calculating a distance (depth) to a subject. Information of the calculated distance is supplied to the control unit 30.

[0052] The storage control unit 36 performs processing of recording an image file such as still image data or moving image data, attribute information of the image file, a thumbnail image, or the like on a recording medium such as a nonvolatile memory. Various actual aspects of the storage control unit 36 can be considered. For example, the storage control unit 36 may be in an aspect as a circuit unit that performs recording / reproducing on a flash memory built in the smartphone 1, or may be in an aspect as a card recording / reproducing circuit that performs recording / reproducing access to a memory card (for example, a portable flash memory) that can be attached to and detached from the smartphone 1. Furthermore, as an aspect built in the smartphone 1, the storage control unit 36 may be realized as a hard disk drive (HDD), a solid state drive (SSD), or the like.

[0053] The communication unit 38 comprehensively indicates a device that performs communication with an external device. For example, components that perform network communication, mobile phone communication, wired communication, near field wireless communication, and the like are indicated.

[0054] The control unit 30 includes a microcomputer including a central processing unit (CPU). The memory unit 37 stores information and the like used for processing by the control unit 30. The memory unit 37 includes, for example, a read only memory (ROM), a random access memory (RAM), a flash memory, and the like. The memory unit 37 may be a memory area built in a microcomputer chip as the control unit 30 or may be configured by a separate memory chip.

[0055] The RAM in the memory unit 37 is used for temporary storage of data, programs, and the like as a work area at the time of various data processing of the CPU of the control unit 30. The ROM and the flash memory (nonvolatile memory) in the memory unit 37 are used to store an operating system (OS) for the CPU to control each unit, an application program for various operations, various setting information, and the like.

[0056] The control unit 30 controls the entire smartphone 1 by executing a program stored in the ROM, the flash memory, or the like of the memory unit 37. For example, the control unit 30 controls zoom, focus, aperture adjustment, and the like with respect to the optical system of the telephoto camera unit 31T, the wide-angle camera unit 31W, and the ultra wide-angle camera unit 31U, controls a shutter speed with respect to the image sensor, and performs signal processing with respect to captured image data, an imaging operation and a recording operation according to a user's operation, a reproduction operation of a recorded image file, and the like.

[0057] Fig. 10 illustrates an imaging signal processing unit 30a, a user interface (UI) processing unit 30b, a selection control unit 30c, and a mode control unit 30d regarding functions provided by the camera application of the present embodiment in the control unit 30.

[0058] The imaging signal processing unit 30a performs processing on captured image data captured and input by each of the telephoto camera unit 31T, the wide-angle camera unit 31W, and the ultra wide-angle camera unit 31U, for example, RAW correction processing, Y / C development processing, resolution conversion processing, codec processing, file generation processing, and the like. In Figs. 11 and 18, imaging signal processing units 30aT, 30aW, and 30aU are illustrated corresponding to the telephoto camera unit 31T, the wide-angle camera unit 31W, and the ultra wide-angle camera unit 31U, respectively.

[0059] Image data as a moving image or a still image output through the processing by the imaging signal processing unit 30a (30aT, 30aW, 30aU) is stored in a storage medium by the storage control unit 36 or transmitted to an external device by the communication unit 38.

[0060] The UI processing unit 30b detects display control information of the display unit 32 and operation information by the operation unit 33, and executes processing according thereto. For example, control processing related to display of the monitor region 20, display of various icons and the like, display of the output image selection mode of Fig. 5, and the like on the display 2 as illustrated in Fig. 1 is performed.

[0061] Here, the selection control unit 30c is illustrated as a function of performing the processing of the output image selection mode. Furthermore, the mode control unit 30d is illustrated as a function of performing processing related to mode switching, such as on / off of the output image selection mode. For example, the selection control unit 30c has a function of performing processing in Figs. 14, 16, 17, and 19 to be described later, for example. Furthermore, the mode control unit 30d has a function of performing processing as illustrated in Figs. 12, 13, and 15, for example.

[0062] <3. First embodiment> Hereinafter, a processing example regarding the output image selection mode as the first embodiment will be described. In the first embodiment, in the output image selection mode, the smartphone 1 is an example in which the telephoto camera unit 31T and the wide-angle camera unit 31W simultaneously capture an image of a common subject, and in this case, an output image is selected.

[0063] Fig. 11 illustrates a functional configuration assuming an output image selection mode of the control unit 30. Note that Fig. 11 illustrates the telephoto camera unit 31T, the wide-angle camera unit 31W, and the control unit 30 extracted from the configuration of Fig. 10. The configurations of the telephoto camera unit 31T and the wide-angle camera unit 31W are as described above.

[0064] Regarding the control unit 30, the imaging signal processing unit 30a, the UI processing unit 30b, the selection control unit 30c, and the mode control unit 30d described above are illustrated, but the imaging signal processing unit 30a and the selection control unit 30c are illustrated in detail in relation to the output image selection mode.

[0065] The imaging signal processing unit 30a in the control unit 30 includes the imaging signal processing unit 30aT corresponding to the telephoto camera unit 31T, and the imaging signal processing unit 30aW corresponding to the wide-angle camera unit 31W. The processing of the imaging signal processing units 30aT and 30aW is as described above, but the imaging signal processing unit 30aW may perform the crop processing.

[0066] The selection control unit 30c that performs the output image selection mode processing in the control unit 30 includes a recognition / detection unit 43 and a camera selection unit 44.

[0067] The recognition / detection unit 43 receives the wide-angle image 56 from the wide-angle camera unit 31W, and performs setting of a determination target based on face recognition, a determination of a positional relationship between the determination target and the telephoto field of view 50 and the wide-angle field of view 52, setting of the crop frame 51, an output of a control signal CC including a crop instruction, an output of a control signal CS including an output selection instruction, and the like.

[0068] The camera selection unit 44 selects one of an image from the imaging signal processing unit 30aT and an image from the imaging signal processing unit 30aW as an output image Vout. As a result, the output image Vout becomes the telephoto image 55 from the imaging signal processing unit 30aT, the wide-angle field of view 52 from the imaging signal processing unit 30aW, or the crop image 57 from the imaging signal processing unit 30aW.

[0069] The telephoto image 55 from the imaging signal processing unit 30aT and the wide-angle image 56 and the crop image 57 from the imaging signal processing unit 30aW are supplied to the UI processing unit 30b and used for display in the monitor region 20. That is, the display of the main monitor region 20A, the sub monitor region 20B, and the like in Fig. 5 is executed.

[0070] The mode control unit 30d performs determination processing for mode switching, and the processing of the selection control unit 30c is executed in the case of the output image selection mode. For example, the mode control unit 30d performs mode switching determination on the basis of images from the imaging signal processing units 30aW and 30aT, information from the sensor unit 34, the distance measurement unit 35, the operation unit 33, and the like.

[0071] In this configuration, first, an example of the mode switching processing performed by the function of the mode control unit 30d will be described with reference to Fig. 12. Note that each flowchart described below illustrates an example of processing executed by the control unit 30 on the basis of a program for realizing the functions of Figs. 10 and 11. Furthermore, in the description of each flowchart, the same step numbers are assigned to the same processing as that in an above-mentioned flowchart, and redundant description is avoided.

[0072] Fig. 12 illustrates a processing example of automatically switching on / off of the output image selection mode. For example, it is assumed that the output image selection mode is initially turned off when the camera application is activated in the smartphone 1. While the camera application is activated, the control unit 30 repeats the processing of Fig. 12 and automatically turns on / off the output image selection mode.

[0073] Note that when the output image selection mode is off, the control unit 30 uses a camera unit selected according to the zoom operation or the like of the user. For example, the telephoto camera unit 31T, the wide-angle camera unit 31W, and the ultra wide-angle camera unit 31U are selected according to the operation of the zoom magnification button 22.

[0074] In step S101, the control unit 30 determines whether or not a moving subject is detected in the captured image. For example, when there is a subject that moves in different directions and has a large amount of movement per unit time with respect to the entire movement observed from a pixel differences of a plurality of feature points for each frame of the wide-angle image 56, the control unit 30 detects the subject as a moving subject.

[0075] Furthermore, assuming that the user is holding the smartphone 1 in his or her hand and capturing an image, the user should be moving a capturing direction of the smartphone 1 so as to follow the subject of interest. Then, in a case where a fast moving subject such as a dog or a child running around is targeted, the smartphone 1 is moved relatively intensely. Therefore, an absolute value of a detection value of an IMU, a gyro sensor, or the like built in as the sensor unit 34 of the smartphone 1 may be compared with a threshold, and it may be determined that there is a moving subject when the absolute value is greater than or equal to the threshold.

[0076] When the moving subject is not detected, the processing proceeds to step S103, and it is determined whether or not the off-condition of the output image selection mode is satisfied. However, this is the condition satisfaction determination for switching the output image selection mode from on to off, and when the output image selection mode is already off, the processing from step S103 to Fig. 12 ends without performing the satisfaction determination in particular. That is, the off state of the output image selection mode is continued.

[0077] In a case where a moving subject is detected on the basis of the image analysis or the sensor value, the control unit 30 determines in step S102 whether or not an on-condition for turning on the output image selection mode is satisfied. For example, as the ON condition, it is determined whether or not the motion amount of the moving subject has continued for a predetermined frame. For example, the number of frames corresponding to about 2 seconds to 3 seconds is set as a predetermined frame, and it is confirmed that the amount of motion is greater than or equal to a predetermined value continuously in a period of the predetermined frame. This is determination of a condition for turning on the output image selection mode.

[0078] Even if the motion amount greater than or equal to the predetermined value is detected, when it is determined that the ON condition is not satisfied at a time point less than the predetermined frame or the like, the control unit 30 ends the processing of Fig. 12 from step S102.

[0079] At a certain point in time, when it is detected in step S102 that the motion amount greater than or equal to the predetermined value continues for a predetermined frame period and it is determined that the on-condition is satisfied, the control unit 30 proceeds from step S102 to step S104.

[0080] In step S104, the control unit 30 turns on the output image selection mode. In this case, turning on the output image selection mode means turning on automatic camera switching in the camera selection unit 44 and enabling cropping. The enabling of the cropping means that the imaging signal processing unit 30aW is permitted to perform the cropping processing according to the control signal CC from the recognition / detection unit 43.

[0081] By turning on the output image selection mode in this manner, the processing of the output image selection mode to be described later with reference to Fig. 14 is executed.

[0082] After the output image selection mode is turned on, a moving subject may not be detected in step S101. In this case, the control unit 30 proceeds to step S103 and determines that the off-condition is satisfied. For example, a condition that the moving subject is not continuously detected for a certain period of time is set as the off condition. For example, it is assumed that a moving subject having a predetermined motion amount greater than or equal to the predetermined value continuously cannot be detected in the wide-angle image 56 in the number of frames corresponding to 5 seconds or the like. For example, in a case where the motion of the moving subject stops, or in a case where the moving subject does not return from the wide-angle image 56 in a frame-out state.

[0083] If such an off condition is not satisfied, the control unit 30 ends the processing of Fig. 12. That is, the on state of the output image selection mode is continued. On the other hand, when the off condition is satisfied, the control unit 30 proceeds from step S103 to step S105 and turns off the output image selection mode.

[0084] In the processing of Fig. 12 described above, on / off of the output image selection mode is automatically switched. Note that the output image selection mode is only turned on / off by a user operation, and the automatic switching as described above may not be performed. Furthermore, the output image selection mode is automatically turned on in the processing of steps S101, S102, and S104, but may be turned off by a user's operation. Conversely, after the output image selection mode is turned on by the user's operation, the output image selection mode may be automatically turned off in steps S101, S103, and S105. Furthermore, the output image selection mode may be set to be always ON or the output image selection mode may not be set to be ON by a setting operation of the user.

[0085] Fig. 13 illustrates processing as a modification of Fig. 12. This is a processing example in which the control unit 30 determines the timing at which the output image selection mode should be turned on and the timing at which the output image selection mode should be turned off, but does not execute automatic mode switching, and recommends mode switching to the user.

[0086] The processing in steps S101, S102, and S103 is similar to that in Fig. 12. In a case where a moving subject is detected in step S101 and it is determined that the on-condition is satisfied in step S102, the control unit 30 proceeds to step S111 and performs presentation for recommending turning on the output image selection mode to the user. For example, the control unit 30 performs processing of displaying a message recommending mode-on and buttons of “OK” and “cancel” as a pop-up dialog or the like in the monitor region 20. Then, in steps S113 and S114, the control unit 30 waits for the OK operation or the cancel operation by the user.

[0087] When detecting the user's OK operation, the control unit 30 proceeds from step S113 to step S115, and switches the mode, in this case, switches the output image selection mode to ON. When detecting the user's cancel operation, the control unit 30 ends the processing of Fig. 13 from step S114. That is, the off state of the output image selection mode is continued.

[0088] Furthermore, in a case where it is determined in step S103 that the off condition is satisfied, the control unit 30 proceeds to step S112 and performs presentation for recommending turning off the output image selection mode to the user. For example, the control unit 30 performs processing of displaying a message recommending mode-off and buttons of “OK” and “cancel” as a pop-up dialog or the like in the monitor region 20. Then, in steps S113 and S114, the control unit 30 waits for the OK operation or the cancel operation by the user.

[0089] When detecting the user's OK operation, the control unit 30 proceeds from step S113 to step S115, and switches the mode, in this case, switches the output image selection mode to OFF. When detecting the user's cancel operation, the control unit 30 ends the processing of Fig. 13 from step S114. That is, the on state of the output image selection mode is continued.

[0090] With such processing, it is possible to determine a situation suitable for the output image selection mode, present the situation to the user, and recommend the output image selection mode to be turned on. The user may perform an operation of turning on the output image selection mode if necessary. Furthermore, in a situation where the output image selection mode is no longer necessary, the off of the output image selection mode is recommended, and the user can determine by himself / herself whether or not to continue the mode on.

[0091] Note that, for example, it is not desirable that the same recommendation display is repeatedly performed many times after the user performs a cancel operation on the recommendation presentation. Therefore, in the determination of the satisfaction of the on-condition in step S102 and the satisfaction of the off-condition in step S103, for example, it is conceivable to add a condition that the cancel operation has not been performed for a predetermined time or more in the past.

[0092] Next, a processing example in a case where the output image selection mode is turned on will be described with reference to Fig. 14. That is, it is a processing example by the function of the selection control unit 30c in the control unit 30.

[0093] In a case where the output image selection mode is on, the control unit 30 repeatedly executes the processing of Fig. 14. In step S201, the control unit 30 determines whether or not a condition for determining that the determination target exists within the telephoto field of view 50 is satisfied.

[0094] The conditions in this case include, for example, the following conditions. (a) In the wide-angle image 56, the entire corrected face frame 58c in Fig. 8B is within the telephoto field of view 50. (b) The state of (a) described above continues for a predetermined number of frames, for example, about 1 second to 2 seconds.

[0095] According to the above-described (a) and (b), it can be determined that the range of the face frame 58c to be determined is stably included in the telephoto field of view 50 to some extent in the telephoto image 55. Therefore, for example, if the above-described (a) and (b) are satisfied, the control unit 30 proceeds to step S203 and selects an image captured by the telephoto camera unit 31T. Specifically, according to the control signal CS from the recognition / detection unit 43 in Fig. 11, the camera selection unit 44 selects the telephoto image 55 by the telephoto camera unit 31T as the output image Vout. Therefore, in the processing of step S203, the output image Vout is switched to the telephoto image 55. Alternatively, in a case where the image is already switched to the telephoto image 55, the state in which the telephoto image 55 is selected as the output image Vout is continued.

[0096] In a case where it has not been determined in step S201 that the condition is satisfied, the control unit 30 determines in step S202 whether or not a crop condition is satisfied.

[0097] The conditions in this case include, for example, the following conditions. (c) In the wide-angle image 56, a portion of the corrected face frame 58c in Fig. 8A at a certain ratio goes out of the telephoto field of view 50. (d) The entire face frame 58c is included in the wide-angle field of view 52. (e) A face frame 58c can be accommodated within the size of the crop frame 51.

[0098] The above-described (c) is determination of a situation where a part of the determination target is missing in the telephoto image 55. The above-described (d) is the determination that the portion of the face frame 58c to be determined can be taken out as the crop image 57 from the wide-angle image 56. The above-described (e) is the determination that the entire face frame 58c to be determined in the crop image 57 can be included in the crop image 57. This is because, for example, when the subject approaches and the face frame 58c becomes large in the image, the crop image 57 is not an image obtained by appropriately cutting out the determination target even if cropping is performed with the crop frame 51 having the same size as the telephoto field of view 50.

[0099] For example, when the above-described (c), (d), and (e) are satisfied, the control unit 30 proceeds to step S204, selects an image captured by the wide-angle camera unit 31W, and executes the crop processing. Specifically, the recognition / detection unit 43 in Fig. 11 causes the imaging signal processing unit 30aW to execute the crop processing of the crop frame 51 including the range of the face frame 58c by the control signal CC and output the crop processing. Furthermore, the recognition / detection unit 43 causes the camera selection unit 44 to select an image on a wide-angle camera unit 31W side by the control signal CS. Therefore, in the processing of step S204, the output image Vout is switched to the crop image 57. Alternatively, in a case where the image is already switched to the crop image 57, the state in which the crop image 57 is selected as the output image Vout is continued.

[0100] In a case where it has not been determined in step S202 that the condition is satisfied, the control unit 30 proceeds to step S205 and selects the image of the wide-angle camera unit 31W not subjected to the crop processing, that is, the wide-angle image 56 as the output image Vout. Specifically, the recognition / detection unit 43 in Fig. 11 causes the imaging signal processing unit 30aW to output the wide-angle image 56 without performing the crop processing by the control signal CC. Furthermore, the recognition / detection unit 43 causes the camera selection unit 44 to select an image on the side of the wide-angle camera unit 31W by the control signal CS. Therefore, in the processing of step S205, the output image Vout is switched to the wide-angle image 56. Alternatively, in a case where the wide-angle image 56 has already been switched, the state in which the wide-angle image 56 is selected as the output image Vout is continued.

[0101] In step S206, the control unit 30 determines the end of the output image selection mode. If the processing has not been completed, the processing returns to step S201. For example, when the output image selection mode is turned off by the above-described processing of Fig. 12 or 13, the control unit 30 determines that the mode is ended in step S206 of Fig. 14 and ends the processing of Fig. 14.

[0102] As described above, during the period in which the output image selection mode is turned on, processing of automatically switching the telephoto image 55, the crop image 57, and the wide-angle image 56 as the output image Vout is executed. Generally speaking, the output image Vout is switched as follows.

[0103] When the determination target is within the telephoto field of view 50, the telephoto image 55 is set as the output image Vout. When a part of the determination target comes out of the telephoto field of view 50, the crop image 57 is set as the output image Vout. In a case where a part of the determination target comes out of the wide-angle field of view 52 or becomes greater than or equal to the size of the crop frame 51, the wide-angle image 56 is set as the output image Vout.

[0104] Therefore, the telephoto image 55 and the wide-angle image 56 can be switched via the crop image 57.

[0105] On the other hand, after the telephoto image 55, switching to the crop image 57 is performed, but in the case of the wide-angle image 56, processing of switching to the telephoto image 55 without passing through the crop image 57 may be performed. For this purpose, for example, as a determination condition of step S202, (f) The previously selected image is the telephoto image 55. The following condition may be added.

[0106] In this way, after the output image Vout becomes the wide-angle image 56, the image is switched to the telephoto image 55 without passing through the crop image 57 having a relatively low image quality under the condition satisfaction of step S201. On the other hand, after the telephoto image 55 is obtained, the image with the telephoto field of view 50 can be continued as much as possible using the crop image 57.

[0107] <4. Second embodiment> As a second embodiment, a processing example of more finely dividing the mode in the output image selection mode will be described. Note that, similarly to the first embodiment, the second embodiment is an example in which the smartphone 1 simultaneously images a common subject by the telephoto camera unit 31T and the wide-angle camera unit 31W in the output image selection mode, and selects an output image in that case. Furthermore, the functional configuration for performing the processing of the output image selection mode is also similar to that in Fig. 11.

[0108] First, a mode switching processing example performed by the function of the mode control unit 30d will be described with reference to Fig. 15. In this example, the first mode to the fourth mode are set, but the first mode is a mode in which the output image selection mode is off. Then, the output image selection mode is turned on to provide a second mode, a third mode, and a fourth mode.

[0109] For example, it is assumed that the output image selection mode is initially turned off when the camera application is activated in the smartphone 1. While the camera application is activated, the control unit 30 repeats the processing of Fig. 15 and automatically turns on / off the output image selection mode. In this case, the first mode to the fourth mode are automatically switched.

[0110] Note that, in Fig. 15, an example in which the control unit 30 automatically switches the mode will be described. However, a modification as described in Fig. 13, that is, an example in which the mode switching is recommended to the user according to the result of the mode switching determination and the mode switching is executed according to the user operation is also conceivable.

[0111] In step S120 of Fig. 15, the control unit 30 compares the absolute value of the detection value of the IMU, the gyro sensor, or the like built in as the sensor unit 34 of the smartphone 1 with a threshold thJ, and determines that there is a moving subject when the absolute value exceeds the threshold thJ. This processing is similar to the moving subject determination in Fig. 12, and for example, the presence or absence of a moving subject may be determined from an image.

[0112] If the detected value is less than or equal to the threshold thJ, the control unit 30 proceeds to step S131 and sets the mode to the first mode. Alternatively, when the mode is already the first mode, the first mode is maintained. That is, the camera automatic switching operation is turned off, and the cropping is disabled. The turning off of the camera automatic switching operation means that the automatic switching of the camera selection unit 44 based on the control of the recognition / detection unit 43 in Fig. 11 is not performed. Furthermore, the disabling of the cropping means that the cropping processing is not performed in the imaging signal processing unit 30aW based on the control of the recognition / detection unit 43. That is, these mean that the output image selection mode is turned off. Note that normal camera switching is performed according to the user's zoom operation.

[0113] In a case where the detection value is greater than the threshold thJ in step S120, the control unit 30 proceeds to step S131, and the processing branches depending on whether the current imaging mode is moving image capturing or still image capturing. In the case of moving image capturing, the control unit 30 proceeds to step S132 and sets the mode to the second mode.

[0114] The second mode is a mode in which the camera automatic switching operation is turned off and the cropping is enabled. That is, as the output image selection mode, the camera selection unit 44 does not perform the camera switching, but the imaging signal processing unit 30aW performs the crop processing based on the control of the recognition / detection unit 43. Therefore, the wide-angle image 56 and the crop image 57 are selected as the output image selection mode.

[0115] This is an example in response to a request for not changing the image quality at the telephoto field of view 50 in the case of a moving image. When the crop frame 51 has the same size as the telephoto field of view 50, the telephoto image 55 and the crop image 57 are obtained as images of the telephoto field of view 50. However, when the image of the telephoto field of view 50 is switched between the telephoto image 55 and the crop image 57 in the middle of the moving image, there is a possibility that a sense of discomfort is caused due to a difference in image quality. In a case where it is desired to avoid such a situation, it is preferable to set the second mode in the case of moving image capturing.

[0116] In a case where the image capturing mode is still image capturing, the control unit 30 proceeds from step S121 to step S122. Then, the control unit 30 confirms whether the telephoto field of view is prioritized or the image quality is prioritized. For example, it is assumed that the user selects and sets in advance whether to prioritize imaging with the telephoto field of view 50 or to prioritize image quality maintenance in the output image selection mode.

[0117] In a case where the image quality priority is selected, the control unit 30 proceeds to step S133 and sets the third mode. The third mode is a mode for turning on the camera automatic switching operation and disabling the cropping. That is, as the output image selection mode, the camera of the camera selection unit 44 is switched based on the control of the recognition / detection unit 43, but the crop processing in the imaging signal processing unit 30aW is not performed. Therefore, the wide-angle image 56 and the telephoto image 55 are selected as the output image selection mode.

[0118] By not using the crop image 57 inferior in image quality as the output image Vout, high image quality is required. Note that, since this is the case of still image capturing, the output image Vout is used for display in the monitor region 20. Then, a frame of the output image Vout corresponding to the timing of the shutter operation is recorded as a still image. Therefore, in the case of the third mode, the crop image 57 is not recorded as still image data. This mode is suitable for a user who has recorded a still image with high image quality.

[0119] Note that the wide-angle image 56 is often selected by not performing the crop processing, but such a third mode is effective even in a situation where it is desired to capture an image while grasping the entire image including the background.

[0120] In a case where telephoto priority is selected, the control unit 30 proceeds from step S122 to step S134 to set the fourth mode. The fourth mode is a mode in which the camera automatic switching operation is turned on and the cropping is enabled. That is, as the output image selection mode, both the camera switching of the camera selection unit 44 based on the control of the recognition / detection unit 43 and the crop processing in the imaging signal processing unit 30aW are performed. Therefore, as the output image selection mode, the wide-angle image 56, the telephoto image 55, and the crop image 57 are selected.

[0121] The operation is similar to that of the first embodiment, but by using the telephoto image 55 and the crop image 57, a mode suitable for a user who wants to capture an image with the telephoto field of view 50 as much as possible is obtained.

[0122] Note that the above mode setting is an example. For example, the third mode or the fourth mode may be set even in the case of a moving image.

[0123] The output image selection mode processing in the case of the second mode, the third mode, and the fourth mode will be described.

[0124] Fig. 16 illustrates a processing example of the second mode. During the second mode, the control unit 30 repeatedly executes the processing of Fig. 16. In step S220, the control unit 30 fixes the camera selection to the side of the wide-angle camera unit 31W. For example, the camera selection unit 44 typically sets a side of the imaging signal processing unit 30aW as the output image Vout during the period of the second mode.

[0125] In step S221, the control unit 30 determines whether or not the crop condition is satisfied. The conditions in this case include, for example, the conditions (d) and (e) described above. Note that the condition (c) described above may not be adopted in this case. This is because it is not necessary to consider the telephoto field of view 50 since the telephoto image 55 is not used. That is, the size of crop frame 51 may be varied according to the determination target in the image at that time.

[0126] For example, when (d) and (e) described above are satisfied, the control unit 30 proceeds to step S222 and executes the crop processing. Specifically, the recognition / detection unit 43 in Fig. 11 causes the imaging signal processing unit 30aW to execute the crop processing of the crop frame 51 including the range of the face frame 58c by the control signal CC and output the crop processing. Therefore, the crop image 57 becomes the output image Vout.

[0127] In a case where it has not been determined in step S221 that the crop condition is satisfied, the control unit 30 proceeds to step S223 and stops the crop processing. Specifically, the recognition / detection unit 43 in Fig. 11 instructs the imaging signal processing unit 30aW to stop the crop processing by the control signal CC, and outputs the wide-angle image 56 as it is. Therefore, the wide-angle image 56 becomes the output image Vout.

[0128] In step S224, the control unit 30 determines the end of the second mode by the processing of Fig. 15. If the processing has not been completed, the processing returns to step S221. When the mode transitions to another mode by the processing of Fig. 15, the control unit 30 ends the processing of Fig. 16 from step S224.

[0129] Fig. 17 illustrates a processing example of the third mode. During the period of the third mode, the control unit 30 repeatedly executes the processing of Fig. 17. In step S230, the control unit 30 stops the crop processing. That is, during the period of the third mode, the imaging signal processing unit 30aW does not perform the crop processing.

[0130] In step S231, the control unit 30 determines whether or not a condition that the determination target is within the telephoto field of view 50 is satisfied. The conditions in this case include, for example, the conditions (a) and (b) described above.

[0131] For example, when (a) and (b) described above are satisfied, the control unit 30 proceeds to step S232 and selects the telephoto image 55 as the output image Vout. Specifically, the recognition / detection unit 43 in Fig. 11 causes the camera selection unit 44 to select the telephoto image 55 by the telephoto camera unit 31T by the control signal CS.

[0132] In a case where it has not been determined in step S231 that the condition is satisfied, the control unit 30 proceeds to step S233 and selects the wide-angle image 56 as the output image Vout. Specifically, the recognition / detection unit 43 causes the camera selection unit 44 to select the wide-angle image 56 by the wide-angle camera unit 31W by the control signal CS.

[0133] In step S234, the control unit 30 determines the end of the third mode by the processing of Fig. 15. If the processing has not been completed, the processing returns to step S231. When the mode transitions to another mode by the processing of Fig. 15, the control unit 30 ends the processing of Fig. 17 from step S234.

[0134] The processing example of the fourth mode is similar to that in Fig. 14.

[0135] <5. Third embodiment> As a third embodiment, an example in which captured images of the telephoto camera unit 31T, the wide-angle camera unit 31W, and the ultra wide-angle camera unit 31U are selected in the output image selection mode will be described. In the output image selection mode, the smartphone 1 simultaneously images a common subject by the telephoto camera unit 31T, the wide-angle camera unit 31W, and the ultra wide-angle camera unit 31U, and selects an output image in that case.

[0136] A functional configuration for performing the processing of the output image selection mode is illustrated in Fig. 18. In Fig. 18, the control unit 30 adds an imaging signal processing unit 30aU corresponding to the ultra wide-angle camera unit 31U to the configuration of Fig. 11.

[0137] The recognition / detection unit 43 can issue a crop instruction to either or both of the imaging signal processing units 30aW and 30aU by the control signal CC, for example. Furthermore, the camera selection unit 44 selects the images from the imaging signal processing units 30aT, 30aW, and 30aU as the output image Vout.

[0138] Fig. 19 illustrates a processing example of the output image selection mode. In step S201, the control unit 30 determines whether or not a condition that the determination target is within the telephoto field of view 50 is satisfied. For example, the above conditions (a) and (b) are satisfied. If the condition is satisfied, the control unit 30 proceeds to step S203 and selects the telephoto image 55 as the output image Vout.

[0139] In a case where it has not been determined in step S201 that the condition is satisfied, the control unit 30 proceeds to step S202A and determines whether or not the crop conditions in the wide-angle image 56, for example, (c), (d), and (e) described above are satisfied. Then, in a case where the condition is satisfied, the control unit 30 proceeds to step S204 and selects the crop image 57 from the wide-angle image 56 as the output image Vout.

[0140] In a case where it has not been determined in step S202A that the conditions are satisfied, the control unit 30 proceeds to step S207 and determines whether or not the crop condition in the ultra wide-angle image is satisfied. The conditions in this case include, for example, the following conditions. (g) The entire corrected face frame 58c in Fig. 8A is included in the ultra wide-angle image. (h) Face frame 58c can be accommodated within the size of crop frame 51. (i) The entire face frame 58c is not included in the wide-angle field of view 52.

[0141] Note that, since the above-described (i) is satisfied when the condition is not satisfied in step S202A, the above-described (g) and (h) are only required to be determined in step S207. Then, in a case where the condition is satisfied, the control unit 30 proceeds to step S208 and selects the crop image from the ultra wide-angle image as the output image Vout.

[0142] In a case where it has not been determined in step S207 that the condition is satisfied, the control unit 30 proceeds to step S209 and sets the ultra wide-angle image captured by the ultra wide-angle camera unit 31U as the output image Vout. That is, the captured image is the captured image of the ultra wide-angle camera unit 31U not subjected to the crop processing.

[0143] In step S206, the control unit 30 determines the end of the output image selection mode. If the processing has not been completed, the processing returns to step S201. When it is determined that the output image selection mode is ended, the control unit 30 ends the processing of Fig. 19 from step S206.

[0144] As described above, by also using the captured image of the ultra wide-angle camera unit 31U, an image having the same field of view as the telephoto field of view 50 can be obtained as the output image Vout as long as possible by the crop image from the ultra wide-angle image. Furthermore, even when the subject to be determined cannot be easily captured in the frame, the user finishes the subject to be determined while viewing the ultra wide-angle image, so that the ease of frame-in is improved.

[0145] <6. Other processing examples> Various processing examples as variations of the processing of the above embodiment will be described.

[0146] For the setting of the determination target, an example based on the automatic setting of the face frame 58 and an example based on the designation of the user with respect to the face frame have been described, but not limited to the face frame, a touch frame may be set to a subject at a position touched by the user, and the touch frame may be preferentially set as the determination target. Furthermore, in the initial stage, the face is automatically recognized and set as the determination target, but an example in which the touch frame is changed to the determination target when touched is also conceivable.

[0147] In a case where the crop image 57 and the telephoto image 55 are switched, it is desirable to minimize the difference in image quality between the telephoto image 55 and the crop image 57. This is because the user may feel uncomfortable due to the change in image quality.

[0148] For this purpose, it is desirable to use a method of improving the image quality of the crop image 57. As an example, there is an electronic zoom method using an image sensor having a configuration in which one color filter is provided for four pixels of 2×2, for example. In this case, without using the binning mode in which four pixels are regarded as one pixel, one pixel is treated as one pixel as it is, data in a Bayer array is generated by remosaic processing, and demosaic processing is performed to obtain a high-resolution image. By applying this technique to the range of the crop frame 51, it is possible to improve the image quality of the crop image 57 and make it difficult for the user to feel switching between the telephoto image 55 and the crop image 57.

[0149] Furthermore, whether the crop image 57 or the telephoto image 55 is being output may be displayed in the monitor region 20 or the like so that the user can recognize it.

[0150] As processing in the output image selection mode, exceptional processing may be performed in which the telephoto image 55 is set as the output image Vout in a case where the subject to be determined is not included in the telephoto field of view 50. For example, it can be used in a case where it is not desired to intentionally capture an arbitrary subject. For example, consider a case where it is desired to capture an image of only the entrance of the school, but it is difficult to capture the image due to the presence of people as illustrated in Fig. 20A. In this case, processing of switching the output image Vout to the telephoto image 55 when the detected person is set as the determination target and the determination target is outside the telephoto field of view 50 as illustrated in Fig. 20B is performed, so that it is suitable for a request to capture an image of only the entrance of the school at the telephoto field of view 50 without capturing a person.

[0151] There are various examples as examples of automatically ending the output image selection mode or urging the user to end the output image selection mode depending on an imaging situation or the like. For example, in a case where the tripod imaging mode is set, or in a case where the smartphone 1 is stationary for a long time, it can be determined that the scene is not a scene in which a moving subject is imaged. Therefore, termination processing of the output image selection mode or recommendation of mode termination to the user may be performed.

[0152] Furthermore, in a case where the user turns on the object tracking function or in a case where the user recognizes the face in the captured image, the user may be urged to use the output image selection mode. Furthermore, in a case where the face is not recognized or the object tracking is not operated, the output image selection mode may be ended.

[0153] Furthermore, the processing of the output image selection mode may be performed in a case where a gimbal mechanism is used or in a camera system that automatically tracks a target subject as a so-called pan-tilt-zoom (PTZ) camera.

[0154] <7. Summary and modifications> According to the above-described embodiments, the following effects can be obtained.

[0155] The smartphone 1, which is the information processing device of the embodiment, includes the control unit 30 that performs the output image selection mode processing of acquiring the telephoto image 55 (first image) captured at the telephoto field of view 50 (at the first field of view) and the wide-angle image 56 (second image) that is an image obtained by capturing a common subject simultaneously with the telephoto image 55, the wide-angle image 56 including the subject at the telephoto field of view 50 and being captured at the wide-angle field of view 52 (at the second field of view) that is wider, and selecting the output image Vout from a plurality of images including at least the telephoto image 55, the crop image 57 from the wide-angle image 56, and the wide-angle image 56 on the basis of a positional relationship between the determination target in the subject and the telephoto field of view 50 or the wide-angle field of view 52. By the processing of the control unit 30, the telephoto image 55, the crop image 57, and the wide-angle image 56 are selected depending on the positional relationship between the subject as the determination target and the telephoto field of view 50 or the wide-angle field of view 52, specifically, whether or not the determination target is within the telephoto field of view 50. As a result, automatic selection of an image according to the position of the subject in the captured image is realized. In particular, since the crop image 57 is included in the options, various image selections are realized.

[0156] Furthermore, the selection based on the positional relationship between the determination target and the telephoto field of view 50 or the wide-angle field of view 52 enables, for example, processing of selecting the telephoto image 55 when the determination target is within the telephoto field of view 50, or conversely, processing of selecting the telephoto image 55 when the determination target is outside the telephoto field of view 50 as illustrated in Fig. 20B.

[0157] Note that although the smartphone 1 has been described as an information processing device, a tablet device other than the smartphone 1, a personal computer, a cloud server, or the like can also constitute the information processing device that performs such output image selection mode processing. Furthermore, the information processing device may be an information processing device that does not incorporate a camera and inputs the telephoto image 55 and the wide-angle image 56 from an external camera.

[0158] In the embodiment, an example has been described in which the crop image 57 is an image obtained by cutting out an field of view range having the same size as the telephoto field of view 50 from the wide-angle image 56. As a result, the crop image 57 can be used as a substitute for the telephoto image 55 without discomfort in terms of the field of view. Therefore, even when the determination target is not included in the telephoto field of view 50, by selecting the crop image 57 in the same subject range as the telephoto image 55, it is possible to obtain the output image Vout that continues imaging at the telephoto field of view 50 as much as possible.

[0159] Note that, even if the crop image 57 is obtained by cutting out a field of view range having a size different from the telephoto field of view 50, it may be felt almost equal to the telephoto field of view depending on the size of the crop frame 51. Furthermore, conversely, image switching can be performed in which the telephoto field of view 50, the crop field of view, and the wide-angle field of view 52 are clearly distinguished into three types.

[0160] In the embodiment, the control unit 30 recognizes the determination target in the wide-angle image 56, and determines the positional relationship between the determination target and the telephoto field of view 50 or the wide-angle field of view 52 in the wide-angle image 56. The control unit 30 recognizes a determination target such as a face of a person or an animal in wide-angle image 56. By recognizing the determination target on a side of the wide-angle image 56, the determination target can be searched for in a wide subject range. Then, in the wide-angle image 56, whether the determination target is within the telephoto field of view 50, whether the determination target is not within the telephoto field of view 50 but within the wide-angle image 56, whether the determination target is out of frame from the wide-angle image 56, or the like is determined by satisfying a predetermined condition. This makes it possible to perform determination in a wide subject range.

[0161] Note that the positional relationship between the determination target and the telephoto field of view 50 and the like may be determined not only by the positional relationship but also by satisfaction determination of a temporal condition such as continuously being within the telephoto field of view 50 for a predetermined frame or more or hysteresis determination, as described in step S201 of Fig. 14, for example. As a result, it is possible to prevent the image from being switched unnecessarily frequently.

[0162] In the embodiment, a processing example has been described in which, in a case where the control unit 30 determines that the determination target is included in the telephoto field of view 50, the telephoto image 55 is selected as the output image Vout. As a result, the telephoto image 55 in the period in which the determination target is the subject by the telephoto camera unit 31T can be output.

[0163] In the embodiment, a processing example has been described in which, in a case where the control unit 30 determines that the determination target is included in the wide-angle field of view 52 and not included in the telephoto image 55, the crop image 57 is selected as output image Vout. As a result, even when the determination target is out of the subject by the telephoto camera unit 31T, the crop image 57 in the same image range as the telephoto image 55 can be output. For example, when the telephoto image 55 is output, even if the determination target deviates from the telephoto field of view 50, an image having the same zoom field of view can be continuously output as the crop image 57.

[0164] In the embodiment, by way of example, the control unit 30 selects the wide-angle image 56 in the case of determining that the determination target is not included in the wide-angle field of view 52. In a case where the determination target has framed out from the wide-angle image 56, the control unit 30 outputs the wide-angle image 56 so that the user can more easily follow the determination target. This makes it easy to return to the frame-in state.

[0165] In the embodiment, an example has been described in which the control unit 30 performs control such that the wide-angle image 56 is displayed on the display unit 32 (display 2) together with the selected output image Vout. As described with reference to Figs. 7A and 7B, in the output image selection mode, the sub monitor region 20B is provided in addition to the main monitor region 20A on the display screen so that the user can typically visually recognize the wide-angle image 56 in the sub monitor region 20B. As a result, the positional relationship between the subject as the determination target and the telephoto field of view 50 can be typically recognized, and the user can easily set the determination target to the telephoto field of view 50.

[0166] In the embodiment, an example has been described in which the control unit 30 performs control such that the telephoto field of view 50 is indicated on the display of the wide-angle image 56. As illustrated in Figs. 7A and 7B, telephoto field of view 50 is displayed in wide-angle image 56 in sub-monitor region 20B. As a result, the user can easily push the determination target into the telephoto field of view 50.

[0167] In the embodiment, an example has been described in which the control unit 30 performs the start determination of the output image selection mode processing on the basis of the imaging situation determination. For example, the control unit 30 automatically determines to turn on the output image selection mode by the processing in Figs. 12, 13, 15, and the like. As a result, it is possible to determine that the output image selection mode is a suitable imaging situation.

[0168] In the embodiment, an example has been described in which the control unit 30 automatically turns on the output image selection mode in accordance with the start determination of the output image selection mode processing. The control unit 30 performs processing of automatically turning on the output image selection mode according to the imaging situation by the processing of Figs. 12 and 15. As a result, the function of the output image selection mode can be exerted in an imaging situation in which the output image selection mode is suitable without requiring the operation burden of the user.

[0169] In the embodiment, an example has been described in which the control unit 30 performs presentation control of recommending the output image selection mode to the user according to the start determination of the output image selection mode processing. For example, as in the processing of Fig. 13, the control unit 30 performs control to perform presentation for recommending the output image selection mode to the user according to the imaging situation. As a result, it is possible to notify the user that the output image selection mode of the user is an appropriate imaging situation, and to turn on the output image selection mode according to intention to the user.

[0170] In the embodiment, the control unit 30 performs at least one of determination of the presence or absence of a moving subject, determination of the amount of movement of the camera (smartphone 1) capturing the telephoto image 55 or the wide-angle image 56, and another determination of a moving image and a still image as the imaging situation determination for the output image selection mode. That is, the control unit 30 determines the presence or absence of a moving subject as in the processing of Figs. 12 and 13, and performs another determination of the movement of the camera (smartphone 1) by the gyro sensor and the moving image / still image as in the processing of Fig. 15. With these elements, it is possible to appropriately determine whether or not the imaging situation is suitable for the function of the output image selection mode.

[0171] In the embodiment, an example has been described in which the control unit 30 performs control to switch the telephoto image 55, the crop image 57, and the wide-angle image 56 in the output image selection mode processing. For example, the control unit 30 selects the telephoto image 55, the crop image 57, and the wide-angle image 56 according to the situation as in the processing of Fig. 14. As a result, for example, an image of a zoom field of view using the telephoto image 55 and the crop image 57 can be output even for a fast moving subject.

[0172] In the embodiment, an example has been described in which the control unit 30 performs, as the output image selection mode processing, any of the processing of the fourth mode for switching the telephoto image 55, the crop image 57, and the wide-angle image 56, the processing of the second mode for switching the wide-angle image 56 and the crop image 57, and the processing of the third mode for switching the telephoto image 55 and the wide-angle image 56. For example, as described with reference to Figs. 15 to 17, the control unit 30 selects the processing of the second mode, the processing of the third mode, and the processing of the fourth mode as the output image selection mode processing according to the situation. As a result, the output image selection mode processing itself can be executed according to the situation.

[0173] Note that, in the output image selection mode, an example of typically performing the processing of the second mode and an example of typically performing the processing of the third mode are also conceivable. Furthermore, in the output image selection mode, an example of selectively performing the third mode and the fourth mode, an example of selectively performing the second mode and the fourth mode, an example of selectively performing the second mode and the third mode, and the like are also conceivable. Furthermore, although not described in the example of Fig. 15, processing of switching the telephoto image 55 and the crop image 57 can also be performed.

[0174] In the embodiment, an example has been described in which the determination target is a face of a person or an animal as a subject. The control unit 30 performs face recognition processing of a person or an animal in the wide-angle image 56, and sets the recognized face portion (face frame 58) as a determination target. As a result, the control unit 30 can automatically set the determination target and exert the function of the output image selection mode without an operation burden of the user. Note that, in a case where a plurality of faces is recognized, the face of the subject desired by the user can be set as the determination target by causing the user to designate the face. Furthermore, the determination target is not limited to the face, and may be a whole body of a person or an animal, a hand, a foot, or the like, or may be another object, for example, an automobile, a flight vehicle, a ship, or the like. In particular, a moving object is suitable as a determination target.

[0175] In the embodiment, an example has been described in which the control unit 30 corrects the position (face frame 58) of the determination target in the wide-angle image 56 according to the optical axis deviation between the camera that captures the telephoto image 55 and the camera that captures the wide-angle image 56 and then determines the positional relationship between the determination target and the telephoto field of view 50 or the wide-angle field of view 52 in the output image selection mode processing. The positional relationship between the determination target and the telephoto field of view 50 is determined by the wide-angle image 56 by the wide-angle camera unit 31W after correcting the optical axis deviation due to the lens attachment accuracy and the arrangement position interval in the telephoto camera unit 31T and the wide-angle camera unit 31W, whereby the state in which the determination target is within the telephoto field of view 50 by the telephoto camera unit 31T can be accurately determined.

[0176] In the embodiment, an example has been described in which the control unit 30 performs processing of making the selected output image Vout a record image, a transmission image, or a monitor display image on the display unit in the output image selection mode processing. For example, at the time of capturing a moving image, the control unit 30 sets the selected output image Vout as an image to be recorded on a recording medium or an image to be transmitted to an external device, and also as a monitor image to be displayed on the display unit 32. Furthermore, for example, at the time of capturing a still image, the control unit 30 sets the selected output image Vout as a monitor image to be displayed on the display unit 32, or as an image to be recorded or transmitted at the time of shutter operation. As a result, the selection result by the output image selection mode processing can be visually recognized and recorded.

[0177] The information processing device according to the embodiment includes the telephoto camera unit 31T that captures the telephoto image 55 and the wide-angle camera unit 31W that captures the wide-angle image 56. For example, a smartphone 1, a tablet terminal, a laptop personal computer, or the like has a camera function. In imaging of the smartphone 1 or the like, the function of the output image selection mode of the present disclosure can be exerted, and for example, it is useful in an imaging scene of a fast moving subject.

[0178] The program of the embodiment is a program for causing the control unit 30 of the information processing device to execute processing related to the output image selection mode. For example, the program is a program for causing an information processing device such as a CPU or a DSP, or a device including the information processing device to execute the processing in Figs. 12, 13, 14, 15, 16, 17, and 19.

[0179] That is, the program according to the embodiment is a program causing the information processing device to execute the output image selection mode processing of acquiring the telephoto image 55 captured at the telephoto field of view 50 and the wide-angle image 56 that is an image obtained by capturing a common subject simultaneously with the telephoto image 55, the wide-angle image 56 including the subject at the telephoto field of view 50 and being captured at the wide-angle field of view 52 that is wider, and selecting the output image Vout from a plurality of images including at least the telephoto image 55, the crop image 57 from the wide-angle image 56, and the wide-angle image 56 on the basis of a positional relationship between the determination target in the subject and the telephoto field of view 50 or the wide-angle field of view 52.

[0180] Such a program is included in, for example, the camera application of the smartphone 1, thereby realizing the functions of the present embodiment.

[0181] Such a program may be stored in advance in an HDD as a storage medium built in a device such as a computer device, a ROM in a microcomputer including a CPU, or the like. Alternatively, the program can be temporarily or permanently stored (recorded) in a removable recording medium such as a flexible disk, a compact disc read only memory (CD-ROM), a magneto optical (MO) disk, a digital versatile disc (DVD), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, or a memory card. Such a removable recording medium can be provided as so-called package software. Furthermore, such a program may be installed from the removable recording medium into a personal computer and the like, or may be downloaded from a download site through a network such as a local area network (LAN) or the Internet.

[0182] Furthermore, such a program is suitable for providing the information processing device according to the embodiment in a wide range. For example, by downloading the program to a mobile terminal device such as a smartphone or a tablet, an imaging device, a mobile phone, a personal computer, a game device, or the like, the smartphone or the like can be caused to function as the information processing device of the present disclosure.

[0183] Note that the effects described in the present description are merely examples and are not limited, and other effects may be provided.

[0184] Note that the present technology can also have the following configurations. (1) An information processing device including a control unit that performs output image selection mode processing of: acquiring a first captured image captured at a first field of view and a second captured image that is an image obtained by capturing a common subject simultaneously with the first captured image, the second captured image including the subject at the first field of view and being captured at a second field of view that is wider; and selecting an output image from a plurality of images including at least the first captured image, a crop image from the second captured image, and the second captured image on the basis of a positional relationship between a determination target in the subject and the first field of view or the second field of view. (2) The information processing device according to (1) described above, in which the crop image is an image obtained by cutting out an field of view range having a size same as the first field of view from the second captured image. (3) The information processing device according to (1) or (2) described above, in which the control unit recognizes the determination target in the second captured image, and determines a positional relationship between the determination target and the first field of view or the second field of view in the second captured image. (4) The information processing device according to any one of (1) to (3) described above, in which the control unit performs processing of selecting the first captured image in a case of determining that the determination target is included in the first field of view. (5) The information processing device according to any one of (1) to (4) described above, in which the control unit performs processing of selecting the crop image in a case of determining that the determination target is included in the second field of view and not included in the first field of view. (6) The information processing device according to any one of (1) to (5) described above, in which the control unit performs processing of selecting the second captured image in a case of determining that the determination target is not included in the second field of view. (7) The information processing device according to any one of (1) to (6) described above, in which the control unit performs control such that the second captured image is displayed on a display unit together with the selected output image. (8) The information processing device according to (7) described above, in which the control unit performs control such that the first field of view is indicated on display of the second captured image. (9) The information processing device according to any one of (1) to (7) described above, in which the control unit performs a start determination of the output image selection mode processing on the basis of an imaging situation determination. (10) The information processing device according to (9) described above, in which the control unit automatically turns on an output image selection mode according to the start determination of the output image selection mode processing. (11) The information processing device according to (9) described above, in which the control unit performs presentation control of recommending an output image selection mode to a user according to the start determination of the output image selection mode processing. (12) The information processing device according to any one of (9) to (11) described above, in which the control unit performs, as the imaging situation determination, at least one of a determination of presence or absence of a moving subject, a determination of an amount of movement of a camera capturing the first captured image or the second captured image, and another determination of a moving image and a still image. (13) The information processing device according to any one of (1) to (12) described above, in which the control unit performs control to switch the first captured image, the crop image, and the second captured image in the output image selection mode processing. (14) The information processing device according to any one of (1) to (12) described above, in which the control unit performs, as the output image selection mode processing, any one of: processing of switching the first captured image, the crop image, and the second captured image; processing of switching between the second captured image and the crop image; and processing of switching between the first captured image and the second captured image. (15) The information processing device according to any one of (1) to (14) described above, in which the determination target is a face of a person or an animal as a subject. (16) The information processing device according to any one of (1) to (15) described above, in which in the output image selection mode processing, the control unit corrects a position of the determination target in the second captured image according to an optical axis deviation between a camera that captures the first captured image and a camera that captures the second captured image, and then determines a positional relationship between the determination target and the first field of view or the second field of view. (17) The information processing device according to any one of (1) to (16) described above, in which the control unit performs, in the output image selection mode processing, processing of making the selected output image a record image, a transmission image, or a monitor display image on a display unit. (18) The information processing device according to any one of (1) to (17) described above, further including: a first camera unit that captures the first captured image; and a second camera unit that captures the second captured image. (19) An information processing method including: performing, by an information processing device, output image selection mode processing of: acquiring a first captured image captured at a first field of view and a second captured image that is an image obtained by capturing a common subject simultaneously with the first captured image, the second captured image including the subject at the first field of view and being captured at a second field of view that is wider; and selecting an output image from a plurality of images including at least the first captured image, a crop image from the second captured image, and the second captured image on the basis of a positional relationship between a determination target in the subject and the first field of view or the second field of view. (20) A program causing an information processing device to execute output image selection mode processing of: acquiring a first captured image captured at a first field of view and a second captured image that is an image obtained by capturing a common subject simultaneously with the first captured image, the second captured image including the subject at the first field of view and being captured at a second field of view that is wider; and selecting an output image from a plurality of images including at least the first captured image, a crop image from the second captured image, and the second captured image on the basis of a positional relationship between a determination target in the subject and the first field of view or the second field of view.

[0185] It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and alterations may occur depending on design requirements and other factors insofar as they are within the scope of the appended claims or the equivalents thereof.

[0186] 1　　　Smartphone 6U　　　Ultra wide-angle lens 6W　　　Wide-angle lens 6T　　　Telephoto lens 20　　　Monitor region 20A　　　Main monitor region 20B　　　Sub monitor region 30　　　Control unit 30a　　　Imaging signal processing unit 30b　　　UI processing unit 30c　　　Selection control unit 30d　　　Mode control unit 31T　　　Telephoto camera unit 31W　　　Wide-angle camera unit 31U　　　Ultra wide-angle camera unit 43　　　Recognition / detection unit 44　　　Camera selection unit 50　　　Telephoto field of view 51　　　Crop frame 52　　　Wide-angle field of view 55　　　Telephoto image 56　　　Wide-angle image 57　　　Crop image 58, 58C　　　Face frame

Claims

1. An information processing device, comprising: 　　　a control circuit configured to: 　　　acquire a first image of a target captured at a first field of view and a second image captured simultaneously with the first image, the second image being captured at a second field of view that is wider than and includes the first field of view; determine an output image selection mode; and 　　　select an output image from a plurality of images including at least the first image, a crop image from the second image, and the second image based on a positional relationship between the target , the first field of view, and the second field of view and based on the output image selection mode.

2. The information processing device according to claim 1, wherein 　　　the crop image is an image obtained by cutting out a field of view having a size of the first field of view from the second captured image.

3. The information processing device according to claim 1, wherein the control circuit is configured to: recognize the target in the second image, and determine a positional relationship between the target, the first field of view, and the second field of view in the second image.

4. The information processing device according to claim 1, wherein　　　the control circuit is configured to select the first image when an entirety of the target is in the first field of view.

5. The information processing device according to claim 1, wherein 　　　the control circuit is configured to select the crop image when an entirety of the target is in the second field of view and not in the first field of view.

6. The information processing device according to claim 1, wherein 　　　the control circuit is configured to select the second captured image when an entirety of the target is not in the second field of view.

7. The information processing device according to claim 1, wherein 　　　the control circuit is configured to display the second image on a display together with the selected output image.

8. The information processing device according to claim 7, wherein 　　　the control circuit is configured to indicate the first field of view with the second image on the display.

9. The information processing device according to claim 1, wherein 　　　the control circuit is configured to start an output image selection mode processing in which the output image selection mode is determined based on an imaging situation determination.

10. The information processing device according to claim 9, wherein 　　　the control circuit is configured to, on condition that the imaging situation determination indicates movement, automatically turn on the output image selection mode processing.

11. The information processing device according to claim 9, wherein 　　　the control circuit is configured to, on condition that the imaging situation determination indicates movement, recommend to turn on the output image selection mode processing to a user.

12. The information processing device according to claim 9, wherein 　　　the control circuit is configured to perform, as the imaging situation determination, at least one of a determination of presence or absence of a moving subject, a determination of an amount of movement of a first camera capturing the first image or a second camera capturing the second image, and a determination of a moving image and a still image.

13. The information processing device according to claim 1, wherein 　　　the control circuit is configured to switch between the first image, the crop image, and the second image based on the output image selection mode.

14. The information processing device according to claim 1, wherein 　　　the control circuit is configured to switch, based on the output image selection mode, between any one of: 　　　the first image, the crop image, and the second image; 　　　the second image and the crop image; and 　　　the first image and the second image.

15. The information processing device according to claim 1, wherein 　　　control circuit is configured to receive a setting of the target from a user.

16. The information processing device according to claim 1, wherein 　　　the control circuit is configured to, before determining the positional relationship, correct a position of the target in the second image according to an optical axis deviation between a first camera that captures the first image and a second camera that captures the second image.

17. The information processing device according to claim 1, wherein 　　　the control circuit is configured to make the selected output image a record image, a transmission image, or a monitor display image on a display.

18. The information processing device according to claim 1, wherein the control circuit is further configured to acquire a third image at a third field of view that is wider than and includes the second field of view; and select an output image from a plurality of images including at least the first image, a crop image from the second image, the second image, another crop image from the third image, and the third image based on a positional relationship between the target, the first field of view, the second field of view, and the third field of view, and based on the output image selection mode.

19. An information processing method comprising: 　　　performing, by an information processing device, output image selection mode processing of: 　　　acquiring a first image captured at a first field of view and a second image simultaneously with the first image, the second image being captured at a second field of view that is wider than and includes the first field of view; determining an output image selection mode; and selecting an output image from a plurality of images including at least the first image, a crop image from the second image, and the second image on a basis of a positional relationship between a target, the first field of view, and the second field of view and based on the output image selection mode.

20. A program causing an information processing device to execute output image selection mode processing of: 　　　acquiring a first image captured at a first field of view and a second image simultaneously, the second image being captured at a second field of view that is wider than and includes the first field of view; determining an output image selection mode; and selecting an output image from a plurality of images including at least the first image, a crop image from the second image, and the second image on a basis of a positional relationship between a target, the first field of view, and the second field of view and based on the output image selection mode.

Images