Imaging control device and control method for this

Abstract

Image control device, with: a control unit that is configured to receive an initial user operation and is configured to: In a case where the first user operation is not received, to execute a control to output an image signal to be displayed on a display unit, where the image signal is of one of: a first image corresponding to a first area of ​​a live image of a virtual reality, live VR image, which is a live view image received by an imaging unit; and a second image corresponding to a second area of ​​the live VR image that is enlarged relative to the live VR image, where the second area is a smaller area than the first area, and in a case where the first user operation is received: to execute a control to output an image signal that serves to change the image displayed on the display unit from one of the first image and the second image to the other of the first image and the second image, or to both the first image and the second image; and to issue a preparation instruction in order to perform image preparation processing.

Description

BACKGROUND OF THE INVENTIONField of the invention

[0001] The present invention relates to an image control apparatus, a control method for an electronic device, a program and a storage medium, and more particularly to a control method for displaying an image having a wide video range. Description of the state of the art

[0002] In recent years, cameras with a front and rear pair of integrated wide-angle lenses, such as fisheye lenses, capable of omnidirectional imaging, as well as virtual reality (VR) content generated based on images captured by these cameras have become widespread. Camera modules with wide-angle lenses that can be connected to a smartphone's expansion port are also available, and by using the camera module in combination with an application running on the smartphone, omnidirectional images can be captured.

[0003] To review an image during or after imaging by the imaging devices as described above, a two-dimensional display provided on the camera or smartphone is typically used. Regarding a format in which the image is to be displayed on the display, an omnidirectional image format in which the entire omnidirectional image is displayed on the display, or a format in which a partial area is extracted from the omnidirectional image and displayed on the display can be used.

[0004] Regarding the omnidirectional image format, a dome master format in which a 360-degree image is imaged onto a spherical area, an omnidirectional image format using equirectangular projection, a so-called panoramic format, etc. can be used. A format (hereinafter referred to as a fisheye format) in which two fisheye images respectively captured by a front and rear pair of optical systems provided in a camera capable of performing omnidirectional imaging as described above are displayed as they are can also be used.

[0005] Meanwhile, in the format in which a partial area is extracted from the omnidirectional image and displayed on the display, a partial area extracted based on a direction in which the display is held is usually displayed.

[0006] Japanese Patent Application Laid-Open No. 2014-222825 A discloses a technique for capturing a wide-angle video, preferably extracting an area where a moving body is present from the captured video, and converting an aspect ratio of a preferred image into a display aspect ratio. With Japanese Patent Application Laid-Open No. 2014-222825 A, when an object serving as a monitoring target is a moving body, the monitoring target can be automatically selected and photographed.

[0007] The document DE 10 2018 133 609 A1 discloses a control unit for the display of a wide-angle or panoramic camera, in which the user can switch between the display of a wide-angle view and the display of a partial area from the wide-angle image.

[0008] Document US 2017 / 0 076 477 A1 discloses a display control of a camera with a wide-angle lens (fisheye) and an operating unit which enables the user to choose between a wide-angle view and a view of a section of the wide-angle view, wherein the display of the section is subjected to image correction (distortion correction). SUMMARY OF THE INVENTION

[0009] However, with the format in which a partial area extracted from an omnidirectional image is displayed, it is difficult to review the entire image and a portion of the image in the direction of the photographer's blind spot. To review the entire image and the portion of the image in the direction of the photographer's blind spot, the photographer must modify a display area by pointing the camera in each of several directions or by operating a touch-sensitive panel, which makes the review process cumbersome. On the other hand, with the format in which the entire omnidirectional image is displayed, it is difficult to review the image in detail, and as a result, a photo opportunity may be missed.Furthermore, Japanese Patent Application Laid-Open No. JP 2014-222825 A does not describe area extraction in a case where the object is stationary, and yet the object is not necessarily constantly moving.

[0010] Accordingly, the present invention provides a technique that enables a photographer to more advantageously review an image, so that the photographer can perform imaging at an appropriate time without missing a photographic opportunity.

[0011] An imaging control device according to the present invention comprises: a control unit configured to receive a first user operation, and configured to: in a case where the first user operation is not received, execute control for outputting an image signal to be displayed on a display unit, the image signal being one of: a first image corresponding to a first region of a live image of virtual reality, live VR image which is a live view image output from an imaging unit, andimaging unit; and a second image corresponding to a second area of ​​the live VR image that is enlarged relative to the live VR image, the second area being a smaller area than the first area, and in a case where the first user operation is received: a controller for outputting an image signal that changes the image displayed on the display unit from one of the first image and the second image to the other of the first image and the second image or to both the first image and the second image; and outputting a preparation instruction for executing image preparation processing.

[0012] According to the present invention, a photographer can check an image more advantageously and as a result, the photographer can perform imaging at an appropriate time without missing an opportunity for a photograph.

[0013] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1A to 1D are views showing the external appearance of a camera 100. Fig. 2 is a view showing an internal configuration of the camera 100. Fig. 3 is a flowchart executed by a camera control unit 120. Fig. 4 is a flowchart of display / recording processing executed by the camera control unit 120. Fig. 5A to 5D show examples of displays on a display unit 141. Fig. 6A to 6C show other examples of displays on the display unit 141. Fig. 7 shows another example of a display on the display unit 141. DESCRIPTION OF THE EMBODIMENTS

[0014] Fig. 1A to 1D show the external appearance of a camera 100, which is an example of an imaging control device to which the present invention can be applied. The camera 100 is a digital camera with an integrated lens. Note that the camera 100 may be a single-lens reflex camera, a mirrorless SLR camera, or the like. The present invention is applicable to various types of imaging devices to which wide-angle lenses are attached.

[0015] Fig. 1A shows the external appearance of a front surface of the camera 100. A display unit 141 displays captured images and various types of information. The image displayed on the display unit 141 is switched in response to an imaging preparation instruction. A touch panel 13 provided on a display surface of the display unit 141 receives touch operations performed by a photographer. A release button 11 is a switch that can be half-pressed and fully pressed. Half-pressing the release button 11 outputs an imaging preparation instruction to the camera 100, and fully pressing the release button 11 outputs an imaging instruction to the camera 100. A lens 111 is a wide-angle lens, and an optical flow entering the lens 111 is imaged by an imaging device 112 (in Fig. 1A not shown) into an image.

[0016] Fig. 1B shows the external appearance of a back side of the camera 100. By pressing a button 12, an imaging preparation instruction is issued to the camera 100, and by performing an operation to remove a finger or the like that presses the button 12 from the button 12, an imaging instruction is issued to the camera 100. A lens 101 is a similar wide-angle lens to the lens 111, and an optical flow entering the lens 101 is imaged by an imaging device 102 (which is shown in Fig. 1B not shown) into an image.

[0017] A VR image is formed by the imaging devices 102, 112. A VR image is an image that can be used in a VR display. VR images include omnidirectional images captured by an omnidirectional camera, panoramic images with a wider video range (effective video range) than a display range that can be simultaneously displayed by the display unit, and so on. VR images include not only static images but also moving images and live view images (images obtained from a camera essentially in real time). A VR image has a maximum video range (effective video range) corresponding to a field of view of 360 degrees in a top-down direction (a vertical angle, an angle from the zenith, an inclination angle, a pitch angle, an elevation) and 360 degrees in a left-right direction (a horizontal angle, an azimuth angle).Furthermore, VR images include images that, although smaller than 360 degrees in the up-down direction or 360 degrees in the left-right direction, have a wider viewing angle (field of view) than the viewing angle that can be captured by a conventional camera, or a wider video range (effective video range) than the display area that can be simultaneously displayed on the display unit. For example, an image captured by an omnidirectional camera capable of photographing an object in a field of view (field of view) of 360 degrees in the left-right direction (the horizontal angle, the azimuth angle) and 210 degrees in the vertical angle around the zenith as the center is a type of VR image.Furthermore, for example, an image captured by a camera capable of photographing an object within a field of view (a viewing angle range) of 180 degrees in the left-right direction (the horizontal angle, the azimuth angle) and 180 degrees in a vertical angle centered on the zenith is a type of VR image. In other words, images with a video range corresponding to a field of view of at least 160 degrees (± 80 degrees) in both the up-down and left-right directions and a wider video range than a range that can be viewed by a human simultaneously are types of VR images. When a VR image is used in VR display, a seamless omnidirectional video can be viewed in the left-right direction (a horizontal rotation direction) by changing the posture of the display unit in a left-right rotation direction.In the top-down direction (a vertical rotation direction), a seamless omnidirectional video can be viewed within a range of ± 105 degrees when viewed directly from above (the zenith). However, an area exceeding 105 degrees from directly above is an empty area where no video exists. A VR image can also be defined as an "image with a video area that forms at least part of a virtual space (VR space)."

[0018] A VR display is a display method that can modify the display area of ​​a VR image to display a video with a field of view corresponding to the posture of the display unit. In a case where a user watches a video while wearing a head-mounted display (HMD), which is the display unit, a video with a field of view corresponding to the orientation of the user's face is displayed. For example, at a certain point, a video of a portion of the VR image is displayed with a field of view angle (angle of view) around 0 degrees in the left-right direction (a specific posture or bearing, for example, north) and 90 degrees in the up-down direction (90 degrees from the zenith, i.e., horizontal).When the attitude of the display unit is reversed from this state (for example, when the display surface is modified from a south-facing orientation to a north-facing orientation), the display area is modified to a video of a portion of the same VR image with a viewing angle of 180 degrees in the left-right direction (an opposite bearing, e.g., southward) and 90 degrees in the up-down direction (horizontally). Consequently, when the user moves their face from a north-facing orientation to a south-facing orientation (in other words, when they look back) while viewing the HMD, the video displayed on the HMD similarly changes from a north-facing video to a south-facing video. With a VR display, the user can be provided with a virtual feeling as if they were at the location of the VR image (the VR space).A smartphone mounted on a VR headset (a head-mounted adapter) can be considered a type of HMD. The display area can be moved (scrolled) not only by changing posture, but also in response to user action performed on a touch-sensitive pad, directional button, or similar device.

[0019] Fig. 1C shows the external appearance of a side surface of the camera 100. The lens 101 and the lens 111 are mounted in pairs on the front and rear sides of the camera 100, and the respective viewing angles of the lens 101 and the lens 111 are 180 degrees. In other words, a 360-degree viewing angle is satisfied by the lens 101 and the lens 111. Accordingly, images corresponding to 180-degree viewing angles are captured by the imaging device 102 and the imaging device 112, respectively. A VR image corresponding to a 360-degree viewing angle is formed by the two images captured by the two imaging devices 102, 112.

[0020] Fig. 1D shows an example of a screen displayed on the display unit 141. Here, a fisheye image captured by the imaging device 102 or the imaging device 112 is displayed. A touch panel button 14 is also displayed. The touch panel button 14 is an element used to output an imaging preparation instruction and an imaging instruction. More specifically, the touch panel button 14 is configured to output an imaging preparation instruction to the camera 100 by performing a touch operation to touch a position on the touch panel 13 corresponding to the touch panel button 14.An imaging instruction is then output to the camera 100 by performing an operation to release a finger or a stylus that touches the button of the touch panel 14 from the button of the touch panel 14 (of the touch panel 13). Further, by performing a touch operation to touch a position within the displayed fisheye image, or more specifically, a position on the touch panel corresponding to that position, a display area selection instruction is output while outputting an imaging preparation instruction.

[0021] Note that the display unit 141 may be a separate device from the camera 100. For example, the display unit 141 may be provided on a smartphone or the like provided separately from the camera 100. Furthermore, user operations for outputting an imaging preparation instruction and an imaging instruction may be performed on a separate device from the camera 100. For example, the user operations may be performed on a smartphone provided separately from the camera 100, and the smartphone may output the instructions to the camera 100 according to the user operations.

[0022] Fig. Figure 2 shows an internal configuration of the camera 100. It should be noted that although Fig. 2 shows corresponding functions as separate components. The corresponding functions can be implemented by one or a plurality of hardware elements, such as an ASIC or a programmable logic array (PLA). The corresponding functions can also be implemented by a programmable processor, such as a CPU or an MPU, executing software.

[0023] The lens 101 is a single focus lens with a diaphragm mechanism, and a depth focus is used as a focus position of the lens 101.

[0024] The imaging device 102 is formed by a two-dimensional array of a plurality of pixels including photoelectric conversion elements. The imaging device 102 generates an analog signal by subjecting an object image (an optical image) formed by an optical imaging system including the lens 101 to photoelectric conversion. In other words, imaging in each pixel converts the analog signal into a digital signal using an A / D conversion circuit and outputs RAW image data in a unit of pixel.

[0025] A memory interface unit 106 writes the RAW image data for all the pixels output from the imaging device 102 into a memory 107. Furthermore, the memory interface unit 106 reads the RAW image data held in the memory 107 and outputs the read data to an image processing unit 108. The memory 107 is a volatile storage medium for storing RAW image data for all the pixels corresponding to multiple frames.

[0026] The image processing unit 108 implements processing such as gamma correction, color separation, and color difference matrixing on the RAW image data of all pixels from the memory interface unit 106, adds a synchronization signal thereto, and outputs video data for recording or display. Furthermore, the image processing unit 108 obtains a photometric value from a luminance value of each pixel in the RAW image data input to the image processing unit 108 and provides the photometric values ​​to a camera control unit 120 via a bus 121.

[0027] The lens 111, the imaging device 112, a memory interface unit 116, a memory 117, and an image processing unit 118 have similar functions to the lens 101, the imaging device 102, the memory interface unit 106, the memory 107, and the image processing unit 108. The lens 111, the imaging device 112, the memory interface unit 116, the memory 117, and the image processing unit 118 respectively perform similar processing to the lens 101, the imaging device 102, the memory interface unit 106, the memory 107, and the image processing unit 108.

[0028] The camera control unit 120 is constituted by a CPU, etc., and controls the overall operation of the camera 100. A RAM 160 is used as a working area (a working memory) of the camera control unit 120. Constants and variables used to operate the camera control unit 120 are recorded in the RAM 160, and programs, etc., are expanded thereon. A ROM 180 stores a computer program, etc., for operating the camera control unit 120. For example, the camera control unit 120 controls the overall operation of the camera 100 by expanding and executing the computer program stored in the ROM 180 on the RAM 160.

[0029] A gain control unit 103, a shutter control unit 104, and an aperture control unit 105 are used in exposure control. The camera control unit 120 controls these control units 103 to 105 based on the photometry values ​​provided by the image processing unit 108 and the image processing unit 118 or operating parameters manually set by the user. The gain control unit 103 controls respective gains of the imaging device 102 and the imaging device 112. The shutter control unit 104 controls respective shutter speeds of the imaging device 102 and the imaging device 112. The aperture control unit 105 controls the aperture mechanism of the lens 102 and the aperture mechanism of the lens 111.

[0030] A storage medium interface unit 130 is an interface for connecting a storage medium 131 to the camera 100. The storage medium interface unit 130 records video data input thereto from the image processing unit 108, the image processing unit 118, and a GPU 142 into the storage medium 131 and reads the recorded video data from the storage medium 131. The storage medium 131 is constituted by a semiconductor memory or the like.

[0031] A display interface unit 140 outputs the video data from the image processing unit 108 and the image processing unit 118 and image data rendered in a video RAM (VRAM) 143 by the GPU 142 to a display unit 141. Images displayed on the display unit 141 are mainly used to review images taken by a photographer.

[0032] The GPU 142 is a rendering engine for performing image conversion processing, image extraction processing, magnification processing, distortion correction, and the like on the video data output from the image processing unit 108 and the image processing unit 118, based on an instruction from the camera control unit 120, to render the video data in the VRAM 143. The GPU 142 also performs processing for rendering, in an overlaid manner, various types of displayed information related to the camera 100 and a menu screen in the VRAM 143. The above-described touch panel button 14 is rendered by the GPU 142 as a graphical user interface (GUI) and displayed on the display unit 141 via the display interface unit 140.

[0033] A communication interface unit 150 connects to a smartphone, a PC, or the like (not shown in the figure) on the outside of the camera 100 via a communication unit 151 to transmit the image data displayed on the display unit 141 thereto. Furthermore, the communication interface unit 150 receives control instructions and the like issued to the camera 100 by the smartphone, PC, or the like, and informs the camera control unit 120 of them. The communication unit 151 is a wireless LAN interface that performs wireless communication and executes hardware processing on the communication performed by the communication interface unit 150.

[0034] The release button 11, the button 12, and the touch panel 13 have the functions described above and output operation information (an image preparation instruction, an image preparation instruction, a display area selection instruction, etc.) corresponding to an operation to an input / output controller 170. The input / output controller 170 outputs the operation information from the corresponding operation units to the camera control unit 120, and the camera control unit 120 executes processing according to the operation information from the input / output controller 170. For example, the touch panel 13 is laminated to the display unit 141 to output operation information including coordinate information indicating a position or area of ​​a touch operation, etc.

[0035] Fig. 3 is a flowchart showing overall processing performed by the camera 100. The processing on the flowchart is realized by the camera control unit 120 by expanding and executing the program recorded in the ROM 180 on the RAM 170. The Fig. The flowchart shown in Fig. 3 is started when, for example, a power supply is introduced into the camera 100, and processing from S320 to S370 is repeatedly executed at a predetermined frame rate (at predetermined periods).

[0036] In S301, the camera control unit 120 performs coordinate calibration on the display unit 141 and the touch panel 13 (touch panel initialization). In this way, coordinates on the touch panel 13 and coordinates on an image displayed on the display unit 141 are aligned, and as a result, touch operations performed on the touch panel button 14, touch operations performed on a captured image, etc., can be detected.

[0037] In S302, the camera control unit 120 sets an aperture value in the aperture control unit 105 (aperture setting). In a first aperture setting operation, the camera control unit 120 sets an open aperture value in the aperture control unit 105 as an initial value of the aperture value. Following a second aperture setting operation, the camera control unit 120 sets an aperture value determined during exposure setting processing of S306 in the aperture control unit 105.

[0038] In S303, the camera control unit 120 sets a gain in the gain control unit 103 (gain adjustment). In a first gain adjustment operation, the camera control unit 120 sets 0 dB in the gain control unit 103 as an initial gain value. Following a second gain adjustment operation, the camera control unit 120 sets a gain determined during the exposure adjustment processing of S306 in the gain control unit 103.

[0039] In S304, the camera control unit 120 sets a shutter speed in the shutter control unit 104 (shutter setting). In a first shutter setting operation, the camera control unit 120 sets 1 / 60 second in the shutter control unit 104 as an initial shutter speed value. Following a second shutter setting operation, the camera control unit 120 sets a shutter speed determined during the exposure setting operation of S306 in the shutter control unit 104.

[0040] In S305, the camera control unit 120 executes imaging / reading processing for the purpose of live view imaging. More specifically, the aperture control unit 105 executes processing to control the respective aperture mechanisms of the lenses 101, 111 based on the aperture value set in S302 in response to an instruction from the camera control unit 120. The gain control unit 103 executes processing to control the respective magnifications of the imaging devices 102, 112 based on the magnification set in S303 in response to an instruction from the camera control unit 120. The shutter control unit 104 executes processing to control the respective shutter speeds of the imaging devices 102, 112 based on the shutter speed set in S304 in response to an instruction from the camera control unit 120.The imaging devices 102, 112 then perform imaging based on the values ​​set in S302 to S304 in response to an instruction from the camera control unit 120. Next, the memory interface unit 106 performs processing to record an image (a RAW image) captured by the imaging device 102 in the memory 107 and output the image to the image processing unit 108 in response to an instruction from the camera control unit 120. Similarly, the memory interface unit 116 performs processing to record an image (a RAW image) captured by the imaging device 112 in the memory 117 and output the image to the image processing unit 118 in response to an instruction from the camera control unit 120.

[0041] In S306, the camera control unit 120 determines whether the exposure of the images captured in S305 is appropriate and determines exposure values ​​(the aperture value, the gain, and the shutter speed) for the next imaging operation (exposure adjustment processing). Specifically, the camera control unit 120 obtains photometric values ​​acquired from the image processing units 108, 118 based on the images captured in S305 and calculates an exposure evaluation value based on the obtained photometric values. The camera control unit 120 then compares the calculated exposure evaluation value with a predetermined exposure target value and determines whether the exposure of the captured images is appropriate.If the exposure is not appropriate, the camera control unit 120 determines the exposure values ​​of the next imaging operation so that images are captured closer to an optimal exposure.

[0042] In S307, the camera control unit 120 performs display / recording processing. Fig. Figure 4 is a flowchart showing the display / recording processing.

[0043] In S401, the camera control unit 120 obtains an operating state of the release button 11 and records the obtained operating state in the RAM 160. The RAM 160 stores operating states of the release button 11 corresponding to at least two periods in the form of field data. In this way, a variation in the operating state of the release button 11 can be detected. The operating state of the release button 11 includes three states: a non-pressed state in which the release button 11 is not pressed, a half-pressed state in which the release button 11 is half-pressed, and a fully pressed state in which the release button 11 is fully pressed. The camera control unit 120 updates the field data regarding the release button 11 each time S401 is executed.

[0044] In S402, the camera control unit 120 obtains an operating state of the touch panel button 14 and records the obtained operating state in the RAM 160. The RAM 160 stores operating states of the touch panel button 14 corresponding to at least two periods in the form of field data. In this way, a variation in the operating state of the touch panel button 14 can be detected. The operating state of the touch panel button 14 includes two states: a touched state in which the touch panel button 14 is touched and a non-touched state in which the touch panel button 14 is not touched. The camera control unit 120 updates the field data relating to the touch panel button 14 each time S402 is executed.

[0045] In S403, the camera control unit 120 obtains an operation state of the button 12 and records the obtained operation state in the RAM 160. The RAM 160 stores operation states of the button 12 corresponding to at least two periods in the form of field data. In this way, a variation in the operation state of the button 12 can be detected. The operation state of the button 12 includes two states: a pressed state in which the button 12 is pressed and a non-pressed state in which the button 12 is not pressed. The camera control unit 120 updates the field data relating to the button 12 each time S403 is executed.

[0046] In S404, the camera control unit 120 determines whether or not an image preparation instruction has been issued based on the operation states obtained in S401 to S403. Specifically, if the operation state of the release button 11 is the half-pressed state, the camera control unit 120 determines that an image preparation instruction has been issued and proceeds to S406. Similarly, if the operation state of the touch panel button 14 is the touched state or the operation state of the button 12 is the pressed state, the camera control unit 120 determines that an image preparation instruction has been issued and proceeds to S406.The above-described method for the image preparation instruction is only an example, and the image preparation instruction may be issued using only one operation element, or the image preparation instruction may be issued using a different operation element. In all other cases, the camera control unit 120 determines that an image preparation instruction has not been issued and proceeds to S405.

[0047] In S405, the camera control unit 120 determines whether or not an imaging instruction has been issued based on the operation states obtained in S401 to S403. Specifically, when the operation state of the release button 11 has shifted from the half-pressed state to the pressed state, the camera control unit 120 determines that an imaging instruction has been issued and proceeds to S407. Similarly, when the operation state of the touch panel button 14 has shifted from the touched state to the untouched state, or the operation state of the button 12 has shifted from the pressed state to the unpressed state, the camera control unit 120 determines that an imaging instruction has been issued and proceeds to S407.The imaging instruction method described above is only an example, and the imaging instruction may be issued using only one operation element, or the imaging instruction may be issued using a different operation element. In all other cases, the camera control unit 120 determines that neither an imaging instruction nor an imaging preparation instruction has been issued and proceeds to S408.

[0048] In S406, the camera control unit 120 executes zoom view processing and issues an imaging preparation instruction (an instruction to execute imaging preparation processing). In response to the imaging preparation instruction, at least one of the following processes is executed: autofocus (AF) processing; aperture adjustment by the aperture control unit 105, which corresponds to exposure adjustment (automatic exposure adjustment); exposure time adjustment (shutter time adjustment) by the shutter control unit, which is executed during imaging in S407; white balance adjustment; and gain adjustment by the gain control unit 103. In other words, the imaging preparation instruction is an instruction to execute at least one of the above processes. The camera control unit 120 then ends the display / recording processing.During zoom view processing, the camera control unit 120 controls the processing of the GPU 142, etc., so that a zoom view image obtained by transforming a portion of the area of ​​the VR image captured by the imaging devices 102, 112 so that the corresponding portion is magnified is rendered in the VRAM 143 and displayed on the display unit 141. A zoom view image is a type of live view image that is sequentially updated. An example of a zoom view image is shown below using [Figure 14]. Fig. 5D described.

[0049] In S407, the camera control unit 120 executes recording processing. The camera control unit 120 then ends the Fig. 4. In the recording processing, the camera control unit 120 executes control to record the captured VR image in the storage medium 131. The recording processing may also be referred to as imaging. When a motion picture recording mode is set, the camera control unit 120 executes recording processing to start recording the motion picture (a VR motion picture or the like). When capturing a live view image, the time that can be spent on a single imaging operation is limited, and therefore, the exposure value may be smaller than the exposure value set during the exposure adjustment processing. Furthermore, capturing may be performed at a comparatively low resolution (number of pixels) when capturing a live view image.

[0050] During recording processing, shooting is performed at the exposure value set during exposure adjustment processing and at a higher resolution than the resolution used to capture a live view image. Specifically, after checking the zoom view image displayed in S406, a mapping instruction is issued, after which the recording processing of S407 is executed.

[0051] In S408, the camera control unit 120 executes wide-angle view processing. During the wide-angle view processing, the camera control unit 120 controls the processing of the GPU 142, etc., so that a wide-angle view image formed by at least a part of the area of ​​the captured VR image is rendered in the VRAM 143 and displayed on the display unit 141. A wide-angle view image is a type of live view image. An example of a wide-angle view display is described below using Fig. 5A and Fig. 5B. The range of a zoom view image is narrower or smaller than the range of a wide-angle view image. In other words, a zoom view image displays a portion of the wide-angle view image that has been magnified and subjected to distortion correction or transformation (deformation). In S409, the camera control unit 120 sets the range of the zoom view image in response to the display area selection instruction that can be instructed by the touch panel 13. The camera control unit 120 then ends the display / recording processing.

[0052] Fig. 5A to 5D show examples of displays according to this embodiment. When a user operation for issuing an image preparation instruction or an image instruction has not been performed, a wide-angle image is preferably displayed so that the photographer can check a wide area on the periphery thereof. According to this embodiment, in this case, a fisheye image 501, which is shown in Fig. 5A, which is taken by the imaging device 102, or a fisheye image 503, which is shown in Fig. 5B, which is captured by the imaging device 112, is displayed as a wide-angle view image in response to the wide-angle view processing of S408.

[0053] An object 502 is visible in the fisheye image 501 of Fig. 5A. When the photographer touches the surroundings of the object 502, the touch-sensitive panel 13 outputs coordinate information corresponding to the touched position to the camera control unit 120. Based on the coordinate information from the touch-sensitive panel 13, the camera control unit 120 detects an area 504 ( Fig. 5C), which contains the touched position, as an area of ​​a zoom view image. As shown in Fig. 5C, the region 504 is a region extending radially from a position within a VR image corresponding to an optical axis of the lens 101, or, in other words, a central position of the fisheye image 501.

[0054] When a user operation to issue a mapping preparation instruction from the Fig. 5C is performed by the zoom processing of S406, the display on the display unit 141 is switched from the display of the fisheye image 501 to the display of a zoom view image 505 obtained by transforming (deforming) and enlarging the area 504. More specifically, the GPU 142 generates the zoom view image 505 in response to an instruction from the camera control unit 120 by extracting the area 504 from the fisheye image 501 and implementing magnification processing and distortion correction on the image of the area 504, and renders the generated zoom view image 505 in the VRAM 143. As a result, the image rendered in the VRAM 143 is updated from the fisheye image 501 to the zoom view image 505, and the display on the display unit 141 is switched from displaying the fisheye image 501 to displaying the zoom view image 505.

[0055] The camera control unit 120 determines the intensity of the magnification processing and distortion correction used to obtain the zoom view image 505 based on a distance between the central position of the fisheye image 501 and the position of the area 504. The GPU 142 then performs magnification processing and distortion correction at the intensity determined by the camera control unit 120. For example, the camera control unit 120 determines that the intensity of the distortion correction is lower as the distance decreases, or higher as the distance increases. In a case where the distance does not exceed a threshold, the camera control unit 120 performs control so that the distortion correction is either omitted or implemented at an intensity that ensures that the distortion of the VR image does not change.When distortion correction is implemented at an intensity where the distortion of the VR image varies, the zoom view image 505 is obtained with substantially no distortion in a central portion thereof. Here, distortion refers to distortion with respect to a real image of the object.

[0056] The zoom view image 505 is continuously displayed while the user operation for issuing an image preparation instruction is performed. When the user operation for issuing an image preparation instruction is terminated, the camera control unit 120 performs control to return the display on the display unit 141 from the display of the zoom view image 505 to the display of the fisheye image 501 or the fisheye image 503.

[0057] According to this embodiment, as described above, by displaying a wide-angle image such as a fisheye image before performing the user operation to issue an image preparation instruction, a wide area on the periphery of the camera 100 can be inspected. Then, by switching to a zoom view display in response to the user operation to issue an image preparation instruction, a part of the fisheye image can be inspected in detail. More specifically, while the wide-angle image is displayed, the photographer can specify a target object by performing a touch operation, and in response to the user operation to issue an image preparation instruction, the display is switched to a zoom view display of the target object. As a result, the target object can be inspected in detail.Thus, for example, conditions on the periphery of the camera 100 can be roughly checked from the wide-angle view display before issuing the imaging preparation instruction, and by issuing the imaging preparation instruction, the facial expression on the enlarged target object can be checked from the zoom view display. Then, at the moment the target object smiles, an image of the target object can be captured. According to this embodiment, therefore, the photographer can more advantageously check the image, and as a result, the photographer can perform imaging at an appropriate time without missing a photo opportunity.

[0058] It should be noted that although a preferred embodiment of the present invention has been described above, the present invention is of course not limited to this embodiment, and various modifications and changes may be made within the scope of the spirit thereof. Various modified examples described below may be combined with the above embodiment where applicable.

[0059] For example, in the example described above, the camera 100 is configured to include two imaging systems, but the present invention can also be applied to a camera including one imaging system or three or more imaging systems.

[0060] Furthermore, in the example described above, a part of the VR image (i.e., one of the fisheye images) is displayed without being modified during the wide-angle view display, but the present invention is not limited to this, and instead, according to the control of the camera control unit 120, a transformed image or the entire area of ​​the VR image may be displayed as the wide-angle view display. As shown in Fig. For example, as shown in Figure 6A, two fisheye images 601, 602, respectively captured by the two imaging devices 102, 112, can both be displayed as the wide-angle view display. As shown in Fig. 6B, a panoramic image 603 corresponding to the entire area of ​​the VR image may also be displayed as the wide-angle view display. As shown in Fig. 6C, a dome master image 604 corresponding to the entire area of ​​the VR image may also be displayed as the wide-angle view display. Images in other formats may also be displayed as the wide-angle view display.

[0061] The panoramic image 603 is obtained by geometrically transforming each part of the fisheye images 601, 602 into peripheral parts and central parts to expand and combine the fisheye images 601, 602. As described by Fig. 6B, the panoramic image 603 is a rectangular image obtained by geometrically transforming and thereby combining the fisheye images 601, 602. The dome master image 604 is similarly obtained by geometrically transforming and thereby combining the fisheye images 601, 602. The dome master image 604 shows the object as if viewed from the sky above the object. An upper half of the dome master image 604 corresponds to the fisheye image 601, and a lower half of the dome master image 604 corresponds to the fisheye image 602. These geometric transformations can be realized using predetermined information (a table or a function) representing a correspondence relationship between image coordinates before transformation and image coordinates after transformation.

[0062] Furthermore, the VR image recorded by the recording processing (mapping) may be a fisheye image, a panoramic image, a dome master image, or an image in another format.

[0063] Furthermore, in the example described above, the range of the zoom view image is determined according to a touch operation, but the range of the zoom view image may be determined by another method. For example, the camera control unit 120 may automatically detect a predetermined object (a person, a face, a vehicle, a building, a factory, or the like) from the VR image (an omnidirectional image) and automatically set a part where the predetermined object exists as the range of the zoom view image. The predetermined object may be detected using a known image recognition function. Furthermore, when a plurality of objects are detected, an object to serve as a main object may be automatically selected, and the range of the zoom view image may be determined to include the selected main object.The main object can be determined by a method of applying, for example, weights so that a part with high contrast, a face, a part of the captured image with a large size, a part near the camera 100, the face of a person registered in the camera 100 (a face detected by personal authentication), or the like is prioritized. On a VR image (a panoramic image or a dome master image) obtained by combining a plurality of images, boundaries exist between the plurality of images. The points of line 605 in . Fig. 6C indicate boundaries between the fisheye image 601 and the fisheye image 602. The camera control unit 120 can automatically detect the boundaries of the VR image and a part (a region 606 in Fig. 6C, for example), which includes at least part of the boundaries, as the range of the zoom view image. The camera 100 can be provided with a variety of operation modes with different methods of determining the range of the zoom view image (method of obtaining the zoom view image).

[0064] Here, a plurality of predetermined objects may exist on the VR image. In a configuration where parts where the predetermined object exists are subjected to zoom view display, the camera control unit 120 may, but need not, execute control to display a single zoom view image corresponding to one of the plurality of predetermined objects as the zoom view image. The camera control unit 120 may, as zoom view display control, execute control to display a plurality of zoom view images corresponding to the plurality of predetermined objects sequentially on the display unit 141. During this control, the camera control unit 120 may switch the displayed zoom view image between the plurality of zoom view images either automatically or in response to a user operation.

[0065] Furthermore, in the example described above, a wide-angle view display is switched to a zoom view display in response to the user operation of issuing an image preparation instruction, but the present invention is not limited to this. Instead, a zoom view display may be implemented under the control of the camera control unit 120 until the user operation of issuing an image preparation instruction is performed, and in response to the user operation of issuing an image preparation instruction, the zoom view display may be switched to the wide-angle view display. For example, a video viewed in a specific direction from the camera 100, which has been enlarged by being subjected to the zoom view display but is not distorted and does not appear unnatural, can be checked before the image preparation instruction is issued.Then, when the imaging preparation instruction is issued, the display is switched to the wide-angle view display, and as a result, an imaging operation can be performed at the moment when an event suitable for photography occurs on the periphery of the camera 100 (for example, the moment when a large number of fireworks explode).

[0066] In addition, the zoom view display and the wide-angle view display do not need to be switched. As in Fig.As shown in FIG. 7, before the imaging preparation instruction is issued (in an imaging standby state), one of the zoom view image and a wide-angle view image is displayed. Then, in response to the user operation of issuing an imaging preparation instruction, display is switched from one of the zoom view image and the wide-angle view image to displaying both images under the control of the camera control unit 120. When the zoom view image and the wide-angle view image are both displayed, the photographer can perform imaging by viewing the wide-angle view image without missing a moment when an event suitable for photography occurs on the periphery of the camera 100.Furthermore, by viewing the zoom view image, the photographer can perform imaging without missing a photo opportunity that occurs within a narrow range, such as a change in facial expression. Note that in the imaging standby state (while the imaging preparation instruction is being issued), one of the zoom view image and the wide-angle view image is displayed, not both. This is because displaying both images in the imaging standby state increases the image processing load, resulting in an increase in power consumption (in other words, to save energy).

[0067] Furthermore, in the imaging standby state and during the period in which the imaging preparation instruction is issued, display of the zoom view image and the wide-angle view image can be automatically switched according to an operation mode. For example, when a person imaging mode (first mode), which is one of a plurality of imaging modes, is set, a wide-angle view image is displayed in the imaging standby state, and the display is switched to either the zoom view image or both images in response to the imaging preparation instruction. Thereby, as described above, the photographer can perform imaging without missing a small photographic opportunity that occurs within a narrow range, such as a change in facial expression.Meanwhile, in a fireworks imaging mode, a sports imaging mode, or a landscape imaging mode (a second mode), which is one of the plurality of imaging modes, the zoom view image is displayed in the imaging standby state, and the display is switched to either the wide-angle view image or both images in response to the imaging preparation instruction. As a result, as described above, the photographer can perform imaging without missing an event occurring in a wide area on the periphery of the camera or an event occurring at an unpredictable position.

[0068] Furthermore, the various types of control described above as being performed by the camera control unit 120 may be implemented using a single piece of hardware, or the overall control of the device may be realized by dividing the processing among a plurality of pieces of hardware.

[0069] Furthermore, in the above embodiment, a case where the present invention is applied to the camera 100 was described as an example, but the present invention is not limited to this example and can be applied to any electronic device capable of executing control for displaying at least a part of a VR image. Specifically, the present invention can be applied to a personal computer, a PDA, a mobile phone terminal, a portable image viewer, a printer device with a display, a digital photo frame, etc. The present invention can also be applied to a music player, a game device, an electronic book reader, a tablet terminal, a smartphone, a projection device, a home appliance, or an in-vehicle device with a display, etc.

[0070] Furthermore, the present invention is not limited to a main body of an imaging device, but can also be applied to a control device that remotely controls an imaging device (including a network camera) by communicating with the imaging device through wired or wireless communication. Examples of devices for remotely controlling an imaging device include a smartphone, a tablet PC, a desktop PC, and so on. The imaging device can be remotely controlled by transmitting instructions for implementing various operations and settings from the control device side to the imaging device based on operations performed on the control device side and processing executed on the control device side.In addition, a live view image captured by the imaging device can be received through wireless or wired communication and displayed on the control device side. <Andere Ausführungsbeispiele>

[0071] An embodiment(s) of the present invention may also be implemented by a computer of a system or device that retrieves and executes computer-executable instructions (e.g., one or more programs) recorded on a storage medium (more fully referred to as a "non-transitory computer-readable storage medium") to perform the functions of one or more of the above-described embodiments, and / or that includes one or more circuits (e.g., application-specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiments, and by a method implemented by the computer of the system or device by, for example, reading and executing the computer-executable instructions from the storage medium,to perform the functions of one or more of the above-described embodiments, and / or control the one or more circuits for performing the functions of one or more of the above-described embodiments. The computer may have one or more processors (e.g., central processing unit (CPU), microprocessing unit (MPU)) and may include a network of separate computers or separate processors to retrieve and execute the computer-executable instructions. The computer-executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may, for example, be one or more of a hard disk, random access memory (RAM), read-only memory (ROM), distributed computer system memory, an optical disk (such as a compact disk (CD),a “Digital Versatile Disc” (DVD), or Blu-ray Disc (BD)™, a flash storage device, a memory card and the like.

[0072] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

[0073] An imaging control device comprises: a control unit configured to receive a first user operation, and configured to: when the first user operation is not received, execute control to output an image signal to be displayed on a display unit, the image signal being one of: a first image corresponding to a first area of ​​a live VR image; and a second image corresponding to a second area of ​​the live VR image, and when the first user operation is received: execute control to output an image signal for changing the image displayed on the display unit from one of the first image and the second image to the other of the first image and the second image, or to both the first image and the second image; and output a preparation instruction for executing image preparation processing.

Claims

[1] Image control device, comprising: a control unit configured to receive a first user operation and configured to: in a case where the first user operation is not received, execute control to output an image signal to be displayed on a display unit, the image signal being one of: a first image corresponding to a first region of a live virtual reality image, live VR image, which is a live view image received from an imaging unit; and a second image corresponding to a second area of ​​the live VR image that is enlarged relative to the live VR image, the second area being a smaller area than the first area, and in a case where the first user operation is received: execute control to output an image signal for changing the image displayed on the display unit from one of the first image and the second image to the other of the first image and the second image or to both the first image and the second image; and issue a preparation instruction to perform mapping preparation processing. [2] The image control device according to claim 1, wherein the control unit is further configured to execute control to output the image signal as the first image to be displayed on the display unit in a case where the first user operation is not received. [3] The image control device according to claim 1, wherein the control unit is further configured to execute control to output the image signal as the second image to be displayed on the display unit in a case where the first user operation is not received. [4] The imaging control device according to claim 1, wherein the control unit is further configured to: execute control to output the image signal as the first image to be displayed on the display unit in a case where a first operation mode is set and the first user operation is not received, and execute control to output the image signal as the second image to be displayed on the display unit in a case where a second operation mode is set and the first user operation is not received. [5] The image control device according to any one of claims 1 to 4, wherein the control unit is further configured to execute control to change the image signal to be output for display on the display unit to the other of the first image and the second image in response to receiving the first user operation. [6] The image control device according to any one of claims 1 to 4, wherein the control unit is further configured to execute control to change the image signal to be output for display on the display unit to both the first image and the second image in response to receiving the first user operation. [7] The image control device according to any one of claims 1 to 6, wherein the control unit is further configured to execute control to return the image signal to be output for display on the display unit to one of the first image and the second image in response to completion of reception of the first user operation. [8] The imaging control device according to any one of claims 1 to 7, wherein the first area is an entire area of ​​the live VR image. [9] The imaging control device according to any one of claims 1 to 8, wherein the first user operation is received by an operation unit in the case where a user has performed a first operation on the operation unit, and the control unit is further configured to issue an instruction for instructing the imaging unit to capture a VR image in response to receiving a second user operation performed on the operation unit by a user. [10] The image control device according to claim 9, wherein the first user operation is an operation of half-pressing a shutter button, and the second user operation is an operation of fully pressing the shutter button. [11] The imaging control device according to claim 9, further comprising a touch panel as the operation unit and a touch detection unit configured to be capable of detecting a touch operation performed on the touch panel, wherein the first user operation is a first touch operation performed on a specific display element displayed on the display surface, and the second user operation is a second touch operation performed on the specific display element. [12] The imaging control device according to claim 11, wherein the first touch operation is an operation for starting a contact on the touch-sensitive panel, and the second touch operation is an operation for releasing a contact from the touch-sensitive panel following the first touch operation. [13] The imaging control apparatus according to any one of claims 1 to 12, wherein the live VR image includes a distortion with respect to a real image of an object, and a transformation includes a correction process for correcting the distortion. [14] The image control apparatus according to claim 13, further comprising an image processing unit configured to execute the correction processing. [15] The imaging control apparatus according to claim 14, wherein the image processing unit is configured to perform the correction processing at an intensity based on a distance between a position on the live VR image corresponding to an optical axis of a lens used during imaging and a position on the live VR image corresponding to the second image. [16] The imaging control apparatus according to claim 15, wherein, in a case where the distance does not exceed a threshold, the image processing unit either omits the correction processing or executes the correction processing at an intensity that ensures that the distortion of the live VR image is not changed. [17] The imaging control apparatus according to any one of claims 1 to 16, further comprising an imaging unit including a plurality of imaging devices for respectively capturing a plurality of images constituting the live VR image. [18] The imaging control apparatus according to claim 17, wherein the live VR image is a single image obtained by combining the plurality of images, and in a case where the first user operation has been performed, the control unit executes control to output an image signal to be displayed on the display unit, the second image including a boundary between the plurality of images. [19] The image control device according to any one of claims 1 to 11, further comprising a touch-sensitive panel and a touch detection unit configured to be capable of detecting a touch operation performed on the touch-sensitive panel, wherein, in a case where a touch is performed on the first image displayed on the display unit and then the first user operation is performed, the control unit is configured to execute control to output an image signal to be displayed on the display unit, the second image corresponding to the touched position on the first image [20] The image control device according to any one of claims 1 to 19, wherein, in a case where the first user operation has been performed, the control unit is configured to execute control to output an image signal to be displayed on the display unit, the second image corresponding to a part of the first image in which a predetermined object exists. [21] The image control device according to claim 20, wherein, in a case where a plurality of predetermined objects exist on the live VR image and the first user operation has been performed, the control unit is configured to execute control to output an image signal so that a plurality of second images corresponding to the plurality of predetermined objects are to be displayed in order on the display unit. [22] The image control apparatus according to claim 21, wherein, during the control to display the plurality of second images in order on the display unit, the control unit changes, in response to a second user operation, the image signal being output so that the second image to be displayed on the display unit changes between the plurality of second images. [23] The imaging control apparatus according to any one of claims 1 to 22, wherein the preparation instruction is an instruction to execute at least one of autofocus, exposure adjustment, aperture adjustment, shutter speed adjustment, white balance, and gain adjustment as imaging preparation processing in response to receiving the first user operation. [24] A control method for an imaging control device, comprising: Receiving a first user operation; in a case where the first user operation has not been performed, executing control to output an image signal to be displayed on a display unit, the image signal being one of: a first image corresponding to a first region of a live image of a virtual reality, live VR image, which is a live view image captured by the imaging unit; and a second image corresponding to a second area of ​​the live VR image that is enlarged relative to the live VR image, the second area being a smaller area than the first area, and in a case where the first user operation is received: Executing control for outputting an image signal for changing the image displayed on the display unit from one of the first image and the second image to the other of the first image and the second image or to both the first image and the second image; and Issue a preparation instruction to execute image preparation processing. [25] A program for causing a computer to operate as each unit of the imaging control apparatus according to any one of claims 1 to 23. [26] A computer-readable medium storing a program, the program causing a computer to operate as each unit of the imaging control apparatus according to any one of claims 1 to 23.

Images