Electronic device, control method thereof, and program

Abstract

To reduce the likelihood of a certain symptom occurring in a user watching a VR video.SOLUTION: An electronic device includes image acquisition means that acquires a photographed image, acquisition means that acquires, when the photographed image is acquired by imaging using an imaging device equipped with a twin lens, information which corresponds to the photographed image and is first information relating to a phenomenon that may be a cause of a specific symptom, and control means that correlates the photographed image with the first information and records it in recording means.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an electronic device, a control method thereof, and a program. [Background technology]

[0002] An imaging device may capture an image with parallax (parallax image) using a lens with two optical systems in one lens mount (hereinafter, twin lens). A technology is known that displays the captured parallax image (hereinafter, "VR video") as a VR video with a three-dimensional effect.

[0003] Patent Document 1 describes a technique for determining whether a frame is playable so as not to display an unpleasant image when a user is viewing a VR video. In this technique, the device controls a display unit to display a playable frame instead of a non-playable frame. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-199985 A Summary of the Invention [Problem to be solved by the invention]

[0005] Furthermore, in cases where there is a large difference between the left eye image and the right eye image in a VR video, a user watching the VR video may experience certain symptoms (such as fatigue or VR sickness). However, display devices that display VR video and users watching VR video have not been able to grasp in advance the possibility that a specific symptom may occur in the user. For this reason, VR video that may cause a specific symptom may be displayed and played back, and it has not been easy to reduce the possibility that a user may experience a specific symptom.

[0006] Therefore, an object of the present invention is to provide a technology that contributes to reducing the possibility of certain symptoms occurring in users watching VR videos. [Means for solving the problem]

[0007] One aspect of the present invention is a method for producing a composition comprising the steps of: An image acquisition means for acquiring a captured image; an acquisition means for acquiring, when the captured image is an image acquired by imaging using an imaging device equipped with a twin lens, first information corresponding to the captured image and relating to a phenomenon that may be a cause of the occurrence of a specific symptom; a control means for recording the captured image and the first information in association with each other in a recording means; The electronic device is characterized by having the following features.

[0008] One aspect of the present invention is a method for producing a composition comprising the steps of: an acquisition means for acquiring a captured image and information associated with the captured image, the first information being related to a phenomenon that may be a cause of the occurrence of a specific symptom; a control means for controlling display of the captured image on a display device based on the first information when the captured image is an image captured by an imaging device equipped with a twin lens; The electronic device is characterized by having the following features.

[0009] One aspect of the present invention is a method for producing a composition comprising the steps of: an image acquisition step of acquiring a captured image; an acquisition step of acquiring, when the captured image is an image acquired by imaging using an imaging device equipped with a twin lens, first information corresponding to the captured image and relating to a phenomenon that may be a cause of the occurrence of a specific symptom; a control step of recording the captured image and the first information in association with each other in a recording means; The present invention relates to a method for controlling an electronic device, comprising the steps of:

[0010] One aspect of the present invention is a method for producing a composition comprising the steps of: an acquisition step of acquiring a captured image and first information associated with the captured image, the first information being related to a phenomenon that may be a cause of the occurrence of a specific symptom; a control step of controlling display of the captured image on a display device based on the first information when the captured image is an image captured by an imaging device equipped with a twin lens; The present invention relates to a method for controlling an electronic device, comprising the steps of: Effect of the Invention

[0011] According to the present invention, it is possible to contribute to reducing the possibility of a user watching a VR video developing certain symptoms. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is an external configuration diagram of a camera according to a first embodiment. [Diagram 2] FIG. 1 is a diagram showing the internal configuration of a camera according to a first embodiment. [Diagram 3] FIG. 1 is a diagram illustrating a lens unit according to a first embodiment. [Figure 4] 4 is a flowchart of processing in the camera according to the first embodiment. [Diagram 5] 3 shows the data structure of a moving image file according to the first embodiment. [Figure 6] FIG. 11 is a system configuration diagram according to a second embodiment. [Figure 7] FIG. 11 is a configuration diagram of an HMD according to a second embodiment. [Figure 8] 10 is a flowchart of processing in a camera according to a second embodiment. [Figure 9] 10 is a flowchart of processing of an HMD according to the second embodiment. [Figure 10] FIG. 11 is a diagram illustrating image processing according to the second embodiment. [Figure 11] FIG. 11 is a diagram illustrating the configuration of a personal computer according to a third embodiment. [Figure 12] 13 is a flowchart of processing by a personal computer according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0014] <Embodiment 1> The imaging device (electronic device) according to the first embodiment is realized by a digital camera equipped with a twin lens (VR180 lens) for capturing moving images. The digital camera 100 that captures moving images will be described below.

[0015] 1A and 1B are diagrams showing an example of the external configuration of digital camera (hereinafter, camera) 100. Fig. 1A is a perspective view of camera 100 as seen from the front, and Fig. 1B is a perspective view of camera 100 as seen from the back.

[0016] The camera 100 has, on its top surface, a shutter button 101, a power switch 102, a mode changeover switch 103, a main electronic dial 104, a sub electronic dial 105, a video button 106, and an outside viewfinder display unit 107.

[0017] The shutter button 101 is an operation unit for preparing for shooting or issuing shooting instructions. The power switch 102 is an operation unit for switching the power of the camera 100 on and off. The mode changeover switch 103 is an operation unit for switching between various modes. The main electronic dial 104 is a rotary operation unit for changing settings such as the shutter speed and aperture. The sub electronic dial 105 is a rotary operation unit for moving the selection frame (cursor) and forwarding images. The video button 106 is an operation unit for issuing instructions to start and stop video shooting (recording). The outside viewfinder display unit 107 displays various settings such as the shutter speed and aperture.

[0018] Camera 100 also has, on the rear surface, a display unit 108, a touch panel 109, direction keys 110, a SET button 111, an AE lock button 112, a magnification button 113, a playback button 114, and a menu button 115. Camera 100 also has, on the rear surface, an eyepiece unit 116, an eyepiece detection unit 118, and a touch bar 119.

[0019] The display unit 108 displays images and various information. The touch panel 109 is an operation unit that detects touch operations on the display surface (touch operation surface) of the display unit 108. The directional keys 110 are an operation unit having keys (four-way keys) that can be pressed up, down, left, and right. The camera 100 is capable of control according to the position pressed on the directional keys 110. The SET button 111 is an operation unit that is pressed mainly to confirm a selection item. The AE lock button 112 is an operation unit that is pressed to fix the exposure state in a shooting standby state.

[0020] The enlargement button 113 is an operation unit for switching the enlargement mode on and off in the live view display (LV display) in the shooting mode. When the enlargement mode is on, the live view image (LV image) is enlarged or reduced by operating the main electronic dial 104. The enlargement button 113 is also used to enlarge the playback image or increase the magnification ratio in the playback mode.

[0021] The playback button 114 is an operation unit for switching between a shooting mode and a playback mode. In the shooting mode, pressing the playback button 114 switches to the playback mode, and the latest image among the images recorded in the recording medium 227 can be displayed on the display unit 108.

[0022] When the menu button 115 is pressed, a menu screen on which various settings can be made is displayed on the display unit 108. The user can intuitively make various settings by using the direction keys 110 and the SET button 111 while looking at the menu screen displayed on the display unit 108.

[0023] The eyepiece unit 116 is a portion for placing an eye on an eyepiece finder (a peer-in type finder) 117. A user can view an image displayed on an EVF 217 (Electronic View Finder) through the eyepiece unit 116. The eyepiece detection unit 118 is a sensor that detects whether the user places the eye of the eyepiece unit 116 close to the eyepiece unit 116.

[0024] The touch bar 119 is a line-shaped touch operation unit (line touch sensor) capable of receiving a touch operation. The touch bar 119 is disposed at a position where it can be touched (touched) by the thumb of the right hand when the grip unit 120 is held in the right hand (held with the little finger, ring finger, and middle finger of the right hand) so that the shutter button 101 can be pressed with the index finger of the right hand. That is, the touch bar 119 can be operated in a state (shooting posture) where the eyepiece unit 116 is placed close to the eyepiece 116, the eyepiece viewfinder 117 is looked at, and the user is ready to press the shutter button 101 at any time. The touch bar 119 can receive a tap operation (operation of touching and releasing without moving within a predetermined period of time) on the touch bar 119, a slide operation to the left and right (operation of touching and then moving the touch position while keeping the touch), and the like. The touch bar 119 is an operation unit different from the touch panel 109, and does not have a display function. The touch bar of the first embodiment Reference numeral 119 denotes a multi-function bar, which functions as, for example, an M-Fn bar.

[0025] The camera 100 also includes a grip section 120, a thumb rest section 121, a terminal cover 122, a lid 123, a communication terminal 124, and the like. The grip section 120 is a holding section formed in a shape that is easy to hold with the right hand when the user holds the camera 100. The shutter button 101 and the main electronic dial 104 are arranged at positions that can be operated with the index finger of the right hand when the camera 100 is held by gripping the grip section 120 with the little finger, ring finger, and middle finger of the right hand. In a similar state, the sub electronic dial 105 and the touch bar 119 are arranged at positions that can be operated with the thumb of the right hand. The thumb rest section 121 (thumb standby position) is a grip section provided on the rear side of the camera 100 at a position where it is easy to place the thumb of the right hand that is gripping the grip section 120 when none of the operation sections are being operated. The thumb rest section 121 is made of a rubber member or the like for increasing the holding force (grip feeling).

[0026] Terminal cover 122 protects connectors such as a connection cable that connects camera 100 to an external device. Lid 123 protects recording medium 227 and the slot by closing the slot for storing recording medium 227, which will be described later. Communication terminal 124 is a terminal for communicating with lens unit 200, which will be described later and is detachable from camera 100.

[0027] Fig. 2 is a diagram showing an example of the internal configuration of the camera 100. Note that the same components as those in Fig. 1 are given the same reference numerals and descriptions thereof will be omitted where appropriate. A lens unit 200 is attached to the camera 100.

[0028] First, the lens unit 200 will be described. The lens unit 200 is a type of interchangeable lens that is detachable from the camera 100. The lens unit 200 is, for example, a single lens, and is an example of a normal lens. The lens unit 200 has an aperture 201, a lens 202, an aperture drive circuit 203, an AF (autofocus) drive circuit 204, a lens system control circuit 205, a communication terminal 206, and the like.

[0029] The aperture diameter of the diaphragm 201 is adjustable. The lens 202 is composed of a plurality of lenses. The diaphragm drive circuit 203 adjusts the amount of light by controlling the aperture diameter of the diaphragm 201. The AF drive circuit 204 drives the lens 202 to adjust the focus. The lens system control circuit 205 controls the diaphragm drive circuit 203, the AF drive circuit 204, etc. based on instructions from a system control unit 50, which will be described later.

[0030] The lens system control circuit 205 controls the aperture 201 via an aperture drive circuit 203, and adjusts the focus by displacing the position of the lens 202 via an AF drive circuit 204. The lens system control circuit 205 is capable of communicating with the camera 100. Specifically, the lens system control circuit 205 communicates with a communication terminal 206 of the lens unit 200 and a communication terminal 124 of the camera 100. The communication terminal 206 is a terminal through which the lens unit 200 communicates with the camera 100.

[0031] Next, a description will be given of the camera 100. The camera 100 has a shutter 210, an imaging unit 211, an A / D converter 212, a memory control unit 213, an image processing unit 214, a memory 215, a D / A converter 216, an EVF 217, a display unit 108, and a system control unit 50.

[0032] The shutter 210 is a focal plane shutter. The shutter 210 can freely control the exposure time of the imaging unit 211 based on instructions from the system control unit 50. The imaging unit 211 includes an element (such as a CCD or CMOS element) that converts an optical image into an electrical signal. The imaging unit 211 is an imaging element (image sensor). The imaging unit 211 may have an imaging surface phase difference sensor that outputs defocus amount information to the system control unit 50. A / D conversion The converter 212 converts the analog signal output from the imaging unit 211 into a digital signal.

[0033] The image processing unit 214 performs predetermined processing (pixel interpolation, resizing such as reduction, color conversion, etc.) on data (data from the A / D converter 212 or data from the memory control unit 213). The image processing unit 214 also performs predetermined arithmetic processing using the captured image data. The system control unit 50 performs exposure control and distance measurement control based on the obtained arithmetic results. Specifically, TTL (through-the-lens) AF processing, AE (auto exposure) processing, EF (flash pre-flash) processing, etc. are performed. Furthermore, the image processing unit 214 performs predetermined arithmetic processing using the captured image data, and TTL AWB (auto white balance) processing based on the obtained arithmetic results.

[0034] The image data from the A / D converter 212 is written to the memory 215 via the image processing unit 214 and the memory control unit 213. Alternatively, the image data from the A / D converter 212 is written to the memory 215 via the memory control unit 213 (without via the image processing unit 214). The memory 215 stores image data obtained by the imaging unit 211 and then converted into digital data by the A / D converter 212. The memory 215 stores image data to be displayed on the display unit 108 or the EVF 217. The memory 215 has a storage capacity sufficient to store a predetermined number of still images, or a storage capacity sufficient to store moving images or audio for a predetermined period of time. The memory 215 also serves as a memory for displaying images (video memory).

[0035] The D / A converter 216 converts the image display data stored in the memory 215 into an analog signal and supplies it to the display unit 108 and the EVF 217. Therefore, the image data for display written in the memory 215 is supplied to the display unit 108 and the EVF 217 via the D / A converter 216. The display unit 108 and the EVF 217 perform display according to the analog signal from the D / A converter 216. The display unit 108 and the EVF 217 are, for example, displays such as an LCD or an organic EL. The digital signal that has been A / D converted by the A / D converter 212 and stored in the memory 215 is converted into an analog signal by the D / A converter 216. Thereafter, the analog signal is sequentially transferred to the display unit 108 and / or the EVF 217, and is displayed on the display unit 108 and / or the EVF 217. In this manner, a live view display is performed.

[0036] The system control unit 50 is a control unit that includes at least one processor and / or at least one circuit. That is, the system control unit 50 may be a processor, a circuit, or a combination of a processor and a circuit.

[0037] The system control unit 50 controls the entire camera 100. The system control unit 50 realizes each process of the flowchart described below by executing a program recorded in the non-volatile memory 219. The system control unit 50 also performs display control by controlling the memory 215, the D / A converter 216, the display unit 108, the EVF 217, etc.

[0038] The camera 100 also includes a system memory 218 , a non-volatile memory 219 , a system timer 220 , a communication unit 221 , an attitude detection unit 222 , and an eye proximity detection unit 118 .

[0039] For example, a RAM is used as the system memory 218. In the system memory 218, constants and variables for the operation of the system control unit 50, programs read from the non-volatile memory 219, and the like are developed.

[0040] The non-volatile memory 219 is an electrically erasable and recordable memory. For example, an EEPROM is used as the non-volatile memory 219. The non-volatile memory 219 includes a system control The system timer 220 is a timing unit that measures the time used for various controls and the time of a built-in clock.

[0041] The communication unit 221 transmits and receives video signals or audio signals to and from an external device connected wirelessly or via a wired cable. The communication unit 221 can also connect to a wireless LAN (Local Area Network) and the Internet. The communication unit 221 can also communicate with an external device via Bluetooth (registered trademark) or Bluetooth Low Energy. The communication unit 221 can transmit images (including live images) captured by the imaging unit 211 or images recorded in the recording medium 227. The communication unit 221 can receive image data or various other information from an external device.

[0042] The attitude detection unit 222 detects the attitude of the camera 100 with respect to the direction of gravity. Based on the attitude detected by the attitude detection unit 222, it is possible to determine whether the image captured by the imaging unit 211 is an image captured by holding the camera 100 horizontally or vertically. The system control unit 50 can add orientation information corresponding to the attitude detected by the attitude detection unit 222 to the image file of the image captured by the imaging unit 211. The system control unit 50 can also rotate the image according to the attitude detected by the attitude detection unit 222 and then record the rotated image. For example, an acceleration sensor or a gyro sensor can be used for the attitude detection unit 222. It is also possible to detect the movement of the camera 100 (panning, tilting, lifting, whether or not it is stationary, etc.) using the attitude detection unit 222.

[0043] The eyepiece detection unit 118 can detect the approach of an object to the eyepiece unit 116 of the eyepiece finder 117 incorporating the EVF 217. For example, an infrared proximity sensor can be used for the eyepiece detection unit 118. When an object approaches, infrared rays projected from a light projecting unit of the eyepiece detection unit 118 are reflected by the object and received by a light receiving unit of the infrared proximity sensor. The distance from the eyepiece unit 116 to the object can be determined based on the amount of infrared rays received. In this way, the eyepiece detection unit 118 performs eyepiece detection to detect the proximity of the object to the eyepiece unit 116. The eyepiece detection unit 118 is an eyepiece detection sensor that detects the approach (approach) and departure (away) of an eye (object) to the eyepiece unit 116 of the eyepiece finder 117.

[0044] The eye contact detection unit 118 detects that an eye has been placed near the eyepiece unit 116 when an object approaching within a predetermined distance from the non-eye contact state (non-approaching state) is detected. On the other hand, the eye contact detection unit 118 detects that an object that was detected as approaching from the eye contact state (approaching state) has moved away by more than a predetermined distance. The threshold value for detecting the eye contact and the threshold value for detecting the eye moving away may be different, for example, by providing a hysteresis. Furthermore, after the eye contact is detected, the eye contact state is assumed to remain until the eye moving away is detected. After the eye moving away is detected, the non-eye contact state is assumed to remain until the eye contact is detected.

[0045] The system control unit 50 switches between display (display state) / non-display (non-display state) of the display unit 108 and the EVF 217 depending on the state detected by the eye proximity detection unit 118. Specifically, when at least the shooting standby state is present and the display destination switching setting is automatic switching, the display destination is set to the display unit 108 when the eye is not placed in contact with the camera, the display is turned on, and the EVF 217 is hidden. Also, when the eye is placed in contact with the camera, the display destination is set to the EVF 217, the display is turned on, and the display unit 108 is hidden. Note that the eye proximity detection unit 118 is not limited to an infrared proximity sensor, and other sensors may be used as long as they can detect a state that can be regarded as being in contact with the eye.

[0046] The camera 100 also includes an outside-finder display unit 107, an outside-finder display drive circuit 223, a power supply control unit 224, a power supply unit 225, a recording medium I / F 226, an operation unit 228, and a video signal output unit 229. It has a power I / F.

[0047] The outside-finder display unit 107 displays various settings (such as shutter speed or aperture) of the camera 100 via an outside-finder display drive circuit 223. The power supply control unit 224 includes a battery detection circuit, a DC-DC converter, and a switch circuit (a circuit for switching between blocks to which electricity is applied). The power supply control unit 224 detects whether a battery is installed, the type of battery, and the remaining battery level. The power supply control unit 224 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50 to supply the required voltage to each unit (including the recording medium 227) for the required period of time.

[0048] Power supply unit 225 is a primary battery (such as an alkaline battery or a lithium battery), a secondary battery (such as a NiCd battery, a NiMH battery, or a Li battery), an AC adapter, or the like. Recording medium I / F 226 is an interface with recording medium 227 (such as a memory card or a hard disk). Recording medium 227 is, for example, a memory card for recording captured images. Recording medium 227 includes a semiconductor memory, a magnetic disk, or the like. Recording medium 227 may be detachable from camera 100, or may be built into camera 100.

[0049] The operation unit 228 is an input unit (operation member) that accepts operations from the user (user operations). The operation unit 228 is used to input various instructions to the system control unit 50. The operation unit 228 includes the shutter button 101, the power switch 102, the mode switching switch 103, the touch panel 109, and other operation units 229. The other operation units 229 include the main electronic dial 104, the sub electronic dial 105, the video button 106, the direction key 110, the SET button 111, the AE lock button 112, the enlargement button 113, the playback button 114, the menu button 115, the touch bar 119, and the like.

[0050] The shutter button 101 has a first shutter switch 230 and a second shutter switch 231. The first shutter switch 230 is turned on during the operation of the shutter button 101 (so-called half-press (shooting preparation instruction)) and generates a first shutter switch signal SW1. In response to the generation of the first shutter switch signal SW1, the system control unit 50 starts a shooting preparation process such as an AF process, an AE process, an AWB process, or an EF process. The second shutter switch 231 is turned on when the operation of the shutter button 101 is completed (so-called full press (shooting instruction)) and generates a second shutter switch signal SW2. In response to the generation of the second shutter switch signal SW2, the system control unit 50 starts a series of shooting processes (processes from reading out a signal from the imaging unit 211 to generating an image file including a shot image and writing it to the recording medium 227).

[0051] The mode changeover switch 103 changes the operation mode of the system control unit 50 to one of a still image shooting mode, a video shooting mode, a playback mode, and the like. Modes included in the still image shooting mode include an auto shooting mode, an auto scene determination mode, a manual mode, an aperture priority mode (Av mode), a shutter speed priority mode (Tv mode), and a program AE mode (P mode). Modes included in the still image shooting mode include various scene modes and custom modes that are shooting settings according to shooting scenes. A user can directly switch to one of the above-mentioned shooting modes by using the mode changeover switch 103. Alternatively, a user can selectively switch to one of the displayed modes by using the operation unit 228 after once switching to a list screen of shooting modes by using the mode changeover switch 103. Similarly, the video shooting mode may also include a plurality of modes.

[0052] The touch panel 109 is a surface of the display unit 108 (the operation surface of the touch panel 109). It is a touch sensor that detects various touch operations. Touch panel 109 and display unit 108 can be configured as one unit. For example, touch panel 109 is attached to the upper layer of the display surface of display unit 108 so as not to impede the transmission of light emitted from display unit 108 (so that the transmittance of light does not impede the display of display unit 108). Then, input coordinates on touch panel 109 are associated with display coordinates on the display surface of display unit 108. This makes it possible to configure a GUI (Graphical User Interface) that makes it seem as if the user can directly operate the screen displayed on display unit 108.

[0053] The touch panel 109 can use any of various types such as a resistive film type, a capacitive type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, and an optical sensor type. Some types detect a touch by contact with the touch panel 109, and others detect a touch by the approach of a finger or a pen to the touch panel 109, and either type may be used.

[0054] The system control unit 50 can detect the following operations or states on the touch panel 109. A finger or pen that has not been touching the touch panel 109 touches the touch panel 109 again, that is, the start of touching (hereinafter referred to as Touch-Down). A state in which the touch panel 109 is touched with a finger or a pen (hereinafter referred to as Touch-On). The touch panel 109 is moved while being touched by a finger or a pen (hereinafter referred to as Touch-Move). The finger or pen that has been touching the touch panel 109 is released from the touch panel 109, that is, the touch ends (hereinafter, referred to as "touch-up"). A state in which nothing is being touched on the touch panel 109 (hereinafter referred to as Touch-Off).

[0055] When touch-down is detected, touch-on is also detected at the same time. After touch-down, touch-on will usually continue to be detected unless touch-up is detected. If touch-move is detected, touch-on will also be detected at the same time. Even if touch-on is detected, touch-move will not be detected if the touch position does not move. After it is detected that all fingers or pens that were touching have touched up, touch-off will occur.

[0056] These operations and states, and the position coordinates of the touch panel 109 touched by a finger or pen are notified to the system control unit 50 through the internal bus. The system control unit 50 determines what kind of operation (touch operation) has been performed on the touch panel 109 based on the notified information. For touch-move, the moving direction of the finger or pen moving on the touch panel 109 can also be determined for each vertical component and horizontal component on the touch panel 109 based on the change in the position coordinates. If a touch-move of a predetermined distance or more is detected, it is determined that a slide operation has been performed. An operation in which a finger is touched on the touch panel 109, quickly moved for a certain distance, and then released is called a flick. In other words, a flick is an operation in which a finger is quickly traced on the touch panel 109 as if flicking it. If a touch-move of a predetermined distance or more at a predetermined speed or more is detected and a touch-up is detected as it is, it is determined that a flick has been performed (it can be determined that a flick has occurred following a slide operation). Furthermore, a touch operation in which multiple points (for example, two points) are touched together (multi-touch) and the touch positions are brought closer together is called a pinch in, and a touch operation in which the touch positions are moved farther apart is called a pinch out. Pinch out and pinch in are collectively called pinch operations (or simply pinch).

[0057] Fig. 3 is a schematic diagram showing an example of the configuration of the lens unit 300. Fig. 6A shows a state in which the lens unit 300 is attached to the camera 100.

[0058] Lens unit 300 is a type of interchangeable lens that can be attached to and detached from camera 100. Lens unit 300 is a twin lens that enables shooting with parallax between left and right images. Lens unit 300 has two optical systems, each with a wide viewing angle of approximately 180 degrees, and can shoot the range of the front hemisphere. Specifically, the two optical systems of lens unit 300 can each shoot a subject with a viewing field (angle of view) of 180 degrees in the left-right direction (horizontal angle, azimuth angle, yaw angle) and 180 degrees in the up-down direction (vertical angle, elevation angle, pitch angle).

[0059] The lens unit 300 has a right-eye optical system 301R, a left-eye optical system 301L, and a lens system control circuit 303. The right-eye optical system 301R and the left-eye optical system 301L each have a plurality of lenses, a reflecting mirror, and the like. The right-eye optical system 301R corresponds to an example of a first optical system, and the left-eye optical system 301L corresponds to an example of a second optical system. The lens 302R located on the subject side of the right-eye optical system 301R and the lens 302L located on the subject side of the left-eye optical system 301L face in the same direction. The optical axes of the two lenses 302R and 302L are approximately parallel.

[0060] The lens unit 300 of the first embodiment is a lens for VR180 (a lens for capturing images for so-called VR180, which is a format of VR images that allows two-eye stereoscopic vision). The lens for VR180 has a fisheye lens that allows the right-eye optical system 301R and the left-eye optical system 301L to capture a range of approximately 180 degrees. It is sufficient that the right-eye optical system 301R and the left-eye optical system 301L can each acquire an image that allows two-eye VR display as VR180, and the lens for VR180 may be a lens that can capture a wide viewing angle range of about 160 degrees, which is narrower than the range of 180 degrees. The lens for VR180 can form a right image (first image) formed via the right-eye optical system 301R and a left image (second image) formed via the left-eye optical system 301L having parallax from the right image, on one or two imaging elements of the attached camera.

[0061] Furthermore, the lens unit 300 is attached to the camera 100 via the lens mount portion 304 and the camera mount portion 305 of the camera 100. This electrically connects the system control portion 50 of the camera 100 and the lens system control circuit 303 of the lens unit 300 via the communication terminal 124 of the camera 100 and the communication terminal 306 of the lens unit 300.

[0062] In the first embodiment, a right image formed through the right eye optical system 301R and a left image formed through the left eye optical system 301L having a parallax from the right image are imaged side by side on the imaging unit 211 of the camera 100. That is, two optical images formed by the right eye optical system 301R and the left eye optical system 301L are formed on one imaging element. The imaging unit 211 converts the imaged subject image (optical signal) into an analog electrical signal. In this way, by using the lens unit 300, two images having parallax can be simultaneously acquired (as a set) from two locations (optical systems) of the right eye optical system 301R and the left eye optical system 301L. In addition, by dividing the acquired image into an image for the left eye and an image for the right eye and displaying them in VR, the user can view a stereoscopic VR image (so-called VR180) in a range of approximately 180 degrees.

[0063] Here, a VR image is an image that can be displayed in VR, as described later. VR images include omnidirectional images (spherical images) taken with an omnidirectional camera (spherical camera) and panoramic images with a wider image range (effective image range) than the display range that can be displayed at one time on a display unit. VR images are not limited to still images, but also include videos and live images (images that are almost real-time from the camera). The VR image includes an image range (effective image range) of up to 360 degrees in the left-right direction and 360 degrees in the up-down direction. The VR image also includes an image having a wider angle of view than the angle of view that can be captured by a normal camera, or a wider image range than the display range that can be displayed at one time on a display unit, even if the angle of view is less than 360 degrees in the left-right direction or less than 360 degrees in the up-down direction. The image captured by the camera 100 using the lens unit 300 described above is a type of VR image. The VR image can be displayed in VR by, for example, setting the display mode of a display device (a display device that can display VR images) to "VR view." By displaying a VR image having a 360-degree angle of view in VR and changing the attitude of the display device in the left-right direction (horizontal rotation direction), the user can view a seamless omnidirectional image in the left-right direction.

[0064] Here, VR display (VR view) is a display method (display mode) capable of changing the display range, which displays an image of a VR image with a field of view corresponding to the posture of the display device. VR display includes "single-eye VR display (single-eye VR view)" in which a VR image is transformed by mapping the VR image onto a virtual sphere (transformation in which distortion correction is performed) to display one image. VR display also includes "two-eye VR display (two-eye VR view)" in which a VR image for the left eye and a VR image for the right eye are transformed by mapping them onto a virtual sphere, respectively, and displayed side by side in the left and right regions. Stereoscopic vision is possible by performing "two-eye VR display" using a VR image for the left eye and a VR image for the right eye that have parallax from each other. In any VR display, for example, when a user wears a display device such as an HMD (head mounted display), an image with a field of view corresponding to the orientation of the user's face is displayed. For example, assume that a VR image with a field of view centered on 0 degrees left and right (a specific direction, for example, north) and 90 degrees up and down (90 degrees from the zenith, i.e. horizontal) is displayed at a certain point in time. When the attitude of the display device is flipped from this state (for example, the display surface is changed from facing south to facing north), the display range of the same VR image is changed to an image with a field of view centered on 180 degrees in the left-right direction (the opposite direction, for example, south) and 90 degrees in the up-down direction. That is, when the user faces from north to south (i.e. turns backwards) while wearing the HMD, the image displayed on the HMD is also changed from a north image to a south image. Note that the VR image captured using the lens unit 300 of the first embodiment is a VR180 image capturing a range of approximately 180 degrees in front, and no image exists in a range of approximately 180 degrees behind. When such an image of VR180 is displayed in VR and the attitude of the display device is changed to the side where no image exists, a blank area is displayed.

[0065] By displaying the VR image in this way, the user visually feels as if he or she is inside the VR image (in the VR space). The method of displaying the VR image is not limited to changing the posture of the display device. For example, the display range may be moved (scrolled) in response to user operation via a touch panel or directional buttons. Furthermore, during VR display (when in the "VR view" display mode), in addition to changing the display range due to posture changes, the display range may also be changed in response to touch-move on the touch panel, dragging with a mouse, pressing a directional button, etc. A smartphone attached to VR goggles (head-mounted adapter) is a type of HMD.

[0066] The process (information processing; information processing method) of the camera 100 according to the first embodiment will be described with reference to the flowchart of Fig. 4. In the flowchart of Fig. 4, the camera 100 acquires information related to the shaking of the camera 100 (such as the degree of camera shake during shooting) as information related to a phenomenon that causes VR sickness (hereinafter referred to as "phenomenon information").

[0067] In the flowchart of FIG. 4, the process starts when a user who operates the camera 100 (hereinafter referred to as an "operator") turns on the power of the camera 100.

[0068] In step S401, the system control unit 50 detects the lens ( In the following, information about the attached lens (such as design values ​​or individual values) is obtained. Information about the attached lens is referred to as "lens information."

[0069] In step S402, the system control unit 50 acquires (performs image acquisition) a captured image from the imaging unit 211. Here, the captured image acquired by imaging with the camera 100 equipped with a VR180 lens includes a right-eye image for the user's right eye and a left-eye image for the user's left eye.

[0070] In step S403, the system control unit 50 displays on the EVF 217 the captured image acquired in step S402.

[0071] In step S404, the system control unit 50 determines whether or not a recording start instruction (an instruction to record a captured image) has been issued by the operator operating the shutter button 101. If it is determined that a recording start instruction has been issued, the process proceeds to step S405. If it is determined that a recording start instruction has not been issued, the process returns to step S402. Therefore, live view display is repeated on the EVF 217 until a recording start instruction is issued.

[0072] The process of steps S405 to S412 is executed for a captured image currently acquired by imaging section 211 (camera 100). That is, the process of steps S405 to S412 is executed for each captured image (each frame of a moving image).

[0073] In step S405, the system control unit 50 acquires a captured image (a captured image currently acquired by imaging by the imaging unit 211) from the imaging unit 211. When capturing a moving image, the system control unit 50 also acquires audio at the same time.

[0074] In step S406, the system control unit 50 acquires shooting information (such as the shutter speed at the time of shooting) and metadata (such as the camera name and lens name) indicating information related to the camera 100. When a RAW image is shot, the metadata includes information necessary for development.

[0075] In step S407, the system control unit 50 records the captured image acquired in step S405 in a file (storage area). When a moving image is captured, the system control unit 50 also records audio in the file at the same time. The file is stored in the recording medium 227, for example.

[0076] In step S408, the system control unit 50 records the shooting information and metadata acquired in step S406 in a file. In the first embodiment, the system control unit 50 records the shooting information and metadata in the same file as the captured image.

[0077] In step S409, the system control unit 50 records the lens information acquired in step S401 in a file. In the first embodiment, the system control unit 50 records the lens information in the same file as the captured image.

[0078] In this way, in steps S407 to S409, the captured image, shooting information, metadata, and lens information are recorded in the same file. Therefore, the captured image, shooting information, metadata, and lens information are recorded in association with each other (see FIG. 5).

[0079] In step S410, the system control unit 50 determines whether the attached lens is a VR180 lens (twin lens) (the captured image is an image acquired by imaging with the camera 100 equipped with a VR180 lens). If it is determined that the attached lens is a VR180 lens, If so, the process proceeds to step S411. If it is determined that the attached lens is not a VR180 lens, the process proceeds to step S413.

[0080] In step S411, the system control unit 50 acquires information (motion information) relating to the movement of the camera 100 from the attitude detection unit 222. Specifically, the system control unit 50 acquires, as the motion information, the amount of change in the current attitude of the camera 100 from the attitude of the camera 100 one frame before, from an acceleration sensor or the like.

[0081] In step S412, the system control unit 50 converts the movement information of the camera 100 into information (shake information) on the degree of shaking (level of the phenomenon), and records the shake information in a file. As described above, the shake information is information on a phenomenon (phenomenon information) that can be a cause of VR sickness. In the first embodiment, the system control unit 50 records the shake information in the same file as the captured image in association with the captured image. If the attached lens is not a VR180 lens (twin lens), the system control unit 50 does not record the shake information in a file.

[0082] The blur information may be recorded to indicate whether the degree of blur is "large," "medium," "small," or "none" depending on the magnitude of the motion information acquired in step S411. The degree of blur may also be expressed as a numerical value, for example, 10 when the degree of blur is large and 0 when there is no blur. The system control unit 50 may record the motion information (the amount of change in the attitude of the camera 100) as is in a file, and the motion information may be converted into blur information in another device that handles the file.

[0083] 5 shows an example of the internal data structure of a file (movie file) that records captured images, etc. The movie file is stored in the memory 215 or the recording medium 227, etc. The movie file includes a header 501 that stores information such as a file format, shooting information 502, metadata 503 related to VR180, metadata 504 unrelated to VR180, and data 505 for each frame of the movie.

[0084] Metadata 503 includes information (such as the lens name and lens individual value information) only when a VR180 lens is attached to camera 100. Data 505 includes captured images (image data), audio (audio data), and metadata for each frame. Blur information is recorded in the metadata of data 505 for each frame. As a result, various data are recorded in the video file so that frames with a large degree of blur can be identified.

[0085] In step S413, the system control unit 50 determines whether or not an instruction to end recording (an instruction to end recording of captured images) has been given by the operator. In the first embodiment, the operator of the camera 100 gives the instruction to end recording by pressing the shutter button 101 again. If it is determined that an instruction to end recording has been given, the process of this flowchart ends. If it is determined that an instruction to end recording has not been given, the process proceeds to step S405. In this case, the processes of steps S405 to S412 are repeated, allowing captured images (images of each frame of the video) to be recorded one after another in a file.

[0086] In step S412, the system control unit 50 may acquire any phenomenon information. For example, the system control unit 50 may control the image processing unit 214 to detect the difference between the left eye image and the right eye image (image magnification difference, vertical misalignment, rotational misalignment, luminance difference, contrast difference, color difference) as phenomenon information (phenomenon that may cause VR sickness). The difference between the left eye image and the right eye image can be detected by general image processing. For example, the image processing unit 214 can acquire the luminance difference between the two images by comparing the average luminance in a predetermined area of ​​the left eye image and a predetermined area of ​​the right eye image.

[0087] Also, immediately after the process of the flowchart in FIG. 4 is completed (immediately after the video recording is completed), the system control unit 50 may record summary information in the metadata 503 based on the blur information (phenomenon information) in the metadata of each frame. The summary information is, for example, aggregate information on the level of a phenomenon that may cause VR sickness (hereinafter referred to as the "phenomenon level"). The summary information includes, for example, information on the maximum value of the phenomenon level (degree of blur) or information on a section where the phenomenon level is high. This allows the display device that has received the video file to quickly obtain, when displaying a captured image, information on the phenomenon level that causes VR sickness and information on a section where VR sickness may occur from the video file.

[0088] Furthermore, in step S412, if the system control unit 50 determines that the phenomenon level (degree of blur) is equal to or greater than a predetermined level, the system control unit 50 may issue a warning (notification) to the operator to prompt the operator to reshoot in order to obtain a new video (captured image). In this case, the system control unit 50 issues this warning during or immediately after recording the video.

[0089] When displaying a list of multiple videos as thumbnails, the system control unit 50 may superimpose an icon (such as text "VR Sickness: High" or "VR Sickness: Low") indicating the level of VR sickness expected to occur on each thumbnail. This may allow the operator to visually confirm at a glance which videos are likely to cause VR sickness. The level of VR sickness expected to occur is correlated with phenomenon information (such as the degree of blur or the degree of difference between the right eye image and the left eye image). Therefore, the system control unit 50 can determine the level of VR sickness expected to occur based on the phenomenon information. In addition, the system control unit 50 may display details of the phenomenon information (such as the type of phenomenon causing VR sickness, the phenomenon level, and the VR sickness occurrence period) as the shooting information display of the video in the camera 100.

[0090] In addition, there is a risk of blurring immediately after the start of movie recording and immediately before the end of movie recording due to pressing of the shutter button 101. For this reason, the system control unit 50 may perform control so that movie recording is not performed immediately after an instruction to start movie recording and immediately before an instruction to end movie recording is given.

[0091] Furthermore, in the first embodiment, the metadata is recorded in the video file, but it may be stored in a separate file (sidecar file) related to (corresponding to) the video file.

[0092] In this flowchart, a moving image (video) has been described, but a still image may be used instead of the moving image. In this case, only the captured image and metadata of the first frame are recorded in the data 505 for each frame as shown in FIG.

[0093] The captured images may be recorded in the general MP4 video or JPEG still image format, or in the RAW video or RAW still image format.

[0094] Thus, in the first embodiment, the camera records information on camera shake or information on the difference between the left eye image and the right eye image (brightness difference, contrast difference, color difference, etc.) as phenomenon information in a file (recording medium) when capturing an image. This allows the display device that displays a moving image, which is a VR video, to read the phenomenon information from the file along with the captured image and appropriately determine whether or not the user is likely to experience VR sickness based on the phenomenon information. Then, the display device can appropriately control the display of the moving image (captured image) based on the phenomenon information. This makes it possible to prevent the user from experiencing VR sickness when watching a moving image.

[0095] <Embodiment 2> The system according to the second embodiment includes a camera equipped with a twin lens (VR180 lens) and an HMD that displays a video file recorded by the camera. In the second embodiment, the HMD is Then, the HMD judges whether the recorded video file was recorded with a VR180 lens attached to the camera. If the HMD judges that the video file was recorded with a VR180 lens attached to the camera, the HMD controls the display state of the video (captured image) based on the phenomenon information in the video file. Note that since the same reference numerals as those in the first embodiment perform the same operations or processes as those in the first embodiment, the description will be omitted.

[0096] 6A and 6B are diagrams showing the system configuration of the camera 100 and HMD 600 according to embodiment 2. In Fig. 6A and 6B, the configurations of the camera 100 and the lens unit 300 (VR180 lens) are the same as those in embodiment 1.

[0097] 6A shows a system configuration in which a camera 100 and an HMD 600 according to the second embodiment are connected and communicate with each other. The HMD 600 displays an image (moving image or still image) captured by the camera 100. Communication 610 is general wireless communication or wired communication.

[0098] 6B shows the configuration of a system that provides (outputs) a file 620 recorded by the camera 100 according to the second embodiment to the HMD 600. The file 620 is a moving image file (see FIG. 5) recorded by the camera 100. The file 620 may also be a still image file.

[0099] 7 is a block diagram showing the configuration of the HMD 600. The HMD 600 is a display device (electronic device) that displays captured images. The HMD 600 has a control unit 701, a ROM 702, a RAM 703, an external storage device 704, an operation unit 705, a display unit 706, a communication unit 707, an external interface (I / F) 708, and a system bus 709.

[0100] The control unit 701 controls the entire HMD 600. The control unit 701 is, for example, a Central Processing Unit (CPU).

[0101] The ROM 702 is a read only memory that stores programs and parameters that do not require modification. The ROM 702 is a read only memory (ROM). A predetermined information processing program (program code readable by the control unit 701) is stored in the ROM 702, and the control unit 701 executes this program code.

[0102] The RAM 703 is a random access memory (RAM) that temporarily stores programs and data supplied from an external device or the like.

[0103] The external storage device 704 is a hard disk or a flash memory installed inside the HMD 600. Alternatively, the external storage device 704 includes a memory card that is detachable from the HMD 600.

[0104] The operation unit 705 includes operation members such as buttons that accept user operations. The display unit 706 displays images. The display unit 706 has a left eye display unit (a display unit for the left eye to see) and a right eye display unit (a display unit for the right eye to see).

[0105] The communication unit 707 is a communication unit for connecting to the camera 100. The external I / F 708 transmits and receives video signals and files (such as the video file shown in FIG. 5) to and from an external device such as the camera 100. The system bus 709 is a system bus that connects each component so that they can communicate with each other.

[0106] The moving image file recorded by the camera 100 is written to the external storage device 704.

[0107] Fig. 8 is a flowchart regarding recording of a moving image file by the camera 100 according to embodiment 2. Steps S801 to S809 in Fig. 8 are similar to steps S401 to S409 in Fig. 4. Steps S810 to S811 are similar to steps S411 to S412. Therefore, a description of steps S801 to S811 will be omitted.

[0108] In step S812, the system control unit 50 determines whether or not the operator has issued an instruction to end recording. If it is determined that an instruction to end recording has been issued, the process of this flowchart ends. If it is determined that an instruction to end recording has not been issued, the process proceeds to step S805.

[0109] The operation of the HMD 600 will be described with reference to the flowchart in Fig. 9. Note that a user who wears the HMD 600 on his / her head and watches a video will be referred to as a "viewer" hereinafter.

[0110] The process of the flowchart in Fig. 9 starts when the HMD 600 reads a moving image file (for example, acquires a moving image file from the camera 100 or the external storage device 704). The HMD 600 can use the read moving image file to play and display a moving image using a specific application, start playing the moving image, and stop playing the moving image. The configuration of the moving image file is assumed to be the same as that shown in Fig. 5 according to the first embodiment. That is, in the moving image file, a captured image and blur information are associated with each other.

[0111] In step S901, the control unit 701 acquires the shooting information 502, the metadata 503, and the metadata 504 from the loaded moving image file.

[0112] In step S902, the control unit 701 determines whether or not a playback start instruction for a video has been issued by the viewer. If it is determined that a playback start instruction has been issued, the process proceeds to step S903. If it is determined that a playback start instruction has not been issued, the process of step S902 is repeated.

[0113] In step S903, the control unit 701 obtains the data (captured image, audio, and metadata) of the next frame from the data 505 of the loaded moving image file.

[0114] In step S904, the control unit 701 determines whether the loaded video file was shot and recorded by a camera equipped with a VR180 lens. If it is determined that the video file was shot and recorded by a camera equipped with a VR180 lens, the process proceeds to step S905. If it is determined that the video file was not shot and recorded by a camera equipped with a VR180 lens, the process proceeds to step S907.

[0115] In step S905, the control unit 701 acquires blur information as phenomenon information from metadata of the data of the next frame from the data 505 of the moving image file acquired in step S903.

[0116] In step S906, the control unit 701 determines whether the degree of blur is equal to or greater than a predetermined level based on the blur information acquired in step S905. That is, the control unit 701 determines (detects) whether the user viewing the captured image is likely to experience VR sickness. If it is determined that the degree of blur is equal to or greater than the predetermined level (if it is detected that the user is likely to experience VR sickness), the process proceeds to step S908. If it is determined that the degree of blur is less than the predetermined level, the process proceeds to step S907.

[0117] In step S907, the control unit 701 controls the Of the data 505, a captured image of the next frame of data is displayed on the display unit 706. For example, the control unit 701 applies a decoding process to a captured image encoded in a format such as H.264. The control unit 701 converts the decoded captured image into an image conforming to equirectangular projection. Then, the control unit 701 displays, on the display unit 706, a captured image obtained by performing perspective projection conversion on the captured image conforming to the equirectangular projection.

[0118] In this case, when the decoding process is performed, a twin-eye circular fisheye image is generated, such as image 1001 shown in Fig. 10A. When the captured image is converted to conform to equirectangular projection, an image such as image 1002 is generated, as shown in Fig. 10B. When the captured image is subjected to perspective projection conversion, a part of image 1002 is displayed in a deformed manner according to the attitude of the HMD 600. Note that in step S907, playback processing of the audio data of the next frame is also performed at the same time.

[0119] In step S908, the control unit 701 issues a warning (notification) because a certain level of blurring occurs in the captured image of the next frame data among the data 505 of the video file acquired in step S903. The control unit 701 displays, for example, a predetermined image indicating the warning (an image indicating the possibility of VR sickness) on the display unit 706. Note that the control unit 701 may issue the warning by audio. Furthermore, the control unit 701 stops the playback display of the video before displaying the captured image of the next frame data.

[0120] Then, when the process of step S908 ends, the process proceeds to step S902. Therefore, the playback of the video is stopped until another instruction to start playing the video is given.

[0121] In step S909, the control unit 701 determines whether or not to end the playback of the moving image. If the captured image of the final frame of the moving image file acquired in step S903 has been displayed, or if an instruction to end playback is given by the viewer, the control unit 701 determines to end the playback of the moving image. If it is determined that the playback of the moving image is to end, the process of this flowchart ends. If it is determined that the playback of the moving image is not to end, the process proceeds to step S903. That is, by repeating the processes of steps S903 to S907, each frame of the moving image can be displayed in sequence.

[0122] In the second embodiment, in step S908, the control unit 701 stops the playback of the video. However, the control unit 701 may skip (not play) only frames with a certain degree of blurring or more, and continue display playback from the other frames. The control unit 701 may also display on the display unit 706 the level of VR sickness that is expected to occur in the user who viewed the captured image. For example, the control unit 701 determines the level of VR sickness that is expected to occur based on the phenomenon information. Then, when the level of VR sickness that is expected to occur (or the level of the phenomenon information) is high (higher than a certain level), the control unit 701 may superimpose an icon indicating the level of VR sickness (for example, the text "VR sickness: large") on the video.

[0123] Alternatively, the control unit 701 may stop playing the video if the degree of blur is equal to or greater than the first threshold, and may continue playing the video while skipping the next frame if the degree of blur is equal to or greater than the second threshold and less than the first threshold. This allows the playback of the video to be stopped if the level of VR sickness expected to occur is high, and allows the playback of the video itself to be continued if the level of VR sickness expected to occur is not so high. In other words, it is possible to achieve harmony between the playback of the VR video and the reduction of the possibility of VR sickness.

[0124] In step S906, the control unit 701 determines a threshold value (predetermined level) for determining the degree of blurring according to the type and function of the display device that displays the moving image. For example, since the display device according to the second embodiment is an HMD, even slight shaking may lead to VR sickness in the viewer. For this reason, the control unit 701 sets the threshold value low. On the other hand, when the display device is a general display, there is a certain distance between the display device and the viewer, so that even if the viewer perceives slight shaking in the video, it is highly likely that the viewer will not experience VR sickness. For this reason, the control unit 701 sets the threshold value higher than when the display device is an HMD. Also, when the HMD has an anti-shake function for the video and can apply a certain anti-shake process to the reproduced video, it is possible to cancel out slight shaking by the anti-shake process. For this reason, the control unit 701 sets the threshold value lower than when the display device is an HMD without an anti-shake function.

[0125] Also, similar to the first embodiment, when the HMD 600 displays thumbnails of multiple videos in a list, an icon indicating the level of VR sickness (for example, text such as "VR sickness: large" or "VR sickness: small") may be superimposed on each thumbnail. Also, when the HMD 600 displays shooting information of a video, details of phenomenon information (type and degree of VR sickness, occurrence period, etc.) may be displayed.

[0126] Furthermore, there is a possibility that blurring may occur immediately after the start and immediately before the end of video recording due to the operation of starting and ending video recording (pressing the shutter button 101) on the camera 100. For this reason, when playing back a video on the HMD 600, the first few frames and the last few frames of the video may not be displayed.

[0127] In this way, according to the second embodiment, when playing back a video (VR video) on a display device, phenomenon information such as camera shake is acquired, and if the phenomenon level is above a certain level, the playback state on the display device side is changed. This makes it possible to prevent the viewer from getting VR sickness when watching VR.

[0128] <Embodiment 3> The electronic device according to the third embodiment operates on a personal computer (hereinafter, referred to as a "PC") that handles video files. The operation of the PC is realized by software.

[0129] In the third embodiment, the personal computer 1100 analyzes a phenomenon in a video that may be a cause of VR sickness. Then, the personal computer 1100 writes information on the phenomenon that may be a cause of VR sickness into a video file. Note that in the third embodiment, the same reference numerals as those in the first or second embodiment are the same operations or processes as those in the first or second embodiment, and therefore the description thereof will be omitted.

[0130] 11 is a block diagram showing the configuration of a personal computer 1100. The personal computer 1100 is an electronic device capable of performing various processes on captured images. The personal computer 1100 has a control unit 1101, a ROM 1102, a RAM 1103, an external storage device 1104, an operation unit 1105, a display unit 1106, a communication unit 1107, an external interface (I / F) 1108, and a system bus 1109.

[0131] The control unit 1101 controls the entire personal computer 1100. The control unit 1101 is, for example, a Central Processing Unit (CPU).

[0132] The ROM 1102 is a read only memory (ROM) that stores programs and parameters that do not require modification.

[0133] The RAM 1103 temporarily stores programs and data supplied from an external device. It is a Random Access Memory (RAM) that stores

[0134] The external storage device 1104 is an external storage device installed inside the personal computer 1100. Alternatively, the external storage device 1104 is an external storage device that is detachable from the personal computer 1100. The external storage device 1104 includes a hard disk, a flash memory, a floppy disk (FD), an optical disk (such as a Compact Disk (CD)), a magnetic disk, an optical card, an IC card, a memory card, or the like. Video files captured by a camera, etc. are written to the external storage device 1104.

[0135] The operation unit 1105 includes operation members such as buttons that accept user operations.

[0136] The display unit 1106 displays data held by the personal computer 1100 and data supplied to the personal computer 1100 .

[0137] A communication unit 1107 is a communication unit for communicating with other devices. An external I / F 1108 transmits and receives video signals and files to and from an external device such as the camera 100.

[0138] The system bus 1109 is a system bus that connects each component so that they can communicate with each other. The ROM 1102 stores a predetermined information processing program as a program code that can be read by the control unit 1101. The control unit 1101 executes the process indicated by the program code.

[0139] Next, the operation of the personal computer 1100 according to the third embodiment will be described with reference to the flowchart of FIG.

[0140] The process of the flowchart in Fig. 12 starts when the personal computer 1100 reads a moving image file (obtains a moving image file from the camera 100 or the like). The personal computer 1100 can use the read moving image file to play and display, start playback, and stop playback of a moving image using a specific application. The configuration of the moving image file is assumed to be the same as the configuration shown in Fig. 5 of the first embodiment.

[0141] In step S1201, the control unit 1101 acquires the shooting information 502, the metadata 503, and the metadata 504 from the loaded moving image file.

[0142] In step S1202, the control unit 1101 determines whether the loaded video file (captured image) was recorded by a camera equipped with a VR180 lens (twin lens). If it is determined that the video file was recorded by a camera equipped with a VR180 lens, the process proceeds to step S1203. If it is determined that the video file was not recorded by a camera equipped with a VR180 lens, the process of this flowchart ends.

[0143] In step S1203, the control unit 1101 determines whether or not phenomenon information is recorded in the loaded video file. Here, the control unit 1101 checks the contents of the metadata 503 and the metadata 504, and checks the metadata of the first frame of the data 505. If it is determined that the phenomenon information is not recorded in the video file, the process proceeds to step S1204. If it is determined that the phenomenon information is recorded in the video file, the process of this flowchart ends.

[0144] In step S1204, the control unit 1101 determines whether or not an analysis instruction for the loaded video file (an instruction to analyze a phenomenon that may be a cause of VR sickness) has been issued. If it is determined that an analysis instruction has been issued, the process proceeds to step S1205. If it is determined that the above has not been done, the process of this flowchart ends.

[0145] In step S1205, the control unit 1101 sets the current frame number N to 1 (the number indicating the first frame).

[0146] In step S1206, the control unit 1101 obtains the captured image of the Nth frame from the moving image file.

[0147] In step S1207, the control unit 1101 analyzes (detects) a phenomenon that may be a cause of VR sickness in the captured image acquired in step S1206. Specifically, the luminance difference, contrast difference, and color difference between the right eye image and the left eye image may be a cause of VR sickness. For this reason, the control unit 1101 detects the difference (luminance difference, contrast difference, or color difference) between the right eye image and the left eye image in the captured image of the current frame, and acquires the degree of the difference as a phenomenon that may be a cause of VR sickness.

[0148] Furthermore, when determining the degree of blur as a phenomenon that may cause VR sickness, the control unit 1101 acquires captured images of several frames before and after the current frame. Then, the control unit 1101 extracts feature points from each captured image using a known feature point extraction process, and then calculates the movement distance and movement direction of the feature points between these frames. The control unit 1101 acquires the movement distance (amount of change) of the feature points as the degree of blur, since a large change in the movement distance between frames or frequent changes in the movement direction can lead to VR sickness.

[0149] In step S1208, the control unit 1101 writes the information on the phenomenon that may cause VR sickness (phenomenon information) acquired in step S1207 into the metadata of the N-th frame in the video file. That is, the N-th frame and the phenomenon information are associated with each other.

[0150] In step S1209, the control unit 1101 determines whether the Nth frame is the final frame of the video file (video). If it is determined that the Nth frame is the final frame, the process of this flowchart ends. If it is determined that the Nth frame is not the final frame, the process proceeds to step S1210.

[0151] In step S1210, the control unit 1101 adds 1 to the current frame number N.

[0152] By performing the process of the flowchart shown in FIG. 12, even if the phenomenon information is not recorded by the camera, the personal computer can later save the phenomenon information in a video file.

[0153] Immediately after the analysis of all frames is completed, the phenomenon information of the metadata of each frame may be compiled, and summary information such as the maximum level of VR sickness (maximum level of phenomenon information) and sections where the level of VR sickness is high may be recorded in the metadata 503. In this way, the display device (viewing device) can quickly obtain information on the level of VR sickness and sections where VR sickness occurs from the video file.

[0154] Thus, in the third embodiment, the electronic device acquires phenomenon information (information such as image blur, brightness difference between the left eye image and the right eye image, contrast difference, and color difference) and writes the phenomenon information to a video file. This allows the display device (viewing device) to read the phenomenon information during VR viewing and perform display control based on the phenomenon information. This makes it possible to prevent the user from getting VR sickness while viewing VR.

[0155] In each embodiment, instead of "VR sickness," "an image pickup device equipped with a twin lens" is used. For example, "VR sickness" may be read as "fatigue," "discomfort," "headache," "stomach awareness," "nausea," "vomiting," "paleness," "sweating," "fatigue," "drowsiness," "disorientation," or "lethargy."

[0156] There may also be an electronic device (information processing system) having all or at least a part of the configuration of the camera 100 according to the embodiment 1 and the HMD 600 according to the embodiment 2. In this case, the electronic device records, for example, a captured image acquired by itself by imaging in association with phenomenon information in a file, and when playing back the captured image, controls the display of the captured image based on the phenomenon information recorded in the file.

[0157] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0158] Also, in the above, "If A is equal to or greater than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2." Conversely, "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2." Therefore, unless a contradiction occurs, "equal to or greater than A" may be read as "equal to or greater than A (high; long; many)," and "equal to or less than A" may be read as "equal to or less than A (low; short; few)." And, "equal to or greater than A" may be read as "equal to or greater than A," and "equal to or less than A" may be read as "equal to or less than A."

[0159] Each functional unit in each of the above embodiments may or may not be an individual piece of hardware. The functions of two or more functional units may be realized by common hardware. Each of a plurality of functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0160] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

[0161] The disclosure of the above embodiments includes the following configurations, methods, and programs. (Configuration 1) An image acquisition means for acquiring a captured image; an acquisition means for acquiring, when the captured image is an image acquired by imaging using an imaging device equipped with a twin lens, first information corresponding to the captured image and relating to a phenomenon that may be a cause of the occurrence of a specific symptom; a control means for recording the captured image and the first information in association with each other in a recording means; 1. An electronic device comprising: (Configuration 2) the control means, when the captured image is not an image acquired by imaging with an imaging device equipped with a twin lens, does not record the captured image in the recording means in association with the first information; 2. The electronic device according to configuration 1. (Configuration 3) the acquiring means acquires the first information by detecting a phenomenon that may be a cause of the specific symptom from the captured image. 3. The electronic device according to configuration 1 or 2. (Configuration 4) The captured image is an image currently being captured by the imaging device. 4. The electronic device according to any one of configurations 1 to 3. (Configuration 5) The imaging device further includes a notification unit that prompts a user to perform imaging to obtain a new captured image when a level of a phenomenon that may be a cause of the specific symptom is higher than a predetermined level. 5. The electronic device according to any one of configurations 1 to 4. (Configuration 6) an acquisition means for acquiring a captured image and information associated with the captured image, the first information being related to a phenomenon that may be a cause of a specific symptom; a control means for controlling display of the captured image on a display device based on the first information when the captured image is an image captured by an imaging device equipped with a twin lens; 1. An electronic device comprising: (Configuration 7) The acquisition means acquires, as the first information, a level of a phenomenon that may be a cause of the specific symptom; The electronic device further includes a warning unit that warns a user when a level of a phenomenon that may be a cause of the specific symptom is higher than a first level. 7. The electronic device according to configuration 6. (Configuration 8) the captured image is an image of one frame of a video, The control means 1) reproduces the video on the display device, and 2) stops reproduction of the video when a level of the phenomenon that may be a cause of the specific symptom is higher than the first level. 8. The electronic device according to configuration 7. (Configuration 9) the control means controls display of the captured image on the display device based on a type or function of the display device in addition to the first information. 9. The electronic device according to any one of configurations 6 to 8. (Configuration 10) a determination means for determining a level of the specific symptom that is expected to occur in the user when the user views the captured image based on the first information; The control means controls the display device to display the level of the specific symptom when the level of the specific symptom determined by the determination means is higher than a second level. 10. The electronic device according to any one of configurations 6 to 9. (Configuration 11) When the captured image is an image acquired by imaging using an imaging device equipped with the twin lenses, the captured image includes a right-eye image that is an image for the right eye and a left-eye image that is an image for the left eye, The first information includes at least one of information regarding a shake of the imaging device and information regarding a difference between the right eye image and the left eye image. 11. The electronic device according to any one of configurations 1 to 10. (Configuration 12) The difference between the right eye image and the left eye image is at least one of an image magnification difference, a vertical shift, a rotational shift, a luminance difference, a contrast difference, and a color difference. 12. The electronic device according to configuration 11. (Configuration 13) The specific symptom is VR sickness or fatigue. 13. The electronic device according to any one of configurations 1 to 12. (Method 1) an image acquisition step of acquiring a captured image; an acquisition step of acquiring, when the captured image is an image acquired by imaging using an imaging device equipped with a twin lens, first information corresponding to the captured image and relating to a phenomenon that may be a cause of the occurrence of a specific symptom; a control step of recording the captured image and the first information in association with each other in a recording means; 13. A method for controlling an electronic device comprising: (Method 2) an acquisition step of acquiring a captured image and first information associated with the captured image, the first information being related to a phenomenon that may be a cause of the occurrence of a specific symptom; a control step of controlling display of the captured image on a display device based on the first information when the captured image is an image captured by an imaging device equipped with a twin lens; 13. A method for controlling an electronic device comprising: (program) A program for causing a computer to function as each of the means of the electronic device described in any one of configurations 1 to 13. [Explanation of symbols]

[0162] 100: camera, 50: system control unit, 211: imaging unit 227: Recording media

Claims

1. image acquisition means for acquiring a captured image; an acquisition means for acquiring, when the captured image is an image acquired by imaging using an imaging device equipped with a twin lens, first information corresponding to the captured image, the first information relating to a phenomenon that may be a cause of the occurrence of a specific symptom; a control means for recording the captured image and the first information in association with each other in a recording means; and When the captured image is an image acquired by imaging using the imaging device equipped with the twin lenses, the captured image includes a right-eye image that is an image for the right eye and a left-eye image that is an image for the left eye, the first information includes information of a difference between the right-eye image and the left-eye image; An electronic device characterized by:

2. the control means does not record the captured image in the recording means in association with the first information when the captured image is not an image acquired by imaging with an imaging device equipped with a twin lens; 2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

3. the acquiring means acquires the first information by detecting a phenomenon that may be a cause of the specific symptom from the captured image.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

4. The captured image is an image currently being captured by the imaging device.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

5. The imaging device further includes a notification unit that prompts a user to take a new image when the level of a phenomenon that may be a cause of the specific symptom is higher than a predetermined level.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

6. The acquiring means acquires first information, which is information relating to the luminance difference between the right eye image and the left eye image, by comparing the average luminance of a predetermined area of ​​the left eye image with the average luminance of a predetermined area of ​​the right eye image.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

7. The image acquisition means acquires a video as the captured image, the control means, after recording the video, records information on a maximum value of a level of a phenomenon that may be a cause of the specific symptom in the video or information on a section in the video where a level of a phenomenon that may be a cause of the specific symptom in the video is greater than a threshold value; 2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

8. an acquisition means for acquiring a captured image and first information associated with the captured image, the first information being related to a phenomenon that may be a cause of the specific symptom; a control means for controlling display of the captured image on a display device based on the first information when the captured image is an image captured by an imaging device equipped with a twin lens; and When the captured image is an image acquired by imaging using the imaging device equipped with the twin lenses, the captured image includes a right-eye image that is an image for the right eye and a left-eye image that is an image for the left eye, the first information includes information of a difference between the right-eye image and the left-eye image; An electronic device characterized by:

9. the acquiring means acquires, as the first information, a level of a phenomenon that may be a cause of the specific symptom; the electronic device further comprises a warning unit that warns a user when a level of a phenomenon that may be a cause of the specific symptom is higher than a first level.

9. The electronic device according to claim 8.

10. The device further comprises a determination means for determining whether the level of a phenomenon that may be a cause of the specific symptom is greater than the first level; the first level is set lower when the captured image is displayed on a head-mounted display device than when the captured image is displayed on a non-head-mounted display device; 10. The electronic device according to claim 9.

11. The device further comprises a determination means for determining whether the level of a phenomenon that may be a cause of the specific symptom is greater than the first level; the first level is set lower when the captured image is displayed on a head-mounted display device having an image stabilization function than when the captured image is displayed on a head-mounted display device not having the image stabilization function; 10. The electronic device according to claim 9.

12. the captured image is a video, The control means 1) plays back the video on the display device, and 2) skips playback of frames in which a level of a phenomenon that may be a cause of the specific symptom is higher than a first level, and continues playback from other frames.

9. The electronic device according to claim 8.

13. the control means controls display of the captured image on the display device based on the type or function of the display device in addition to the first information.

9. The electronic device according to claim 8.

14. a determination means for determining a level of the specific symptom that is expected to occur in the user when the user views the captured image based on the first information; the control means controls the display device to display the level of the specific symptom when the level of the specific symptom determined by the determination means is higher than a second level.

9. The electronic device according to claim 8.

15. The captured image is a video, The control means 1) plays the video on the display device, 2) stops the video when a level of a phenomenon that may be a cause of the specific symptom in a frame to be played next is higher than a first level, and 3) continues playing the video while skipping the frame to be played next when a level of a phenomenon that may be a cause of the specific symptom in the frame to be played next is higher than a second level and lower than the first level.

9. The electronic device according to claim 8.

16. The difference between the right eye image and the left eye image is at least one of an image magnification difference, a vertical misalignment, a rotational misalignment, a luminance difference, a contrast difference, and a color difference.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

17. The specific symptom is VR sickness.

17. The electronic device according to claim 1.

18. an image acquisition step of acquiring a captured image; an acquisition step of acquiring, when the captured image is an image acquired by imaging using an imaging device equipped with a twin lens, first information corresponding to the captured image, the first information relating to a phenomenon that may be a cause of the occurrence of a specific symptom; a control step of recording the captured image and the first information in association with each other in a recording means; and When the captured image is an image acquired by imaging using the imaging device equipped with the twin lenses, the captured image includes a right-eye image that is an image for the right eye and a left-eye image that is an image for the left eye, the first information includes information of a difference between the right-eye image and the left-eye image; 10. A method for controlling an electronic device comprising:

19. an acquiring step of acquiring a captured image and first information associated with the captured image, the first information being related to a phenomenon that may be a cause of the specific symptom; a control step of controlling display of the captured image on a display device based on the first information when the captured image is an image captured by an imaging device equipped with a twin lens; and When the captured image is an image acquired by imaging using the imaging device equipped with the twin lenses, the captured image includes a right-eye image that is an image for the right eye and a left-eye image that is an image for the left eye, the first information includes information of a difference between the right-eye image and the left-eye image; 10. A method for controlling an electronic device comprising:

20. A program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 16.

Images