Image processing device, image processing method, storage medium and program

Abstract

To improve the convenience of a user when viewing a stereoscopic image.SOLUTION: An image processing device comprises: display control means which causes display means to display a stereoscopic image by using image data including a plurality of images with different view points; and image processing means which processes the image data on the basis of meta data annexed to the image data. The image processing device performs processing of converting the image data into a file type that can be displayed by the display means when the meta data includes information indicating that the image data corresponds to a file type that cannot be displayed by the display means.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, storage medium and regarding the program. [Background technology]

[0002] In recent years, head-mounted displays capable of viewing stereoscopic images have become widespread. Stereoscopic images can be captured using a camera equipped with a compound lens such as a twin-lens. As a related technology, the technology of Patent Document 1 has been proposed. The technology of Patent Document 1 relates to a twin-lens digital camera, in which the same subject is captured by two shooting optical systems, and images of the same subject are formed on each of two solid-state imaging elements, and image signals are generated by each of the two solid-state imaging elements. In addition, the technology of Patent Document 2 has been proposed. The technology of Patent Document 2 relates to a head-mounted display, and uses an image for the right eye and an image for the left eye to show a stereoscopic image to a user. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2004-120527 A [Patent Document 2] JP 2019-029721 A Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, when an imaging device equipped with a compound lens such as a twin lens forms light from different optical systems on one sensor, the subject image through the optical system located on the right side is formed on the right side of the sensor, and the subject image through the optical system located on the left side is formed on the left side of the sensor. At this time, since the subject image is formed in a state inverted in the vertical and horizontal directions, the orientation of the subject image can be corrected by reading out the image data obtained by this sensor so as to invert it in the vertical and horizontal directions. However, by reading out in this manner, the subject image generated using the optical system on the right side is located on the left side, and the subject image generated using the optical system on the left side is located on the right side. That is, unlike the case of shooting using two shooting optical systems and two imaging elements, the left and right positions of the subject images corresponding to each shooting optical system are inverted with respect to the left and right positions of the two shooting optical systems. If the image data in which the positions of the left and right subject images are swapped is displayed on a head-mounted display that displays a stereoscopic image using images shot using two shooting optical systems and two imaging elements, the stereoscopic image cannot be displayed normally. This is because the area of ​​image data captured by the left optical system is processed as image data for the right eye, and the area of ​​image data captured by the right optical system is processed as image data for the left eye.

[0005] Therefore, an object of the present invention is to improve the convenience for users when viewing stereoscopic images. [Means for solving the problem]

[0006] In order to achieve the above object, the image processing device of the present invention includes a receiving means for receiving, from an imaging device equipped with a twin lens, an image file including image data including a first image area and a second image area formed through a right-eye optical system and a left-eye optical system of the twin lens, respectively, and metadata related to the image data, a display control means for displaying a stereoscopic image on a display means using the image data, and a first determination means for determining whether an application needs to be upgraded based on the file format of the image file indicated by the metadata; outside and a version-upgrading unit that acquires predetermined data from an external device and executes a process for upgrading the application. After the application has been upgraded, the version-upgrading unit analyzes the image data and compares the first image area with the second image area to thereby determine whether the first image area and the second image area are identical in the image data. of The present invention is characterized in that it has a second judgment means for judging whether the first image area and the second image area are in a left-right reversed state in the image data, and an image processing means for executing a process to swap the first image area and the second image area in the image data if the judgment result of the second judgment means indicates that the first image area and the second image area are in a left-right reversed state in the image data, and the display control means causes the display means to display the stereoscopic image using image data in which the process to swap the first image area and the second image area in the image data has been executed. Effect of the Invention

[0007] According to the present invention, it is possible to improve the convenience for the user when viewing a stereoscopic image. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 illustrates an example of a system. [Diagram 2] FIG. 1 illustrates an example of an imaging device. [Diagram 3] FIG. 1 is a cross-sectional view showing an example of a twin lens. [Figure 4] FIG. 1 is a diagram illustrating an example of a head mounted display. [Diagram 5] FIG. 1 illustrates an example of a personal computer. [Figure 6] FIG. 1 is a diagram showing a first example of the principle of photography. [Figure 7] FIG. 13 is a diagram showing a second example of the photographing principle. [Figure 8] 5 is a flowchart showing an example of a processing flow of the head mounted display in the first embodiment. [Figure 9] 13 is a flowchart showing an example of a processing flow of a head mounted display not equipped with an image replacement function. [Figure 10] 13 is a flowchart showing an example of a processing flow of a head mounted display in a third embodiment. [Figure 11] 13 is a flowchart showing an example of a processing flow of the camera body in the fourth embodiment. [Figure 12] 13 is a flowchart showing an example of a processing flow of a personal computer in the fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, each embodiment of the present invention will be described in detail with reference to the drawings. However, the configurations described in the following embodiments are merely examples, and the scope of the present invention is not limited to the configurations described in the embodiments.

[0010] First Embodiment FIG. 1 is a diagram showing an example of a system 1. The system 1 includes a twin lens 2, a camera body 3, and a head-mounted display 4. If necessary, a personal computer 5 is also added to the system 1. The twin lens 2 is an interchangeable lens that can be attached to and detached from the camera body 3. The camera body 3 is a camera body to which a general monocular interchangeable lens such as a zoom lens or a prime lens can be attached and detached. The interchangeable lens is attached to the camera body 3 to configure an imaging device. When a compound lens is attached to the camera body 3, light from multiple optical systems is guided to one sensor, as described later. The interchangeable lens attached to the camera body 3 is not limited to a twin lens, and may be a lens with three or more eyes. In the system 1, image data generated by photographing a subject with the twin lens 2 attached to the camera body 3 is transmitted to the head-mounted display 4. The head-mounted display 4 displays a stereoscopic image. In each figure, the head-mounted display is written as "HMD" and the personal computer is written as "PC".

[0011] In the example of FIG. 1, the camera body 3, the head mounted display 4, and the personal computer 5 are capable of communicating with each other. Image data generated by the camera body 3 is transmitted to the head mounted display 4. The image data may be processed in the personal computer 5 before being transmitted to the head mounted display 4. The head mounted display 4 or the personal computer 5 may acquire the image data via a portable recording medium such as a memory card connected to the camera body 3. The head mounted display 4 displays a stereoscopic image when the image data acquired from the camera body 3 can be displayed as a stereoscopic image. The personal computer 5 can perform a predetermined process on the image data for displaying a stereoscopic image.

[0012] The imaging device with the twin lens 2 attached to the camera body 3 can capture a plurality of images with different viewpoints for use in displaying a stereoscopic image. Since the camera body 3 has one sensor, the image obtained through the optical system for the left eye of the twin lens 2 is formed in an area on the left side of the sensor with the vertical (up and down) and horizontal (left and right) directions reversed. This is called the image for the left eye. The image obtained through the optical system for the right eye of the twin lens 2 is formed in an area on the right side of the sensor with the vertical and horizontal directions reversed. This is called the image for the right eye. If the image data is read out so that the vertical and horizontal directions are reversed in order to correct the vertical and horizontal directions, the read image data will have the left and right positions of the image for the left eye and the image for the right eye swapped, that is, the relationship between the left and right positions will be reversed. Therefore, the camera body 3 of this embodiment adds metadata to the image data indicating that the left and right positions of the image for the left eye and the image for the right eye are reversed.

[0013] In contrast, if an image is taken with a twin-lens camera that has two sets of one imaging optical system and one image sensor, the left and right positions of the image for the left eye and the image for the right eye are not reversed, so metadata indicating that the images are reversed is not added to the image data. Also, if an image is taken with a monocular lens instead of twin lenses, metadata indicating that the images are reversed is not added to the image data.

[0014] FIG. 2 is a diagram showing an example of an imaging device 200. In the imaging device 200 of the example of FIG. 2, a twin lens 2 as an interchangeable lens is attached to a camera body 3. The imaging device 200 or the camera body 3 can constitute an image processing device. In the imaging device 200 of FIG. 2, each unit other than the twin lens 2 constitutes the camera body 3. The details of the twin lens 2 will be described later. The camera body 3 has a camera control unit 201. The camera control unit 201 is connected to a sensor 202, an image processing unit 203, a display unit 204, a wired communication unit 205, a wireless communication unit 206, an operation unit 207, a power supply unit 208, a memory unit 209, and a card connection unit 210. The sensor 202 is an image sensor composed of a CCD or CMOS sensor. When the twin lens 2 is attached to the camera body 3, the camera control unit 201 and the twin lens 2 are electrically connected. For example, the twin lens 2 communicates various control signals with the camera control unit 201 via communication of electrical contacts provided on the lens mount of the camera body 3.

[0015] The camera control unit 201 has a CPU and a memory. A control program is stored in the memory, and the CPU executes the control program to realize various controls of the camera control unit 201. The sensor 202, which is an image pickup element, photoelectrically converts a subject image formed by two imaging optical systems of the twin lens 2 to generate an imaging signal. As shown in FIG. 2, light guided from the two imaging optical systems of the twin lens 2 forms an image on one sensor 202. The image processing unit 203 performs various image processing on the generated imaging signal to generate image data. The display unit 204 can display a through image by displaying the image data output from the image processing unit 203. The display unit 204 can also play back and display imaging parameters and captured images stored in the storage unit 209.

[0016] The wired communication unit 205 performs wired communication with the head mounted display 4, the personal computer 5, etc. The wireless communication unit 206 performs wireless communication with the head mounted display 4, the personal computer 5, etc. Any method can be applied for the wired communication or wireless communication. The operation unit 207 accepts user operations on various operation members of the camera body 3. Examples of the operation members include a power operation unit, a mode dial, a release button, a rear operation unit, and a touch panel display of the display unit 204. The power supply unit 208 supplies power to the twin lens 2 and the camera body 3. The memory unit 209 stores various information. For example, the memory unit 209 stores a control program executed by the CPU of the camera control unit 201, captured images, etc. The card connection unit 210 is a connection unit for connecting to a portable recording medium such as a memory card. The camera body 3 is not limited to the example of FIG. 2.

[0017] Next, the interchangeable twin lens 2 will be described. FIG. 3 is a cross-sectional view showing an example of the interchangeable twin lens 2. The twin lens 2 has a right optical system 301R and a left optical system 301L. The right optical system 301R and the left optical system 301L are two optical systems arranged parallel and symmetrically to each other. The right optical system 301R and the left optical system 301L each guide light along the optical axis from the subject side to the imaging side in that order. The right optical system 301R has a first lens 310R, a second lens 320R, and a third lens 330R arranged therein. The right optical system 301R also has a first prism 311R and a second prism 321R arranged therein. The light incident from the first lens 310R of the right optical system 301R has its optical path bent by the first prism 311R and the second prism 321R, and is guided to the third lens 330R.

[0018] The left optical system 301L includes a first lens 310L, a second lens 320L, and a third lens 330L. The left optical system 301L includes a first prism 311L and a second prism 321L. The light incident from the first lens 310L of the left optical system 301L is guided to the third lens 330L by bending the optical path at the first prism 311L and the second prism 321L. In this manner, the light incident from each of the right optical system 301R and the left optical system 301L is guided to the sensor 202. The configuration of the interchangeable twin lens 2 is not limited to the example shown in FIG. 3.

[0019] Next, the head mounted display 4 will be described. FIG. 4 is a diagram showing an example of the head mounted display 4. The head mounted display 4 is a display device that displays a stereoscopic image. A user can view a stereoscopic image by wearing the head mounted display 4. The device for viewing a stereoscopic image is not limited to a head mounted display, and may be a stationary display.

[0020] As shown in FIG. 4, the head mounted display 4 has each unit connected to a bus 400. The CPU 401 is a central processing unit that executes a control program stored in a program memory 402. The CPU 401 performs a process of switching between a left eye image and a right eye image included in image data, as described later. The CPU 401 also performs various other controls. In FIG. 4, each unit other than the left eye display unit 410 and the right eye display unit 411 may constitute a controller (image processing device) of the head mounted display 4. The program memory 402 stores a control program. The data memory 403 stores setting conditions of the head mounted display 4. The data memory 403 also stores still images and videos captured by the imaging device 200, and information related to these.

[0021] The power management unit 404 manages the power state of the head mounted display 4. The image processing unit 405 reads still images and videos, and performs predetermined image processing on the read still images and videos. The still images and videos processed by the image processing unit 405 are output to each unit via the bus 400. The storage unit 406 stores the videos and still images captured and formatted by the imaging device 200 with the twin lenses 2 attached to the camera body 3. The communication controller 407 is a controller that performs wired communication with the camera body 3 and the personal computer 5.

[0022] The operation unit 408 accepts various instructions from the user to the head mounted display 4. For example, the operation unit 408 accepts conditions for viewing a stereoscopic image on the head mounted display 4, an operation for replacing a battery, and the like. The wireless communication unit 409 performs wireless communication with the camera body 3 and the personal computer 5. The connection unit 412 connects a portable recording medium such as a memory card that can be connected to the camera body 3 and the personal computer 5.

[0023] As described above, the twin lens 2 as an interchangeable lens has the right optical system 301R and the left optical system 301L. The sensor 202 generates image data including an image for the left eye and an image for the right eye. The left eye display unit 410 is placed in front of the left eye of the user when the user wears the head mounted display 4, and displays the image for the left eye of the image data. The right eye display unit 411 is placed in front of the right eye of the user when the user wears the head mounted display 4, and displays the image for the right eye of the image data.

[0024] Next, the personal computer 5 will be described. The personal computer 5 can constitute an image processing device (or an information processing device) that performs predetermined processing on still images and videos captured by the imaging device 200. As the image processing device, instead of the personal computer 5, a predetermined server (such as a cloud server or an edge computer) or a smart device such as a smartphone may be applied.

[0025] FIG. 5 is a diagram showing an example of a personal computer 5. In the personal computer 5, each unit is connected to a bus 500. A CPU 501 is a central processing unit that executes a control program stored in a program memory 502. The CPU 501 performs various controls including control of a display controller 512. The CPU 501 also performs a process of switching between an image for the left eye and an image for the right eye. The program memory 502 stores the above-mentioned control program. A data memory 503 stores setting conditions of the personal computer 5. The data memory 503 also stores still images and videos acquired from the camera body 3, and information related thereto.

[0026] The power management unit 504 manages the power state of the personal computer 5. The image processing unit 505 reads still images and videos, and performs predetermined image processing on the read still images and videos. Instead of the CPU 501, the image processing unit 505 may perform processing for switching between images for the left eye and images for the right eye. The still images and videos processed by the image processing unit 505 are output to each unit via the bus 500. The storage unit 506 stores the videos and still images captured and formatted by the imaging device 200 with the twin lenses 2 attached to the camera body 3. The communication controller 507 is a controller that performs wired communication with the camera body 3 and the head mounted display 4.

[0027] The operation unit 508 accepts various instructions from the user to the personal computer 5. For example, a keyboard, a mouse, or the like is applied as the operation unit 508. The wireless communication unit 509 performs wireless communication with the camera body 3 and the head mounted display 4. The connection unit 510 connects a portable recording medium such as a memory card that can be connected to the camera body 3 and the head mounted display 4.

[0028] The display unit 511 displays various types of information of the personal computer 5. The display unit 511 can also display still images and moving images captured by the imaging device 200. The display controller 512 controls the display of the display unit 511.

[0029] Next, a first example of the principle of photography will be described. FIG. 6 is a diagram showing a first example of the principle of photography. In FIG. 6, the monocular is an example of photography performed with a monocular lens attached to the camera body 3. In addition, in FIG. 6, the twin lens is a diagram showing an example of this embodiment in which photography is performed with the twin lens 2 attached to the camera body 3. First, the monocular will be described. When a subject 6 is photographed with a monocular lens attached to the camera body 3, an image of the subject 6 is formed on the sensor 12 through the lens 11. At this time, the sensor 12 generates an image 13 that is inverted in the horizontal and vertical directions. At this time, when image data is read from the sensor 12, a process of swapping the horizontal and vertical directions is performed, so that an image 14 that is in the correct orientation in the horizontal and vertical directions is obtained. Note that the image data may be read from the sensor 12 and stored in a memory, and then the horizontal and vertical directions of the image data may be swapped.

[0030] Next, an example of a case where an image is captured with the twin lens 2 attached to the camera body 3, and then the "additional process" of this embodiment is performed will be described. When an image of the subject 6 is captured with the twin lens 2 attached to the camera body 3, a subject image 6A that is inverted in the vertical and horizontal directions is formed in the left area of ​​the sensor 17 through the left lens 15 (the lens of the above-mentioned left optical system 301L). At the same time, a subject image 6B that is inverted in the vertical and horizontal directions is formed in the right area of ​​the sensor 17 through the right lens 16 (the lens of the above-mentioned right optical system 301R). As a result, image data 20 is generated that includes the subject image 6A corresponding to the left lens 15 and the subject image 6B formed through the right lens 16 in a state in which they are inverted in the horizontal and vertical directions. By reading out this image data 20 so that the horizontal and vertical directions are interchanged, image data 21 in which the orientations of the subject images 6A and 6B in the vertical and horizontal directions are corrected is generated.

[0031] Here, the subject images 6A and 6B included in the image data 21 are reversed (switched) in position in the horizontal direction. That is, the subject image 6B corresponding to the right lens 16 is present on the left side of the image data 21, and the subject image 6A corresponding to the left lens 15 is present on the right side of the image data 21.

[0032] Therefore, if the head mounted display 4 performs stereoscopic display using this image data 21 as is, image data including subject image 6B corresponding to right lens 16 will be input to the left eye display unit 410. Similarly, image data including subject image 6A corresponding to left lens 15 will be input to the right eye display unit 411. As a result, the user will not be able to view a stereoscopic image properly.

[0033] Therefore, the CPU 401 of the head mounted display 4 divides the image data 21 into two in the horizontal direction to generate image data 22, and generates image data 23 by swapping the positions of two images included in the image data 22 in the horizontal direction. As a result, image data on the left side of the image data 23, including the subject image 6A corresponding to the left lens 15, is input to the left eye display unit 410 of the head mounted display 4. Also, image data on the right side of the image data 23, including the subject image 6B corresponding to the right lens 16, is displayed on the right eye display unit 411. This allows the user to properly view a stereoscopic image.

[0034] FIG. 7 is a diagram showing a second example of the principle of photography. FIG. 7 is a diagram showing an example of this embodiment in the case where photography is performed with a triple lens attached to the camera body 3. In the case of a triple lens, there are three lenses: a left lens 25, a center lens 26, and a right lens 27. When photography of a subject 6 is performed with a triple lens attached to the camera body 3, a subject image 6C that is inverted in the vertical and horizontal directions is formed in the left area of ​​the sensor 30 through the left lens 25. A subject image 6D that is inverted in the vertical and horizontal directions is formed in the center area of ​​the sensor 30 through the center lens 26. A subject image 6E that is inverted in the vertical and horizontal directions is formed in the right area of ​​the sensor 30 through the right lens 27. As a result, image data 31 is generated that includes the subject image 6C corresponding to the left lens 25, the subject image 6D corresponding to the center lens 26, and the subject image 6E corresponding to the right lens 27 in a state in which they are inverted in the horizontal and vertical directions. By reading out this image data 31 so as to switch the horizontal and vertical directions, image data 32 is generated in which the up-down and left-right directions of subject images 6C, 6D, and 6E are corrected.

[0035] Then, the CPU 401 of the head mounted display 4 of this embodiment performs the following process as an additional process. The CPU 401 divides the image data 32 into three in the horizontal direction to generate image data 33, and generates image data 34 by swapping the positions of two images, the subject image 6C located on the right side and the subject image 6E located on the left side, in the image data 33 in the horizontal direction. It is possible to display a stereoscopic image from a different viewpoint by switching which two subject images are used for a stereoscopic image among the subject image obtained by the left lens 25, the subject image obtained by the center lens 26, and the subject image obtained by the right lens 27. In addition, it is possible to use the subject image obtained by the left lens 25 and the subject image obtained by the right lens 27 for a stereoscopic image, and to use the subject image obtained by the center lens 26 to improve the image quality of the stereoscopic image. For example, by changing the exposure setting for the subject image obtained by the center lens 26, it is possible to use it for processing to pseudo-expand the dynamic range for a saturated portion in the stereoscopic image or a portion where the brightness level is low and the subject image cannot be recognized. This allows a user wearing the head mounted display 4 to properly view the stereoscopic image captured by the three-group lens.

[0036] Although the two-lens and three-lens have been described as examples here, the present invention can also be applied to cases where four or more lenses are used.

[0037] Here, the camera control unit 201 of the camera body 3 of this embodiment can determine whether to turn on or off information requiring left and right image swapping, depending on the interchangeable lens attached to the camera body 3. Meanwhile, there are head mounted displays that can swap the left and right images of image data, and those that cannot.

[0038] The head mounted display 4, which has a function of swapping the left and right images of the image data, acquires image data including an image for the right eye and an image for the left eye, as well as metadata associated with the image data. The image for the right eye is an image formed from the right optical system 301R, and the image for the left eye is an image formed from the left optical system 301L. The metadata (accompanying information) indicates whether the image for the right eye and the image for the left eye included in the image data are in an inverted state. In the case of this embodiment, if the image for the right eye and the image for the left eye are in an inverted state, this is represented by "1", and if the image for the right eye and the image for the left eye are not in an inverted state, this is represented by "0".

[0039] If the metadata indicates that the right eye image and the left eye image are not reversed, the head mounted display 4 does not swap the right eye image and the left eye image included in the image data. Therefore, the right eye image and the left eye image are used for display on the left eye display unit 410 and the right eye display unit 411 without being swapped. On the other hand, if the metadata indicates that the right eye image and the left eye image included in the image data are reversed, the head mounted display 4 generates image data in which the right eye image and the left eye image are swapped. Then, the generated image data is stored in the storage unit 406 and is used for display on the left eye display unit 410 and the right eye display unit 411.

[0040] Here, an example of a head mounted display that does not have a function of switching the right eye image and the left eye image of the image data will be described. When the metadata indicates that the right eye image and the left eye image are not reversed, the head mounted display uses the image data as is for display on the left eye display unit 410 and the right eye display unit 411. On the other hand, when the metadata indicates that the right eye image and the left eye image are reversed, it is determined whether only one of the right eye image and the left eye image of the image data can be used for display on both the right eye display unit 411 and the left eye display unit 410.

[0041] For example, when it is possible to use only the image for the left eye for display on both the right eye display unit 411 and the left eye display unit 410, the head mounted display performs display using only the image for the left eye on both the right eye display unit 411 and the left eye display unit 410. As a result, a user wearing the head mounted display can display an image that does not feel strange, even though the image for the right eye is displayed on the left eye display unit 410 and the image for the left eye is not displayed on the right eye display unit 411, and the image is not displayed in a stereoscopic manner. When the head mounted display 4 does not have a function to display using only either the image for the right eye or the image for the left eye on both the right eye display unit 411 and the left eye display unit 410, an error display may be displayed. The contents notified by the error display include a notification that a stereoscopic image cannot be displayed, a notification that the image data is not in a format suitable for the head mounted display, or a notification encouraging the use of a head mounted display capable of switching between left and right images.

[0042] Next, a process flow of the head mounted display 4 of the first embodiment will be described. FIG. 8 is a flowchart showing an example of a process flow of the head mounted display 4 of the first embodiment. In the flowcharts from FIG. 8 onwards, "step" is written as "S". Each process of the flowchart in FIG. 8 ends when a predetermined end condition is satisfied. For example, each process of the flowchart in FIG. 8 ends when the power supply of the head mounted display 4 is cut off or when an instruction to end the operation is given. This also applies to each flowchart described later.

[0043] In step 801, the CPU 401 of the head mounted display 4 performs a predetermined initialization process. In step 802, the CPU 401 determines whether an operation to select a viewing mode has been performed. Here, it is assumed that the head mounted display 4 allows selection of either the viewing mode or the image swapping mode. For example, the user can use the operation unit 408 to select the viewing mode for viewing still images or videos captured by the imaging device 200. If the CPU 401 determines YES in step 802, it advances the flow to step 803. In step 803, the CPU 401 acquires image data including an image for the left eye and an image for the right eye, and a file including metadata related to the image data, from the camera body 3 to which the twin lens 2 is attached.

[0044] In step 804, the CPU 401 determines whether the image data has a left eye image and a right eye image in a reversed state based on the metadata, and whether the image data needs to be swapped between the left eye image and the right eye image. If the CPU 401 determines NO in step 804, the flow proceeds to step 806. If the CPU 401 determines YES in step 804, the flow proceeds to step 805. Here, the CPU 401 may notify the determination result of step 804. For example, the CPU 401 may display a notification indicating that the left eye image and the right eye image are in a reversed state on either or both of the left eye display unit 410 and the right eye display unit 422. The CPU 401 may also notify the user of the notification by voice. The same notification manner may also be applied to various notifications described below.

[0045] In step 805, CPU 401 performs image processing on the image data to swap the image for the left eye with the image for the right eye. At this time, CPU 401 may perform image processing on the image data using image processing unit 405 other than the image processing to swap the image for the left eye with the image for the right eye. Then, in step 806, left eye display unit 410 performs display using the image included in the left area of ​​the image data, and right eye display unit 411 performs display using the image included in the right area of ​​the image data. Then, the flow returns to step 803.

[0046] In step 802, if the CPU 401 judges NO in step 802, that is, the image replacement mode, the flow proceeds to step 807. In step 807, the CPU 401 selects one of the image data that has been received and has not yet been selected as a processing target. This image data selection may be performed by having the user select any image data, or may be performed by having the user select a condition for selecting an image. In step 808, the CPU 401 judges whether the image data selected as a processing target is image data that requires swapping between a left-eye image and a right-eye image. The CPU 401 can make this judgment based on metadata attached to the image data. If the CPU 401 judges YES in step 808, the flow proceeds to step 809, and if the CPU 401 judges NO, the flow proceeds to step 811.

[0047] In step 809, the CPU 401 switches the left eye image and the right eye image included in the image data. In step 810, the image data in which the left eye image and the right eye image have been switched is stored in a memory card connected to the connection unit 412. The CPU 401 records information indicating that the left eye image and the right eye image have been switched, that is, information indicating that the right eye image and the left eye image are not reversed, in the metadata stored in the memory card together with the image data. This allows, for example, the image data in which the left eye image and the right eye image have been switched and the metadata in which the information indicating this has been recorded to be used in another device (such as the head mounted display 4). In addition, the image data and metadata stored in the memory card may be transmitted to the other device by wired communication via the communication controller 407 or wireless communication via the wireless communication unit 409.

[0048] In step 810, the CPU 401 determines whether all image data has been selected as the processing target. If YES, the flow returns to step 802; if NO, the flow returns to step 807.

[0049] When image processing is performed in step S805 to swap the left-eye image and the right-eye image, the image data after the swap and metadata indicating this may be stored on the memory card, as in step 810.

[0050] In contrast, Fig. 9 is a flowchart showing an example of the process flow of a head mounted display that does not have a function for switching between a left eye image and a right eye image. The flowchart of Fig. 9 will also be briefly explained. This head mounted display can select whether or not to enter a mode in which only one of the images is used for display when the left eye image and the right eye image are in a reversed state.

[0051] In step 901, the CPU of the head mounted display performs initialization, and in step 902, image data including an image for the left eye and an image for the right eye, and metadata related to the image data are acquired. In step 903, if the CPU determines based on the metadata that the image data is not in a reversed state for the image for the left eye and the image for the right eye, the flow proceeds to step 904. On the other hand, if the CPU determines that the image data is in a reversed state, the flow proceeds to step 905.

[0052] In step 904, the left eye display unit displays an image using the left eye image included in the image data, and the right eye display unit displays an image using the right eye image included in the image data. Then, the flow returns to step 902.

[0053] In step 905, the CPU determines whether a mode is selected in which an image is displayed using only one of the left-eye image and the right-eye image included in the image data, and if YES, the flow proceeds to step 906. In step 906, both the left-eye display unit and the right-eye display unit display an image using only one of the left-eye image and the right-eye image. The user may be able to select which image to use. If NO in step 905, the flow proceeds to step 907. In step 907, a notification is displayed that a stereoscopic image cannot be displayed, that the image data is not in a format suitable for a head-mounted display, or a notification is displayed encouraging the use of a head-mounted display capable of swapping left and right images.

[0054] As described above, in this embodiment, even if the camera body 3 does not have a function of switching the left eye image and the right eye image, the head mounted display 4 can switch the left eye image and the right eye image to display a stereoscopic image. If the head mounted display 4 has a function of switching the left eye image and the right eye image, it determines whether it is necessary to switch the left eye image and the right eye image of the image data based on the metadata. Then, the head mounted display 4 switches the left eye image and the right eye image of the image data if necessary. This allows the stereoscopic image to be viewed normally even if the image data is taken with a camera in which the positions of the left eye image and the right eye image are switched, or even if the image data is taken with a camera in which the positions of the left eye image and the right eye image are not switched.

[0055] <Second embodiment> Next, a second embodiment will be described. In the first embodiment, whether the image for the left eye and the image for the right eye are in an inverted state is determined based on metadata, but in the second embodiment, whether the image for the left eye and the image for the right eye are in an inverted state is determined by analyzing image data instead of the acquired metadata.

[0056] In step 804 or 808 in FIG. 8, the CPU 401 compares the image included in the left region of the image data with the image included in the right region to determine whether the positions of the image for the left eye and the image for the right eye are reversed. The position of the subject image 6A in the image for the left eye formed using the left optical system 301L is shifted to the right with respect to the background compared to the position of the subject image 6B in the image for the left eye formed using the right optical system 301R. This is more noticeable the shorter the distance from the camera to the subject 6. For example, focus on the subject that is the target of AF. If the position of this subject in the image area on the left side of the image data is shifted to the left with respect to the background more than the position of this subject in the image area on the right side of the image data, it is understood that the image data is in a reversed state for the image for the right eye and the image for the left eye. Therefore, the CPU 401 can make the determination in step 804 or step 808 based on the result of analyzing the positions of the subject in the image for the left eye and the image for the right eye.

[0057] In the second embodiment, as described above, even if the metadata does not include information on whether the left-eye image and the right-eye image have been swapped, the same effect as in the first embodiment can be obtained. Since the processes other than step 804 or step 808 described above are the same as in the first embodiment, the description thereof will be omitted.

[0058] <Third embodiment> Next, a third embodiment will be described. In this embodiment, there are a plurality of file formats of image data including right-eye and left-eye images for stereoscopic display, and metadata of the image data indicates which file format the image data corresponds to. For example, there are differences between the plurality of file formats, such as whether the positions of the left-eye and right-eye images are reversed, whether the left-eye and right-eye images are included in one image data or are separated into separate image data, and so on. Alternatively, there may be differences in how many images are included, or which compression method is used. In this embodiment, the CPU 401 executes an application to realize processing related to processing and editing of image data of the head mounted display 4.

[0059] If the file format compatible with stereoscopic display differs depending on the manufacturer or model, the CPU 401 of the head mounted display 4 may or may not be able to interpret the metadata acquired from the camera body 3. Also, the CPU 401 can interpret the metadata acquired from the camera body 3, but may or may not be able to perform stereoscopic display of image data in a file format indicated by the metadata. In this embodiment, if the CPU 401 cannot interpret the metadata associated with the image data or cannot perform stereoscopic display of image data in the corresponding file format, the CPU 401 transmits a request to upgrade the application to a specified application server.

[0060] The configuration of the head mounted display 4 in the third embodiment is the same as that in the first embodiment, and therefore a description thereof will be omitted. FIG. 10 is a flowchart showing an example of a processing flow of the head mounted display 4 in the third embodiment. Steps that perform the same processing as in FIG. 8 are given the same step numbers as in FIG. 8, and descriptions of these steps will be omitted. In step 1001 in FIG. 10, the CPU 401 determines whether it is possible to interpret the metadata acquired in step 803, and whether the file format is compatible with the application of the CPU 401. If the CPU 401 determines YES in step 1001, the flow proceeds to step 804.

[0061] If the CPU 401 determines NO in step 1001, the flow proceeds to step 1002. In step 1002, the CPU 401 displays a notification indicating that the metadata of the acquired image data cannot be interpreted or that the image data is in an incompatible file format. Furthermore, the CPU 401 controls the display of a notification urging the user to upgrade the application. For example, the CPU 401 displays a message such as "Please upgrade your application. Do you want to upgrade?"

[0062] In S1003, if the user does not select to perform the version upgrade, the CPU 401 returns the flow to S802. On the other hand, if the user selects to perform the version upgrade, the CPU 401 advances the flow to step 1004. In step 1004, the CPU 401 performs the version upgrade process. Specifically, the CPU 401 controls the communication controller 407 and the wireless communication unit 409 to transmit a version upgrade request to an external application server present on the network. In response to the version upgrade request, the external application server transmits data for upgrading the application to a file format indicated by the above-mentioned metadata to the head mounted display 4. The CPU 401 executes the version upgrade of the application using the received data. This version upgrade of the application enables the right eye display unit 411 and the left eye display unit 410 to display a stereoscopic image using image data in a new file format. Alternatively, the version upgrade may be such that the file format of the image data generated by the camera can be converted into a file format that can be displayed by the right eye display unit 411 and the left eye display unit 410.

[0063] As a result, even if the conditions that the metadata acquired in step 803 can be interpreted and the file format is compatible with the application of CPU 401 are not satisfied, it becomes possible to swap the left and right images. Note that, although an example has been described here in which CPU 401 upgrades the application when the file format is not compatible with stereoscopic display, the present invention is not limited to this. CPU 401 may upgrade the application when it is unable to convert the acquired image data into a file format compatible with stereoscopic display. In this case, CPU 401 upgrades the application so that the acquired image data can be converted into a file format compatible with stereoscopic display.

[0064] <Fourth embodiment> Next, a fourth embodiment will be described. In the fourth embodiment, the camera body 3 performs processing such as swapping between left-eye images and right-eye images and converting the file format of image data according to the processing capacity of the head-mounted display 4. In addition, the head-mounted display 4 notifies the camera body 3 of characteristic information indicating the processing capacity of the head-mounted display 4. If the camera body 3 is provided with data in advance that associates the model number of the head-mounted display 4 with its characteristic information, the head-mounted display 4 may notify the camera body 3 of the model number information instead of the characteristic information. As described above, in the fourth embodiment, the camera body 3 performs processing of file format conversion including swapping between left-eye images and right-eye images as necessary. However, in order to enable continuous shooting by the camera body 3, it is preferable that the processing of file format conversion including swapping between left-eye images and right-eye images is performed by the head-mounted display 4. This is also effective from the viewpoint of suppressing power consumption of the camera body 3.

[0065] FIG. 11 is a flowchart showing an example of the flow of processing of the camera body 3 in the fourth embodiment. In step 1101, the camera control unit 201 performs initialization processing of the camera body 3. In step 1102, the camera control unit 201 determines whether the shooting mode has been selected. The camera body 3 has a plurality of modes, and the user can select a mode using the operation unit 207. Here, the plurality of modes include a shooting mode for performing shooting and a transmission mode for transmitting image data to the head mounted display 4. If the camera control unit 201 determines YES in step 1102, the flow proceeds to step 1103. In step 1103, the camera control unit 201 communicates with the lens attached to the camera body 3 to acquire lens information. The lens information includes information on the configuration of the lens, such as whether it is a monocular lens, a twin lens, or a trio lens. The lens information may also include information on the model number, manufacturing date, focal length, aberration, and the like of the lens. In the example of FIG. 2, lens information indicating that a twin lens is attached is acquired.

[0066] In step 1104, the camera control unit 201 waits until an instruction to capture an image is received. When an instruction to capture an image is received, the camera control unit 201 performs image capture in the next step 1105. In step 1106, the camera control unit 201 displays the captured image on the display unit 204.

[0067] In step 1107, the camera control unit 201 records metadata according to the type of lens attached to the camera body 3 together with the image data. If the lens attached to the camera body 3 is a monocular lens, information indicating that the image was taken with the monocular lens is recorded in the metadata. If the twin lens 2 is attached to the camera body 3, the camera control unit 201 records information indicating that the image data includes two images, an image for the left eye and an image for the right eye, and that the positions of the image for the left eye and the image for the right eye are reversed, in the metadata. Note that if a triple lens is attached to the camera body 3, the camera control unit 201 records information indicating that the image data includes three images, an image for the left eye, a center image, and an image for the right eye, and that the positions of the image for the left eye and the image for the right eye are reversed, in the metadata. At this time, the camera control unit 201 may display a message such as "The image for the left eye and the image for the right eye have been swapped" together with the image. This allows the user to recognize that the image for the left eye and the image for the right eye are reversed. The camera control unit 201 stores file information, in which metadata is added to the captured image (image data), in the memory card connected to the card connection unit 210. Then, the camera control unit 201 returns the flow to step 1102.

[0068] If the result of S1102 is NO, that is, if the camera control unit 201 determines that the mode is a transmission mode for transmitting image data to the head mounted display 4, the camera control unit 201 advances the flow to step 1108. In step 1108, the camera control unit 201 requests characteristic information indicating the processing capacity of the head mounted display 4 from the head mounted display 4, and receives this characteristic information from the head mounted display 4.

[0069] In step 1109, the camera control unit 201 determines, based on the capability information of the head mounted display 4, whether the head mounted display 4 has a function for performing stereoscopic display based on image data generated by this camera body 3. This also includes the case where the head mounted display 4 has a function for converting the file format of the image data generated by the camera body 3 into a file format that the head mounted display 4 can display.

[0070] If the camera control unit 201 determines YES in step S1109, it advances the flow to step 1112. In step 1112, the camera control unit 201 transmits file information including the image data and metadata stored in step 1107 to the head mounted display 4. On the other hand, if the camera control unit 201 determines NO in step S1109, it advances the flow to step 1110.

[0071] In step 1110, the camera control unit 201 determines whether the file formats that can be converted in the image processing unit 203 include a file format that can be displayed stereoscopically by the head mounted display 4. This can be determined based on the characteristic information acquired in step 1108. If the result is YES in step 1110, the camera control unit 201 advances the flow to step 1111, and causes the image processing unit 203 to convert the file format of the image data into a file format that can be displayed stereoscopically by the head mounted display 4. Then, the flow advances to step 1112. If the result is NO in step 1110, the camera control unit 201 notifies the user that the captured image data cannot be displayed on the head mounted display 4, and returns the flow to step 1102. Here, the camera control unit 201 may transmit an upgrade request to an external application server via the wired communication unit 205 or the wireless communication unit 206. In response to the upgrade request, the external application transmits data for upgrading the application to the camera body 3, which can be converted into a file format that can be displayed by the head mounted display 4, as indicated by the characteristic information. The camera control unit 201 uses the received data to upgrade the application in the image processing unit 203 .

[0072] As described above, in this embodiment, the camera body 3 converts the file format of image data for a stereoscopic image in accordance with the characteristics of the head mounted display 4. In this way, it becomes possible to perform stereoscopic display even if the head mounted display 4 does not have a file format conversion function.

[0073] Furthermore, if the camera body 3 has acquired characteristic information of the head mounted display 4 in advance at the time of shooting, the camera body 3 may generate image data that matches the characteristics of the head mounted display 4 in parallel with shooting. However, in a mode with a high processing load of image processing such as video shooting, the process of generating image data that matches the characteristics of the head mounted display 4 in advance in parallel with shooting may be prohibited. In this case, depending on the type of shooting and settings, there are cases where image data that matches the characteristics of the head mounted display 4 can be generated in advance and cases where it cannot. Therefore, if image data that matches the characteristics of the head mounted display 4 has not been generated, the user may be notified of this fact.

[0074] <Fifth embodiment> Next, a fifth embodiment will be described. In the fifth embodiment, image data captured by a camera body 3 equipped with a twin lens 2 is transmitted to a personal computer 5. The head mounted display 4 notifies the personal computer 5 of its own characteristic information, and the personal computer 5 converts the file format based on the capability information. The personal computer 5 then transmits the converted file to the head mounted display 4.

[0075] FIG. 12 is a flowchart showing an example of the flow of processing of the personal computer 5 in the fifth embodiment.

[0076] 12, the CPU 501 waits until it receives file information from the camera body 3. For example, the camera control unit 201 of the camera body 3 transmits file information to the personal computer 5 every time a photograph is taken. In step 1202, the CPU 501 stores the received file information in the data memory 503 or the storage unit 506. The received file information includes captured image data and metadata. The metadata includes information indicating the file format of the image data.

[0077] In step 1203, the CPU 501 requests the head mounted display 4 to transmit characteristic information of the HMD. In step 1204, the CPU 501 waits until it receives characteristic information from the head mounted display 4, and advances the flow to step 1205 upon receiving the characteristic information.

[0078] In step 1205, the CPU 501 determines, based on the received characteristic information, whether the head mounted display 4 has a function for performing stereoscopic display based on the image data received in step 1201. If the CPU 501 determines YES in step 1205, it advances the flow to step 1207, and if the CPU 501 determines NO, it advances the flow to step 1206.

[0079] In step 1206, the CPU 501 converts the file format of the image data received in step 1201 into a file format that can be displayed stereoscopically by the head mounted display 4, and also rewrites the metadata to content corresponding to the converted image data.

[0080] In step S1207, the image data after the file format conversion and the file information including the rewritten metadata are transmitted to the head mounted display 4.

[0081] As described above, according to this embodiment, even if the available file formats are different between the camera body 3 and the head mounted display 4, a stereoscopic image can be displayed on the head mounted display 4. Note that, in this embodiment, an example has been described in which file information including image data and metadata is sent from the camera body 3 to the personal computer 5, but this is not limited to this. If the head mounted display 4 cannot display a stereoscopic image using the file information received from the camera body 3, the head mounted display 4 may send file information including image data and metadata to the personal computer 5. When the personal computer 5 receives image data from the head mounted display 4, it converts the image data to a file format compatible with the head mounted display 4 and modifies the metadata. Thereafter, the personal computer 5 transmits the converted image data and the associated metadata to the head mounted display 4.

[0082] <Other embodiments> Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the above-mentioned embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. The present invention can also be realized by supplying a program that realizes one or more functions of the above-mentioned embodiments to a system or device via a network or storage medium, and having one or more processors of a computer in the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0083] 1 System 2. Dual Lens 3. Camera body 4. Head-mounted display 5. Personal Computers 201 Camera control unit 401 CPU 501 CPU

Claims

1. a receiving means for receiving, from an imaging device equipped with a twin lens, image data including a first image area and a second image area formed through a right-eye optical system and a left-eye optical system of the twin lens, respectively, and an image file including metadata related to the image data; a display control means for displaying a stereoscopic image on a display means using the image data; a first determination means for determining whether an application needs to be updated based on a file format of the image file indicated by the metadata; an upgrade means for acquiring predetermined data from an external device and executing a process for upgrading the application; a second determination means for determining whether the first image area and the second image area in the image data are in a reversed state with respect to the left and right by analyzing the image data and comparing the first image area and the second image area after the application has been upgraded; an image processing means for executing a process of swapping the first image area and the second image area in the image data when a result of the determination by the second determination means indicates that the first image area and the second image area in the image data are in a reversed state with respect to left and right, The image processing device is characterized in that the display control means causes the display means to display the stereoscopic image using image data in which a process has been performed to swap the first image area and the second image area left and right in the image data.

2. 2. The image processing apparatus according to claim 1, further comprising a control unit that requests the external device on a network to upgrade the application.

3. 3. The image processing device according to claim 1, wherein the image data is in a file format of image data generated by capturing a specific subject image with one sensor via each of the right eye optical system and the left eye optical system.

4. 4. The image processing apparatus according to claim 1, further comprising the display means.

5. An imaging element; a generating means for generating a file in a predetermined file format using image data generated from the imaging element; a receiving means for receiving characteristic information indicating a processing capacity of a display means for displaying a stereoscopic image based on image data including a plurality of images having different viewpoints; an image processing unit that performs a process of converting the file generated by the generating unit into a file format that can be displayed three-dimensionally by the display unit when the characteristic information does not include information corresponding to the ability to perform three-dimensional display using the file generated by the generating unit; When an image is captured using a lens having a plurality of optical paths, the images of the subject guided by the respective optical paths are formed at different positions on the image sensor, The image processing device according to claim 1, wherein the generating means generates a file having image data obtained by vertically and horizontally inverting an image of a subject formed on the imaging element.

6. The image processing device according to claim 5, characterized in that the generation means generates a file in which metadata indicating that, when an image is captured using a lens including a first optical path and a second optical path arranged in the horizontal direction, the horizontal positions of the first optical path and the second optical path and the horizontal positions of the image of the subject corresponding to the first optical path and the image of the subject corresponding to the second optical path in the image data are different image data.

7. The image processing device described in any one of claims 1 to 4, characterized in that the second judgment means judges that the first image area and the second image area are in a reversed state from left to right if the position of a specified subject image in the left image area of ​​the first image area and the second image area is shifted to the left relative to the background compared to the position of the specified subject image in the right image area.

8. 8. The image processing apparatus according to claim 7, wherein the predetermined subject image is an image of a subject that is the target of AF (autofocus).

9. a receiving step of receiving, from an imaging device equipped with a twin lens, image data including a first image area and a second image area formed through a right-eye optical system and a left-eye optical system of the twin lens, respectively, and an image file including metadata related to the image data; a display control step of displaying a stereoscopic image on a display means using the image data; a first determination step of determining whether an application needs to be updated based on a file format of the image file indicated by the metadata; an upgrade process for acquiring predetermined data from an external device and executing a process for upgrading the application; a second determination step of, after the application has been upgraded, analyzing the image data and comparing the first image area with the second image area to determine whether the first image area and the second image area in the image data are in a reversed state with respect to left and right; an image processing step of executing a process of left-right swapping the first image area and the second image area in the image data when a result of the second judgment step indicates that the first image area and the second image area in the image data are in a reversed state with respect to left and right, The image processing method is characterized in that, in the display control process, the stereoscopic image is displayed on the display means using image data in which a process has been performed to swap the first image area and the second image area left and right in the image data.

10. A computer-readable storage medium storing a program for causing a computer to execute an image processing method, The program is a receiving step of receiving, from an imaging device equipped with a twin lens, image data including a first image area and a second image area formed through a right-eye optical system and a left-eye optical system of the twin lens, respectively, and an image file including metadata related to the image data; a display control step of displaying a stereoscopic image on a display means using the image data; a first determination step of determining whether an application needs to be updated based on a file format of the image file indicated by the metadata; an upgrade process for acquiring predetermined data from an external device and executing a process for upgrading the application; a second determination step of, after the application has been upgraded, analyzing the image data and comparing the first image area with the second image area to determine whether the first image area and the second image area in the image data are in a reversed state with respect to left and right; an image processing step of executing a process of left-right swapping the first image area and the second image area in the image data when a result of the second judgment step indicates that the first image area and the second image area in the image data are in a reversed state with respect to left and right, A storage medium characterized in that in the display control process, the stereoscopic image is displayed on the display means using image data in which a process has been performed to swap the first image area and the second image area left and right in the image data.

11. A program for causing a computer to execute each means of an image processing device described in any one of claims 1 to 8.

Images