Image generation system, image generation method, and program

Abstract

Provided is an image generation system including an imaging device having an imaging element, the system comprising: an acquisition means for acquiring an image generated by projecting incident light from an optical system based on an equidistant projection method onto the imaging element; a detection means for detecting a posture change of the imaging device; a control means for performing a first blur correction process for moving a correction member including at least one of a lens included in the optical system or the imaging element in a plane perpendicular to the optical axis of the optical system to correct blurring of the image, caused by the posture change, while capturing the image; and a conversion means for performing a conversion process for geometrically converting the image to correct a distortion of the image caused by the equidistant projection method. The conversion process includes an image correction process for determining a position before conversion corresponding to a reference position after conversion in the image on the basis of the posture change and the correction amount of the first blur correction process.

Description

Image generation system, image generation method, and program

[0001] The present invention relates to an image generation system, an image generation method, and a program.

[0002] Systems for capturing images for use in virtual reality (VR) using stereoscopic imaging devices are known. For example, Patent Document 1 discloses a lens device in which two optical systems are arranged in parallel and two image circles are formed in parallel on a single imaging element. The imaging device records, as viewed from the imaging surface, an image formed by the right optical system as a video or still image for the right eye (collectively referred to as an "image"), and an image formed by the left optical system as an image for the left eye. When playing back the images, a user (viewer) views the images using a known 3D display or so-called VR goggles. In this case, an image for the right eye is projected onto the user's right eye, and an image for the left eye is projected onto the user's left eye. Due to the baseline lengths of the two optical systems of the lens device, images with parallax are projected onto the right and left eyes, allowing the user to experience a sense of three-dimensionality.

[0003] In order for a user to feel not only a three-dimensional effect but also a sense of realism when viewing an image through VR goggles, it is desirable that the angle of view of the captured image be 180 degrees or more.

[0004] Images expressed using equirectangular projection are generally used as input images to devices used for VR viewing, such as VR goggles. An equidistant projection optical system (e.g., a fisheye lens) is used to capture an image with a field of view of approximately 180 degrees. By converting a distorted image captured with a fisheye lens into an equirectangular projection image, an image similar to what the human eye sees can be displayed on a display corresponding to each eye.

[0005] Furthermore, a known technique for correcting image blur caused by changes in the posture of an imaging device is to move a blur correction lens or an imaging element included in an optical system within a plane perpendicular to the optical axis during shooting.

[0006] JP 2023-37539 A

[0007] When the image stabilization lens or the image sensor is moved during image capture using a fisheye lens, it is difficult to perform effective image correction such as image stabilization.

[0008] The present invention has been made in view of the above circumstances, and provides an image correction technique for photography using an equidistant projection optical system.

[0009] In order to solve the above-mentioned problems, the present invention provides an image generation system including an imaging device having an image sensor, comprising: acquisition means for acquiring an image formed by projecting incident light from an equidistant projection optical system onto the image sensor; detection means for detecting a change in attitude of the imaging device; control means for performing a first blur correction process that moves a correction member including at least one of a lens included in the optical system and the image sensor within a plane perpendicular to the optical axis of the optical system during capture of the image so as to correct blur in the image caused by the change in attitude; and conversion means for performing a conversion process that geometrically converts the image so as to correct distortion in the image caused by the equidistant projection method, wherein the conversion process includes an image correction process that determines a pre-conversion position in the image that corresponds to a post-conversion reference position based on the change in attitude and the correction amount of the first blur correction process.

[0010] According to the present invention, it is possible to provide an image correction technique for photography using an equidistant projection optical system.

[0011] Other features and advantages of the present invention will become more apparent from the accompanying drawings and the following detailed description of the preferred embodiment of the present invention.

[0012] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0013] 1 is a schematic cross-sectional view of an interchangeable lens 200 (lens device). FIG. 1 is a diagram showing a schematic configuration of an imaging device 100. FIG. 2 is a diagram showing the configuration of a video generation system. FIG. 3 is a diagram showing an example of a frame before equirectangular conversion. FIG. 4 is a diagram showing an example of a frame after equirectangular conversion. FIG. 5 is a diagram explaining a first shake correction process. FIG. 6 is a diagram explaining a first shake correction process. FIG. 7 is a diagram explaining a second shake correction process. FIG. 8 is a diagram explaining a second shake correction process. FIG. 9 is a diagram explaining a first shake correction process. FIG. 10 is a diagram explaining the roles of the first shake correction process and the second shake correction process during video shooting. FIG. 11 is a diagram showing a layout image of an attitude detection unit 209 and an optical system for explaining a method of calculating the correction amount for the second shake correction process. FIG. 11 is a diagram explaining a first shake correction process corresponding to an attitude change in the roll direction. FIG. 12 is a diagram explaining a first shake correction process corresponding to an attitude change in the roll direction.

[0014] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0015] In the following embodiment, a case where photography is performed using two equidistant projection optical systems (e.g., fisheye lenses) arranged horizontally will be mainly described. However, the direction in which the two optical systems are arranged is not limited to the horizontal direction. Furthermore, the number of optical systems is not limited to two, and may be one, or three or more.

[0016] [First embodiment] Configuration of interchangeable lens 200 Fig. 1 is a schematic cross-sectional view of an interchangeable lens 200 (lens device). The interchangeable lens 200 is an equidistant projection lens. In the following description, the interchangeable lens 200 is assumed to be a fisheye lens.

[0017] The interchangeable lens 200 has two optical systems (a first optical system and a second optical system) arranged parallel (symmetrically) to each other, and is configured to form two parallel image circles on a single imaging element. The two optical systems are arranged horizontally, separated by a predetermined distance (baseline length L1). When viewed from the imaging surface, the image formed by the right optical system (the first optical system) is recorded as an image for the right eye, and the image formed by the left optical system (the second optical system) is recorded as an image for the left eye. When playing back the images, the user views them using a known 3D display or so-called VR goggles. In this case, the image for the right eye is projected onto the user's right eye, and the image for the left eye is projected onto the user's left eye. Due to the baseline length L1 of the interchangeable lens 200, images with parallax are projected onto the right and left eyes, allowing the user to experience a three-dimensional effect. In this way, the interchangeable lens 200 is a lens device for stereoscopic photography capable of forming two images with parallax using the first optical system and the second optical system. In this embodiment, moving images and still images are collectively referred to as images, and when not limited to either moving images or still images, the term may refer to either moving images or still images.

[0018] As shown in Fig. 1, the interchangeable lens 200 has a right-eye optical system 201R and a left-eye optical system 201L. In the following, the right-eye optical system 201 will be described with an R suffix, and the left-eye optical system 201L will be described with an L suffix. Descriptions common to both the right-eye optical system 201R and the left-eye optical system 201L will not be described with an R or an L suffix. In Fig. 1, the letters "R" and "L" are omitted from the reference numerals except for some components.

[0019] The right-eye optical system 201R and the left-eye optical system 201L are each capable of capturing images with an angle of view of 180 degrees or more. Each optical system is set with, in order from the subject side, a first optical axis OA1, a second optical axis OA2 that is substantially perpendicular to the first optical axis, and a third optical axis OA3 that is parallel to the first optical axis. Each optical system includes a lens group 211 having a convex surface 211A facing the subject along the first optical axis OA1, a lens group 221 along the second optical axis OA2, and lens groups 231 and 231-2 along the third optical axis OA3. Each optical system further includes a first prism 220 that bends the light beam of the first optical axis OA1 and guides it to the second optical axis OA2, and a second prism 230 that bends the light beam of the second optical axis OA2 and guides it to the third optical axis OA3.

[0020] 2 is a diagram showing a schematic configuration of the imaging device 100. The imaging device 100 has a camera body 110 and an interchangeable lens 200. The interchangeable lens 200 is detachable from the camera body 110.

[0021] In the imaging device 100 , incident light from the right-eye optical system 201 R and the left-eye optical system 201 L that constitute the imaging optical system forms an image on the imaging element 111 .

[0022] In the interchangeable lens 200, a lens control unit 203 processes values ​​detected by various detection units, such as a temperature detection unit 207 and a focus detection unit 208. The lens control unit 203 sends information obtained through this processing to the system control unit 117 and controls the imaging device 100 in cooperation with the system control unit 117. The interchangeable lens 200 also includes a storage unit 204. Information exchanged between the lens control unit 203 and the system control unit 117 is stored in the storage unit 204 as necessary. In response to a request from the system control unit 117, the lens control unit 203 transmits to the system control unit 117 information obtained through detection by the temperature detection unit 207 and the focus detection unit 208, individual identification information of the interchangeable lens 200 (such as lens individual information 205 and manufacturing error information 206), attitude information acquired by an attitude detection unit 209, and information such as the drive amount of the image stabilization lens drive unit 210.

[0023] If the user unintentionally causes blur when taking a picture, the lens control unit 203 sends a command to the blur correction lens drive unit 210 according to attitude information of the interchangeable lens 200 detected by the attitude detection unit 209, and drives the lens to correct the blur. The lens control unit 203 can transmit the drive amount of the blur correction lens drive unit 210, the attitude information detected by the attitude detection unit 209, the temperature detected by the temperature detection unit 207, the focus position detected by the focus detection unit 208, lens individual information 205, and manufacturing error information 206 to the camera body 110.

[0024] The camera body 110 has an image sensor 111, an A / D conversion unit 112, an image processing unit 113, a display unit 114, an operation unit 115, a recording unit 116, a system control unit 117, a memory unit 118, an attitude detection unit 119, an image sensor driving unit 120, and a camera mount 122.

[0025] When the lens mount 202 of the interchangeable lens 200 is attached to the camera mount 122 of the camera body 110, the system control unit 117 and the lens control unit 203 are electrically connected.

[0026] As images of the subject, a right eye image formed via the right eye optical system 201R and a left eye image formed via the left eye optical system 201L are formed side by side on the image sensor 111. The image sensor 111 converts the formed image of the subject (optical signal) into an analog electrical signal. The A / D conversion unit 112 converts the analog electrical signal output from the image sensor 111 into a digital electrical signal (image signal). The image processing unit 113 performs various image processing on the digital electrical signal (image signal) output from the A / D conversion unit 112.

[0027] The display unit 114 displays various types of information. The display unit 114 is realized by using, for example, an electronic viewfinder or a liquid crystal panel. The operation unit 115 functions as a user interface that enables the user to give instructions to the imaging device 100. Note that, if the display unit 114 has a touch panel, the touch panel is a type of operation unit 115.

[0028] The recording unit 116 records various data such as image data that has been subjected to image processing by the image processing unit 113. The recording unit 116 also stores programs executed by the system control unit 117. The recording unit 116 is realized by using, for example, a ROM, a RAM, and a HDD.

[0029] The system control unit 117 controls the entire image capturing apparatus 100. The system control unit 117 is realized by using, for example, a CPU.

[0030] Storage unit 118 stores individual identification information (such as individual camera information 123 and manufacturing error information 124) of camera body 110. The individual identification information of camera body 110 is stored in storage unit 118 during the manufacturing process of camera body 110. The individual identification information of camera body 110 includes, for example, model information of camera body 110, information on the number of pixels of image sensor 111, information on the physical size of image sensor 111, and manufacturing error information for captured images.

[0031] The orientation detection unit 119 detects the orientation of the camera body 110 and passes the detected orientation information to the system control unit 117. Upon receiving a command from the system control unit 117, the image sensor drive unit 120 moves the image sensor 111 to perform shake correction. The shake correction process in which the image sensor drive unit 210 drives the shake correction lens in the lens group 231 and the shake correction process in which the image sensor drive unit 120 drives the image sensor 111 are collectively referred to as the first shake correction process. Therefore, when the first shake correction process is performed in this embodiment, one or both of the shake correction process in which the image sensor 111 is driven and the shake correction process in which the image sensor 111 is driven are performed. When both the shake correction process in which the image sensor 111 is driven and the shake correction process in which the image sensor 111 is driven are performed, the system control unit 117 determines the amount of drive of the image sensor 111 based on the amount of shake correction on the interchangeable lens 200 side received from the lens control unit 203.

[0032] An image captured by imaging device 100 is recorded in recording unit 116. At this time, system control unit 117 records, as metadata, information such as attitude information of interchangeable lens 200, attitude information of camera body 110, correction information indicating the correction amount of the first shake correction process (for example, the drive amount of the shake correction lens and the drive amount of image sensor 111), individual identification information of interchangeable lens 200, and individual identification information of camera body 110, in association with the captured image. For example, recording unit 116 generates and records an image file including the captured image and the metadata, thereby associating the metadata with the captured image.

[0033] Configuration of the Image Generation System Fig. 3 is a diagram showing the configuration of the image generation system. The image generation system includes a personal computer (PC 300) as an image processing device that performs geometric transformation on images (captured images) acquired by the image capture device 100. The following describes a case where the image capture device 100 captures a video. In the PC 300, at least some of the functions of the metadata extraction unit 301, attitude calculation unit 304, shake correction amount calculation unit 305, and geometric transformation unit 306 can be realized, for example, by a processor included in the PC 300 operating in accordance with a program.

[0034] The imaging device 100 captures a moving image using the imaging element 111 and generates a moving image file including the captured moving image and metadata.

[0035] The metadata extraction unit 301 separates and extracts a video 302 and metadata 303 from a video file generated by the imaging device 100. As described above, the metadata 303 includes orientation information of the interchangeable lens 200 and orientation information of the camera body 110.

[0036] Note that when so-called stereoscopic photography is performed using two optical systems, information on the three-dimensional distance to each subject is also obtained in addition to planar image information. Therefore, the PC 300 can estimate the orientation of the camera body 110 based on the image without using orientation information. Furthermore, even for images captured using a single lens, a method for detecting the distance to each subject in the image sensor is used in autofocus detection technology, etc. By utilizing these, it is also possible to detect the orientation. Therefore, in this embodiment, the orientation information acquired by the orientation detection unit 209 and the orientation detection unit 119 is not essential.

[0037] Furthermore, because the camera body 110 and the interchangeable lens 200 are fixed together, the orientation information acquired by the camera body 110 and the interchangeable lens 200 generally coincides. Therefore, the PC 300 may use only one of the orientation information of the interchangeable lens 200 and the orientation information of the camera body 110 as the orientation information of the imaging device 100. Furthermore, the camera body 110 may record only one of the orientation information of the interchangeable lens 200 and the orientation information of the camera body 110 as the orientation information of the imaging device 100 in the metadata.

[0038] The attitude calculation unit 304 can acquire time-series changes in attitude during video capture using the attitude information included in the metadata 303. As an example, the types of attitude information recorded include time-series angular velocity information obtained by a gyro sensor and information on acceleration acting on the image capture device 100 acquired by an acceleration sensor. The attitude calculation unit 304 can perform highly accurate attitude estimation by fusing information acquired from multiple sensors and attitude information estimated from images using a technique such as so-called sensor fusion. Any known technique can be used as the attitude estimation method.

[0039] The shake correction amount calculation unit 305 calculates the component of the user's intended movement and the component of the shake in the video, based on the posture estimated by the posture calculation unit 304 .

[0040] The geometric transformation unit 306 performs image correction on the video 302 by performing a geometric transformation that takes into account the blur components calculated (acquired) by the blur correction amount calculation unit 305. The geometric transformation is performed on each frame of the video. The image correction process performed in conjunction with the geometric transformation by the geometric transformation unit 306 is referred to as a second blur correction process. The geometric transformation performed is equirectangular transformation (conversion to an image expressed using equirectangular projection, which is commonly used in VR). However, the geometric transformation of this embodiment is not limited to equirectangular transformation. The geometric transformation unit 306 can perform any type of geometric transformation as long as it is a process (conversion process) that geometrically transforms the video so as to correct distortion of the video caused by the equidistant projection method. Because the interchangeable lens 200 is a fisheye lens (a lens using the equidistant projection method), the images captured by the imaging device 100 are images using the equidistant projection method. Therefore, in order to display the captured image on the head-mounted display (HMD 400), it is necessary to convert the captured image using a geometric image transformation such as equirectangular projection. At this time, the geometric transformation unit 306 performs the geometric image transformation while correcting manufacturing errors, by referencing the manufacturing error information of the camera body 110 and the interchangeable lens 200 recorded in the metadata described above. Furthermore, the geometric transformation unit 306 performs geometric image transformation on each of multiple frames of a video captured in time series, in accordance with the time-series shake correction amount described above (the correction amount of the first shake correction process), and outputs a converted video 307 expressed in equirectangular projection.

[0041] The HMD 400, which is a display device for VR images, includes an HMD posture detection unit 401, a video conversion unit 402, and a video display unit 403. At least some of the functions of the HMD posture detection unit 401, the video conversion unit 402, and the video display unit 403 can be realized, for example, by a processor included in the HMD 400 operating in accordance with a program.

[0042] The HMD posture detection unit 401 includes various sensors that detect the direction in which the user is facing while wearing the HMD 400, and estimates the posture of the HMD 400 based on information detected by the various sensors.

[0043] The video conversion unit 402 performs image conversion to generate an image that references a specific region of the converted video 307, depending on the orientation of the HMD 400 calculated (estimated) by the HMD orientation detection unit 401. The geometric conversion unit 306 generates frames of the converted video 307, such as those shown in Fig. 5, using images (frames) of an area with a 180-degree angle of view, such as that shown in Fig. 4. The video conversion unit 402 extracts images of an area displayed by the HMD 400 from the converted video 307. Generally, the area of ​​the display image on the HMD 400 is an area with a field of view of approximately 100 degrees, so the video conversion unit 402 generates an image corresponding to the 100-degree area of ​​the 180-degree converted video 307.

[0044] The video display unit 403 includes, for example, a flat display. The video conversion unit 402 displays a central projection image generated from the converted video 307 on the video display unit 403. The video conversion unit 402 plays back VR images by converting and displaying frames in chronological order in accordance with the direction the user is facing, which is detected by the HMD posture detection unit 401. This allows the user to view VR images.

[0045] First Image Blur Correction Process Next, the first image blur correction process will be described with reference to Figures 6A to 6C. Figure 6A is a diagram illustrating one side of the fisheye optical system of the interchangeable lens 200. To explain the first image blur correction process, the center of the image sensor 111 and the center of the third optical axis are simply aligned in the diagram.

[0046] 6B shows an example of a first blur correction process that performs blur correction by moving a part of the optical system of the interchangeable lens 200 (the blur correction lens) in a plane perpendicular to the optical axis. Consider a situation in which blur occurs in the imaging device 100 including the interchangeable lens 200, causing the lens optical axis to point away from the subject being photographed. The subject is projected onto the lens optical axis at a tilted angle equal to the amount of blur. In this case, the subject that was originally intended to be photographed on the optical axis is photographed shifted by the amount of blur. Moving the blur correction lens in the optical system, as shown in FIG. 6B , changes the optical path, correcting blur.

[0047] 6C shows an example of a first blur correction process that performs blur correction by driving the image sensor 111 in a plane perpendicular to the third optical axis of the optical system. Consider a situation in which blur occurs in the image capture device 100, including the interchangeable lens 200, causing the lens optical axis to point away from the subject being captured. The subject is projected onto the lens optical axis of the image sensor 111 at a tilted angle corresponding to the amount of blur. In this case, the subject that was originally intended to be captured on the optical axis is captured shifted by the amount of blur. By moving the image sensor 111 as shown in FIG. 6C, the subject is projected in the same position as before the blur occurred, thereby correcting the blur.

[0048] Second Shake Correction Process Next, the second shake correction process will be described with reference to FIGS. 7A to 7D. FIG. 7A shows an image captured before shake occurs. Here, assuming a video, the first frame of the video is in the state shown in FIG. 7A. FIG. 7B shows an image captured when shake occurs and the image capture device 100 is tilted relative to the state shown in FIG. 7A. In the equidistant projection method, the image height changes in proportion to the angle. Therefore, if the shake angle is θ and the diameter of the image circle with a 180-degree angle of view is D, the amount of change on the image circle is D / 2×θ degrees / 180 degrees. Therefore, point A, which is located at the center of the optical axis in FIG. 7A, moves by D / 2×θ degrees / 180 degrees due to the occurrence of shake. This causes the viewpoint to fluctuate between frames of the video. When the VR image is ultimately displayed on the HMD 400, this shake may cause nausea in the viewer. To correct this shake, the second shake correction process is performed. 7C and 7D, when converting an image into an equirectangular image, the PC 300 performs conversion processing so that point A is located at the center of the equirectangular image. This makes it possible to suppress blurring of moving images expressed using equirectangular projection.

[0049] When performing equirectangular conversion, PC 300 acquires from the metadata information about the center position of the image circle of the image captured with the fisheye lens, the focal length and amount of distortion of the lens, and individual lens information and manufacturing error information related to the tilt of the optical axis, etc. Furthermore, due to the first shake correction process, the center position of the image circle of the fisheye lens is shifted by the amount of movement of the image sensor 111. PC 300 calculates the center position of the image circle for each frame of the video, taking into account the correction amount of the first shake correction process (e.g., the drive amount of the shake correction lens and the drive amount of the image sensor 111), and performs equirectangular conversion based on the calculated center position.

[0050] Correction of blur during exposure In the second image stabilization process, a geometric transformation is performed on each frame of a video that has already been captured. Therefore, the second image stabilization process cannot effectively correct blur during exposure (image blur caused by changes in the posture of the image capture device 100 that occur during the exposure period of the image sensor 111).

[0051] On the other hand, the first image stabilization process moves a stabilization member including at least one of the image stabilization lens and the image sensor 111 during exposure, thereby correcting blur during exposure. However, when a fisheye lens is used, the amount of movement on the image circle due to subject movement differs depending on the radial and circumferential directions from the center of the image circle. FIG. 8 is a schematic diagram illustrating this. For example, consider a case where blur occurs in the Y direction in FIG. 8. If the image capture device 100 tilts by an angle θ, and the central portion of the captured image moves by Y1, the peripheral portion moves only by Y2, which is smaller than Y1. As a result, if the image sensor 111 is moved significantly, blur near the center of the screen is corrected, but greater blur than before correction may occur near the periphery.

[0052] Here, it is considered that the central area of ​​an image viewed by a user is viewed more frequently. Furthermore, the resolution of an optical system is often inferior in the peripheral area compared to the central area. Therefore, some blurring in the peripheral area is acceptable, and suppressing blurring during exposure in the central area leads to substantial image quality improvement. For example, when a user wears the HMD 400 and an image is played back, an area of ​​approximately 100 degrees from the center is often displayed. Regarding the content being filmed, for example, if a performer positioned in the center is being filmed, it is considered that the user will likely focus primarily on the central area. Because parallax is smaller in the peripheral area than in the central area, it is known that a range of approximately 120 degrees provides a sense of three-dimensionality for the user. Therefore, the system control unit 117 determines the correction amount for the first blur correction process so as to minimize blurring in this range.

[0053] For example, the system control unit 117 determines a first change amount and a second change amount in the video caused by a change in the posture of the image capture device 100, and determines a correction amount for the first shake correction process so as to correct an amount that is smaller than the first change amount and larger than the second change amount. The first change amount is the change amount of a first position (e.g., the center position of the video) caused by a change in the posture of the image capture device 100. The second change amount is the change amount of a second position (e.g., a peripheral portion of the video) caused by a change in the posture of the image capture device 100. The second position is farther from the center of the image than the first position. Here, by setting the image height of the peripheral portion corresponding to the second change amount to within a 100-degree range of the field of view typically used in the HMD 400, it becomes possible to correct shake during exposure in a range where the user is likely to gaze.

[0054] Using the First and Second Shake Correction Processes Together As described above, imaging device 100 uses the first shake correction process to suppress shake during exposure to some extent. Furthermore, PC 300 uses the second shake correction process to correct for relatively large fluctuations in shake. In this way, using the first and second shake correction processes together can achieve a favorable shake correction effect.

[0055] FIG. 9 is a diagram illustrating the respective roles of the first shake correction process and the second shake correction process during video capture. The first shake correction process is applied only to the amount of shake that occurs during exposure. Generally, the frame rate of a video is approximately 30 to 60 frames per second, or approximately 30 fps to 60 fps. On the other hand, if the exposure time is too short, the connection between frames in the video may be poor, resulting in a choppy appearance. Therefore, it is considered appropriate for the exposure time to be somewhat close to the frame interval (for example, approximately 1 / 30 to 1 / 100 seconds for 30 fps).

[0056] Furthermore, in the case of a fisheye lens that covers a 180-degree area, significant blurring does not occur during exposure with an exposure time of about 1 / 100 seconds. Therefore, the imaging device 100 performs correction using the first image stabilization process during the exposure time or for a shorter period. Meanwhile, because the second image stabilization process is also used, the imaging device 100 issues a still command to stop the first image stabilization process for the remaining period (outside the exposure time). This makes it possible to prevent differences in the blur stabilization effect between the center and the periphery, as described with reference to FIG. 8 .

[0057] The second shake correction process is performed to correct the remaining shake not corrected by the first shake correction process. To do this, the PC 300 references the orientation information and the correction amount of the first shake correction process included in the metadata. That is, the geometric transformation unit 306 determines the pre-conversion position (the position of point A in the example of FIG. 7B ) corresponding to the post-conversion reference position (the center position in the example of FIG. 7D ) in each frame of the video based on the change in orientation of the imaging device 100 and the correction amount of the first shake correction process. The metadata may also include information on the exposure time. This allows the PC 300 to determine the correction amount of the second shake correction process by referring to the orientation information, the correction amount of the first shake correction process, and the exposure time, respectively.

[0058] Correction Amounts in Tilt and Pan Directions in Second Blur Correction Process FIG. 10 is a diagram showing an image of the layout of the orientation detection unit 209 and optical system to explain the calculation method of the correction amount in the second blur correction process. A fisheye lens is used for the optical system. The positions where light rays roughly intersect with the subject at each angle of view are designated as points B and C in the right and left optical systems, respectively, as shown in the figure. Meanwhile, the detection center point of the orientation detection unit 209 of the interchangeable lens 200 is designated as center point D. In the second blur correction process, blur in the direction of rotation around point B or point C is corrected by correcting the center position when converting to an equirectangular image according to the angle of shake.

[0059] On the other hand, the second blur correction process cannot correct blur in the direction that changes the positions of points B and C. This is because the amount of correction differs depending on whether the subject in the captured image is located close or far away.

[0060] Furthermore, in this embodiment, which includes multiple optical systems, a distance exists between points B and C. Generally, in VR viewing, a method is used to display a stereoscopic image using the parallax resulting from this difference in distance. When the center point D of the interchangeable lens 200 is positioned approximately between points B and C as shown in FIG. 10 , a change in the orientation of the center point D not only causes points B and C to rotate around their respective centers, but also causes each point to move in accordance with its distance from point D. Generally, when performing stereoscopic imaging for VR, the lenses are aligned horizontally. The second shake correction process performs a tilt correction process to correct shake in the tilt direction when the orientation is tilted in the direction of gravity (the so-called tilt direction), and a pan correction process to correct the direction of panning (i.e., when the camera is moved horizontally). When the orientation at the center point D is tilted in the direction of gravity (the so-called tilt direction), the PC 300 simply uses the orientation detection value at point D for correction. However, in the direction perpendicular to the direction of gravity (the so-called panning direction), the PC 300 must perform correction taking into account the amount of movement of points B and C, in addition to correcting for the amount of posture change at center point D. Strictly speaking, the amount of correction varies depending on the distance to the subject. Therefore, as described above, unlike the correction for the rotational direction, the correction for the movement direction may not be performed appropriately. However, by performing the above-described correction for the movement direction in addition to the correction for the rotational direction, a shake correction effect can be achieved. As described above, by making the correction amount for the detected value of the posture change for panning greater than the correction amount for the detected value of the posture change for tilt, it is possible to obtain images with good correction results for both panning and tilting. In other words, the ratio of the correction amount of the second shake correction process to the amount of posture change of the imaging device 100 in the panning direction is greater than the ratio of the correction amount of the second shake correction process to the amount of posture change of the imaging device 100 in the tilt direction.

[0061] Although the case where the orientation detection unit 209 of the interchangeable lens 200 is used has been described here, the same can be said for the case where the orientation detection unit 119 of the camera body 110 is used.

[0062] First shake correction process corresponding to attitude changes in the roll direction FIGS. 11A to 11C are diagrams illustrating the first shake correction process corresponding to attitude changes in the roll direction. FIG. 11A shows one frame of a video captured in an initial attitude before blurring occurs. FIG. 11B shows one frame of a video captured when the attitude changes in the direction of rotation around the first optical axis of the optical system (the so-called roll direction). Here, the imaging device 100 captures images of the left and right optical systems using a single image sensor 111. Therefore, incident light from the left optical system is projected to the left on the image sensor 111, and incident light from the right optical system is projected to the right on the image sensor 111.

[0063] In a typical imaging device, the image projected onto the imaging element is assumed to be inverted by the optical system, and image processing is performed to rotate the captured image around the optical axis. In the imaging device 100, image processing is also performed to rotate the image 180 degrees around the center of the imaging element, and the image is output. Therefore, the image circle of the left optical system captured on the left side of the imaging element 111 is converted to the right, and the image circle of the right optical system captured on the right side of the imaging element 111 is converted to the left, and the output image is thus represented with the left and right optical systems swapped. Meanwhile, the images of the left and right optical systems inverted by the optical system and projected onto the imaging element 111 are inverted by image processing, and then converted by restoring the amount of inversion by the optical system. Therefore, an image like that shown in FIG. 11A is output.

[0064] Next, we will explain blur correction when the image changes as shown in FIG. 11B due to blur. When blur occurs in the direction of rotation around the first optical axis, an image like that shown in FIG. 11B is acquired. Essentially, the image capture device 100 corrects blur using a first blur correction process based on orientation information. Therefore, it is conceivable that the image capture device 100 detects a change in orientation caused by rotation around the first optical axis and performs correction corresponding to the detected orientation change. In this case, if possible, it is desirable to perform correction in the direction of rotation around the center points E and F of the respective image circle projection areas, without moving the positions of the center points E and F of each image circle. However, because the image capture device 100 includes multiple optical systems, performing correction in the direction of rotation of the image sensor 111 also moves the center points E and F.

[0065] For example, in FIG. 11A , the angle between the line connecting the center point E of the image circle and point G1 at the top of the building and the vertical line is θ1. Also, in FIG. 11B , the angle between the line connecting the center point E of the image circle and point G2 at the top of the building and the vertical line is θ2. Points H1 and H2 in the left optical system correspond to points G1 and G2, respectively. In this case, if the image sensor 111 is rotated around the center point E of the image circle of the right optical system as shown in FIG. 11C to correct the change in the angle θ2-θ1, the position of the image sensor 111 onto which the center point F of the image circle of the left optical system is projected will move to point F2. As a result, if the first image blur correction process is performed when the orientation of the image capture device 100 changes in the rotational direction around the optical axis, an excessive amount of blur will be generated in the left optical system.

[0066] Therefore, in the case of an imaging device 100 that uses multiple optical systems for a single imaging element 111, when an attitude change occurs that rotates around the first optical axis of an individual optical system, correction of the rotation direction is not performed, and shake correction is performed only for attitude changes other than those in the direction of rotation around the optical axis. The lens individual information 205 of the interchangeable lens 200 records information indicating whether the interchangeable lens 200 has multiple optical systems. The imaging device 100 can determine whether the interchangeable lens 200 (lens device) has multiple optical systems by referencing the lens individual information 205 and based on whether the interchangeable lens 200 holds information indicating that the interchangeable lens 200 has multiple optical systems. Based on this determination result, the imaging device 100 can determine whether or not to perform correction to rotate the imaging element 111 around the optical axis in the first shake correction process.

[0067] On the other hand, in the present embodiment, the interchangeable lens 200 may be configured with a monocular fisheye lens optical system (one optical system). In such a case, information indicating that the interchangeable lens 200 has a monocular optical system (or information indicating that the interchangeable lens 200 does not have multiple optical systems) is recorded in the lens individual information 205. In this case, by referencing the lens individual information 205, the imaging device 100 can determine that the interchangeable lens 200 has a monocular optical system and determine to perform correction by rotating the image sensor 111 around the optical axis in the first image blur correction process.

[0068] Summary of the First Embodiment As described above, according to the first embodiment, an image generation system including an image capture device 100 having an image sensor 111 captures a video using the image sensor 111. Here, incident light from an equidistant projection optical system is projected onto the image sensor 111. The image generation system detects a change in posture of the image capture device 100. The image generation system performs a first shake correction process to move a correction member including at least one of a lens (a shake correction lens) included in the optical system and the image sensor 111 within a plane perpendicular to the optical axis of the optical system to correct image blur caused by a change in posture of the image capture device 100 during image capture. The image generation system also performs a transformation process to geometrically transform the image to correct image distortion caused by the equidistant projection method. The transformation process includes an image correction process to determine a pre-transformation position (e.g., the position of point A in FIG. 7B ) in the image that corresponds to a post-transformation reference position (e.g., the position of point A in FIG. 7D ) based on the change in posture of the image capture device 100 and the correction amount of the first shake correction process, so as to correct image blur.

[0069] With this configuration, according to this embodiment, when capturing an image using an equidistant projection optical system, it is possible to effectively correct image blur caused by changes in the orientation of the imaging device.

[0070] In the above description, the camera body 110 and the interchangeable lens 200 in the imaging device 100 are separable. However, even when an imaging device with an integrated lens is used, it is possible to perform the shake correction of this embodiment. In the above description, the PC 300 performs the geometric transformation processing including the second correction processing. However, the imaging device 100 may be configured to perform the geometric transformation processing including the second correction processing. In this manner, the imaging device 100 may include some or all of the functions of the PC 300. In the above description, the imaging device 100 captures video. However, even when the imaging device 100 captures still images, image shake can be corrected based on a change in the posture of the imaging device 100 and the correction amount of the first shake correction processing.

[0071] The present invention can also be realized by a process in which a program that realizes one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

[0072] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.

[0073] This application claims priority based on Japanese Patent Application No. 2024-027868, filed February 27, 2024, the entire contents of which are incorporated herein by reference.

[0074] 100...imaging device, 110...camera body, 200...interchangeable lens, 300...PC, 400...HMD

Claims

1. An image generation system comprising an imaging device having an image sensor, comprising: acquisition means for acquiring an image formed by projecting incident light from an equidistant projection optical system onto the image sensor; detection means for detecting a change in attitude of the imaging device; control means for performing a first blur correction process for moving a correction member including at least one of a lens included in the optical system and the image sensor within a plane perpendicular to the optical axis of the optical system so as to correct blur in the image caused by the change in attitude while the image is being captured; and conversion means for performing a conversion process for geometrically converting the image so as to correct distortion in the image caused by the equidistant projection method, wherein the conversion process includes an image correction process for determining a pre-conversion position in the image that corresponds to a post-conversion reference position based on the change in attitude and the amount of correction by the first blur correction process.

2. The image generation system according to claim 1, wherein the reference position is a central position.

3. An image generation system as described in claim 1 or 2, further comprising a first recording means for recording correction information indicating the amount of correction of the first blur correction process in association with the image, and wherein the conversion means performs the image correction process based on the correction information.

4. An image generation system as described in any one of claims 1 to 3, further comprising a second recording means for recording posture information indicating the posture change in association with the image, and wherein the conversion means performs the image correction processing based on the posture information.

5. An image generation system according to any one of claims 1 to 4, characterized in that the control means controls the correction member to stop moving during a period when the image sensor is not exposed during capture of the image.

6. The image generation system according to any one of claims 1 to 5, characterized in that the control means determines the amount of change in a first position and the amount of change in a second position of the image caused by the change in posture, the second position being farther from the center of the image than the first position, and the control means determines the amount of correction for the first blur correction process based on the amount of change in the first position and the amount of change in the second position.

7. The image generation system according to claim 6, wherein the control means determines an amount smaller than the amount of change in the first position and larger than the amount of change in the second position as the amount of correction for the first image stabilization process.

8. The image generating system according to claim 6 or 7, wherein the first position is a center position of the image.

9. An image generation system as claimed in any one of claims 1 to 8, characterized in that incident light from each of a plurality of optical systems using an equidistant projection method is projected onto the imaging element, and the control means controls so as not to perform the first blur correction process for blur in the image caused by a change in the attitude of the imaging device corresponding to rotation around the optical axis of an individual optical system among the plurality of optical systems.

10. An image generation system as described in any one of claims 1 to 8, characterized in that incident light from each of one or more equidistant projection optical systems possessed by a lens device is projected onto the imaging element, and the control means determines whether the lens device has multiple optical systems, and if the lens device has multiple optical systems, controls so as not to perform the first blur correction process for blur in the image caused by a change in attitude of the imaging device corresponding to rotation around the optical axis of an individual optical system among the multiple optical systems.

11. The image generation system according to claim 10, wherein the control means determines whether the lens device has multiple optical systems based on whether the lens device holds information indicating that the lens device has multiple optical systems.

12. An image generation system as described in any one of claims 1 to 11, characterized in that incident light from each of two equidistant projection optical systems arranged horizontally is projected onto the image sensor, and the ratio of the correction amount of the image correction process to the amount of change in attitude of the image capture device in the pan direction is greater than the ratio of the correction amount of the image correction process to the amount of change in attitude of the image capture device in the tilt direction.

13. The image generation system according to any one of claims 1 to 12, wherein the transformation process includes equirectangular transformation.

14. An image generation system according to any one of claims 1 to 13, characterized in that the image is a video, and the conversion process includes a second blur correction process that corrects blur between each frame of the video caused by the posture change.

15. An image generation system according to any one of claims 1 to 14, characterized in that the image generation system includes a lens device having one or more equidistant projection optical systems, and the imaging device to which the lens device can be attached or detached.

16. An image generation system according to any one of claims 1 to 15, characterized in that the image generation system comprises an image processing device different from the imaging device, and the image processing device comprises the conversion means.

17. An image generation method executed by an image generation system equipped with an imaging device having an image sensor, comprising: an acquisition step of acquiring an image in which incident light from an equidistant projection optical system is projected onto the imaging element; a detection step of detecting a change in attitude of the imaging device; a control step of performing a first blur correction process of moving a correction member including at least one of a lens included in the optical system and the imaging element within a plane perpendicular to the optical axis of the optical system so as to correct blur in the image caused by the change in attitude while the image is being captured; and a transformation step of performing a transformation process of geometrically transforming the image so as to correct distortion in the image caused by the equidistant projection system, wherein the transformation process includes an image correction process of determining a pre-conversion position in the image corresponding to a post-conversion reference position based on the change in attitude and the correction amount of the first blur correction process.

18. A program for causing a computer to function as each means of the image generation system according to any one of claims 1 to 16.

Images