Image processing device, imaging apparatus, and image processing method
Patent Information
- Application Number
- JP2021192588
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2021-11-26
- Publication Date
- 2024-11-27
- Estimated Expiration
- 2041-11-26
AI Technical Summary
Existing image processing systems risk losing critical information about the imaging device during various processing stages, which is essential for accurate image correction and stereoscopic vision.
An image processing apparatus and method that acquires and records information about the imaging device as an information image, combining it with the captured image to ensure stable storage and retrieval for subsequent image processing.
Ensures stable storage and retrieval of imaging device information, enabling accurate image correction and stereoscopic vision by addressing the loss of metadata during processing.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a technique for performing image processing on a captured image.
Background Art
[0002] As an imaging device, Patent Document 1 discloses an imaging device that captures subject images formed by two optical systems arranged on the left and right with an imaging element to generate left-eye and right-eye imaging images capable of stereoscopic vision. Further, Patent Document 1 also discloses recording information regarding the optical axis deviation between the two optical systems and the aberration of each optical system in an image file together with the captured image, and performing correction processing on the captured image using the information in a subsequent process to enable good stereoscopic vision.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] However, when information is written into an image file together with a captured image as in the imaging device disclosed in Patent Document 1, there is a risk that the information may be lost in the process of performing various processes other than correction processing on the captured image.
[0005] The present invention provides an image processing device or the like capable of stably storing information used for image processing on a captured image. 0]
Means for Solving the Problems
[0006] One aspect of the present invention is an image processing apparatus that processes an image captured by imaging. The image processing apparatus is characterized by comprising: an acquisition means for acquiring information about the imaging apparatus used for imaging; a generation means for generating an information image as an image containing information about the imaging apparatus; and an addition means for adding and recording the information image to the image captured image. An imaging apparatus having the above-mentioned image processing apparatus and an image sensor for capturing an image of a subject formed by an imaging lens also constitutes another aspect of the present invention.
[0007] Furthermore, another aspect of the present invention is an image processing method, which is a method for processing an image captured by imaging. This image processing method is characterized by comprising the steps of: acquiring information about the imaging device used for imaging; generating an information image as an image containing information about the imaging device; and adding the information image to the image captured image and recording it. A program that causes a computer to execute processing according to the above image processing method also constitutes another aspect of the present invention. [Effects of the Invention]
[0008] According to the present invention, information about the imaging device used for image processing of captured images can be added to the imaging device as an information image and recorded, thereby enabling stable storage of information about the imaging device. [Brief explanation of the drawing]
[0009] [Figure 1] Cross-sectional view of the stereo imaging lens in Example 1. [Figure 2] A block diagram showing the configuration of the imaging device in Example 1. [Figure 3] A flowchart showing the imaging process in Example 1. [Figure 4] A figure showing Example 1 of the captured image in Example 1. [Figure 5] This figure shows an example of an image captured in Example 1, specifically example 2. [Figure 6] This figure shows Example 3 of the captured image in Example 1. [Figure 7]A diagram showing Example 4 of the captured image in Example 1. [Figure 8] A diagram showing Example 5 of the captured image in Example 1. [Figure 9] A diagram showing Example 6 of the captured image in Example 1. [Figure 10] A diagram showing the projection method in Example 1. [Figure 11] A diagram showing an example of the converted image in Example 1. [Figure 12] A flowchart showing the image conversion process in Example 1. [Figure 13] A flowchart showing the output image generation process in Example 2. [Figure 14] A flowchart showing the image conversion process in Example 3. [Figure 15] A flowchart showing the imaging process in Example 4. [Figure 16] A diagram showing an example of the converted image in Example 4. [Figure 17] A flowchart showing the imaging process in Example 5.
Best Mode for Carrying Out the Invention
[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings.
Embodiment
[0011] FIG. 1 shows a cross-section of a stereo imaging lens (hereinafter simply referred to as an imaging lens) 200 that is detachably (exchangeably) attached to a camera body, which is the main body of a lens exchangeable imaging device according to Example 1 of the present invention. In FIG. 1, the left side is the subject side (front side), and the right side is the image side (rear side). Note that the imaging lens 200 may be provided in a lens-integrated imaging device.
[0012] The imaging lens 200 includes a right-eye optical system 201R as the first optical system and a left-eye optical system 201L as the second optical system. The right-eye optical system 201R and the left-eye optical system 201L are circumferential fisheye lenses having an angle of view of 180° or more, and are arranged in parallel in the left-right direction (the vertical direction in the plane of FIG. 1). Note that the first optical system and the second optical system may be arranged in parallel in the up-down direction (the direction perpendicular to the plane of FIG. 1). In the following description, the components of the right-eye optical system 201R are suffixed with R at the end of the reference numeral, and the components of the left-eye optical system 201L are suffixed with L at the end of the reference numeral.
[0013] The right-eye and left-eye optical systems 201R and 201L each have, in order from the object side (subject side) to the image side, a first lens group 211R, 211L, a second lens group 221R, 221L, and a third lens group 231R, 231L. Each lens group is composed of one or more lenses.
[0014] The first lens groups 211R and 211L each have a first optical axis OA;R and OA1L. The first optical axes OA1R and OA1L are separated from each other by a distance (optical axis interval) L1 in the left-right direction. The distance L1 is also referred to as the baseline length. The first lens groups 211R and 211L each have a convex front lens surface 211A on the subject side thereof, whereby the right-eye and left-eye optical systems 201R and 201L have an angle of view of 180° or more.
[0015] The second lens groups 221R and 221L each have a second optical axis OA2R and OA2L that extend perpendicular to the first optical axes OA1R and OA1L in the left-right direction.
[0016] The third lens groups 231R and 231L each have third optical axes OA3R and OA3L that extend perpendicular to the second optical axes OA2R and OA2L (parallel to the first optical axes OA1R and OA1L). Each third lens group consists of a front lens 231a and a rear lens 231b arranged sequentially from the object side to the image side. The third optical axes OA3R and OA3L are separated from each other by a distance L2 shorter than the baseline length L1 (narrow optical axis spacing) in the left-right direction. In the following description, the direction in which the first optical axis OA1(R, L) and the third optical axis OA3(R, L) extend is called the optical axis direction.
[0017] Between the first lens group 211R, 211L and the second lens group 221R, 221L, there are first prisms 220R, 220L, respectively, which are reflective elements that bend the optical path of light that has passed through the first lens group 211R, 211L toward the second lens group 221R, 221L. Also, between the second lens group 221R, 221L and the third lens group 231R, 231L, there are second prisms 230R, 230L, which are reflective elements that bend the optical path of light that has passed through the second lens group 221R, 221L toward the third lens group 231R, 231L.
[0018] The right and left eye optical systems 201R and 201L are held by lens holding members 212R and 212L, respectively, and housed within the outer cover member 203, and are screw-fastened to the lens top base 300. The lens top base 300 is screw-fastened to the lens bottom base 301, which is located within the outer cover member 203. The lens bottom base 301 is held so as to be able to move in the optical axis direction while its rotation is restricted by a linear guide provided on the outer cover member 203. As a result, the right and left eye optical systems 201R and 201L can move in the optical axis direction together with the lens top base 300 and the lens bottom base 301 to adjust the focus. A lens mount 202 is screw-fastened to the rear end of the outer cover member 203.
[0019] Furthermore, the front exterior member 204 is fixed to the front end of the exterior cover member 203 by screws or adhesive. The front exterior member 204 has two openings that expose the first lens AR of 201L and the front lens surface 211A of 211L, respectively, for the right and left eye optical systems 201R and 201L.
[0020] The imaging lens 200 having the configuration described above enables imaging to acquire stereo images (right-eye and left-eye images as disparity images with parallax) with a field of view of 180° or more. The stereo images are observed by the observer as VR images through VR goggles or the like.
[0021] Figure 2 shows the internal configuration of a stereo imaging device (hereinafter also simply referred to as an imaging device) 100, which consists of an imaging lens 200 and a camera body 110 to which it is mounted. The lens mount 202 of the imaging lens 200 is coupled to the camera mount 122 of the camera 110, thereby connecting the imaging lens 200 and the camera body 110 mechanically, electrically, and in a communicative manner.
[0022] The imaging lens 200 forms two subject images (image circles) in the left and right regions of the imaging surface of the image sensor 111 in the camera body 110, using the imaging optical system, which consists of the right eye optical system 201R and the left eye optical system 201L. The imaging lens 200 has a temperature detection unit 307 and a focus detection unit 308. The focus detection unit 308 is composed of a position sensor such as a magnetic, optical, or resistive type, and detects the focus position (the position of the optical systems 201R and 201L in the optical axis direction). The lens control unit 303 transmits information indicating the temperature and focus position detected by these detection units 307 and 308 to the camera control unit 117 in the camera body 110 in response to a request from the camera control unit 117.
[0023] The imaging lens 200 also has a memory unit 304. The memory unit 304 is composed of a memory device such as ROM or RAM and stores lens individual information 305 and lens manufacturing error information 306. The lens control unit 303 transmits the lens individual information 305 and lens manufacturing error information 306 to the camera control unit 117 in response to a request from the camera control unit 117. The lens individual information 305 and lens manufacturing error information 306 are information relating to the imaging lens 200 of the imaging device, and their details will be described later. In the following description, the lens individual information 305 and lens manufacturing error information 306 will also be collectively referred to as lens individual identification information.
[0024] In the camera body 110, the image sensor 111 is composed of photoelectric conversion elements such as a CCD sensor or a CMOS sensor, and performs photoelectric conversion (imaging) of the subject image formed on its imaging surface. The camera control unit 117 is composed of a computer such as a CPU and controls the entire imaging device 100.
[0025] The camera body 110 further includes an A / D converter 112, an image processing unit 113, a display unit 114, an operation unit 115, a recording unit 116, a storage unit 118, and an attitude detection unit 123. The A / D converter 112 converts the analog imaging signal output from the image sensor 111 into a digital imaging signal. The image processing unit 113 is composed of a computer such as a CPU and performs various image processing on the digital imaging signal to generate image data (captured image). The captured image is an image that includes the area in which the captured subject (main subject and background) is captured, that is, the subject image. In addition, the image processing unit 113 performs image conversion processing on the captured image acquired using the fisheye lens imaging lens 200 as image processing to enable the user to see in stereoscopic vision.
[0026] The display unit 114 is composed of a liquid crystal panel or an organic EL panel and displays images and various information. The operation unit 115 is operated by the user to input instructions to the imaging device 100. If the display unit 114 has a touch sensor, the touch sensor is also included in the operation unit 115. The recording unit 116 records various data, such as image data generated by the image processing unit 113, to a recording medium (such as flash memory or a hard disk) or to memory in the cloud.
[0027] The attitude detection unit 123 is composed of an acceleration sensor, a gyroscope, etc., and detects the attitude of the camera body 110 at the start of imaging and changes in attitude during imaging (such as camera shake), and sends this information to the camera control unit 117.
[0028] The memory unit 118 is composed of a memory device such as ROM or RAM and stores camera individual information 125 and camera manufacturing error information 126. The camera individual information 125 and camera manufacturing error information 126 are information relating to the camera body 110 of the imaging device, and their details will be described later. In the following description, the camera individual information 125 and camera manufacturing error information 126 will be collectively referred to as camera individual identification information.
[0029] The camera control unit 117 reads camera identification information from the storage unit 118 when imaging occurs, and generates an information image that includes the camera identification information, lens identification information received from the lens control unit 303, temperature and focus position information, and orientation information of the camera body 110. The information image includes a one-dimensional barcode, a two-dimensional barcode, a sequence of numbers, and brightness information corresponding to the number of recording bits of the captured image. Brightness information corresponding to the number of recording bits is, for example, as follows: When recording the information image, if the recording image is 8 bits, it is possible to record 256 levels of brightness information for each pixel of the image, and this 256 levels of information are represented in the image area allocated to the information image. As a result, if the image is read later, it is possible to obtain 8 bits of information for each pixel.
[0030] The generated information image is then sent to the image processing unit 113. The image processing unit 113 generates a recording image by adding (combining) the information image to the captured image, and records this on a recording medium via the recording unit 116. The camera control unit 117 corresponds to the acquisition means and generation means, and the image processing unit 113 corresponds to the addition means and processing means.
[0031] Furthermore, the memory unit 118 also stores programs for the camera control unit 117 and the image processing unit 113 to perform control and processing.
[0032] The flowchart in Figure 3 shows the processes (image processing methods) that the camera control unit 117 performs from the start to the end of image capture. The camera control unit 117 performs these processes according to the program.
[0033] When the user initiates the image capture start operation on the camera body 110, the camera control unit 117 acquires temperature and focus position information from the lens control unit 303 in step S101.
[0034] The reason for acquiring temperature information is as follows: The lens manufacturing error information 306 that the camera control unit 117 acquires from the lens control unit 303 in the next step is the manufacturing error at temperature during the manufacturing process of the imaging lens 200, which is written to the storage unit 304. Since the magnitude of this manufacturing error (error amount) can change depending on the temperature of the operating environment of the imaging system 100, knowing the exact error amount at temperature in the operating environment allows for appropriate image conversion processing of the captured image.
[0035] Furthermore, information on the focus position is acquired for the following reasons: The fisheye lens of the imaging lens 200 has a configuration close to pan-focus, and is generally in focus from close-up subjects to distant subjects. However, if you want to precisely focus on a subject at a specific distance, or if you set an aperture value with a small F-number, or even due to manufacturing tolerances of the camera body 110 (such as mounting errors of the image sensor 111), focus shifts may occur. For this reason, focus adjustment with the imaging lens 200 is necessary. By acquiring information on the focus position, it is possible to obtain information on the distance of the subject that the user wants to photograph, as well as optical information of the right and left eye optical systems 201R and 201L according to the focus position. For example, aberrations such as distortion of the right and left eye optical systems 201R and 201L may change depending on the focus position. For this reason, by obtaining information on aberrations according to the focus position, image distortion can be appropriately corrected in the image conversion processing of the captured image.
[0036] Next, in step S102, the camera control unit 117 retrieves the lens individual identification information (lens individual information 305 and lens manufacturing error information 306) stored in the memory 304 via the lens control unit 303. The lens individual information 205 includes optical design information (optical design information) of the imaging lens 200. For example, the baseline length L1 of the right eye and left eye optical systems 201R and 201L is set to a distance close to the interpupillary distance of a human eye so that appropriate parallax is obtained in stereoscopic vision by the user. The baseline length L1 corresponds to the distance between the principal points of the right eye and left eye optical systems 201R and 201L. For this reason, information on the distance between principal points is included in the lens individual information.
[0037] Furthermore, the right and left eye optical systems 201R and 201L bend the optical path using the first prisms 220R and 220L and the second prisms 230R and 230L. As a result, the distance between the centers of the left and right image circles formed on the image sensor 111 by the right and left eye optical systems 201R and 201L (the distance L2 between the third optical axes OA3R and OA3L) becomes shorter than the baseline length L1. Assuming a typical full-frame image sensor has a width of about 36 mm, it is necessary to form two image circles within it with a center-to-center distance of about 18 mm or less. Since this center-to-center distance of the image circles differs significantly from the interpupillary distance of humans (average of about 65 mm), it is necessary to know the design value of the distance L2 when performing image conversion processing. For this reason, the information on the distance L2 is also included in the individual lens information.
[0038] Furthermore, the actual right and left eye optical systems 201R and 201L include manufacturing tolerances relative to the design values. Therefore, individual lens manufacturing tolerance information 306 obtained during the manufacturing process of the imaging lens 200 must also be used in the image conversion process. Assembly errors and component tolerances during the manufacturing process of the imaging lens 200 prevent perfect parallelism of the optical axes of the right and left eye optical systems 201R and 201L, resulting in some degree of deviation from parallelism. This causes the center positions and intercenter distances of the left and right image circles on the image sensor 111 to deviate from the design values. Additionally, manufacturing tolerances in the spacing between lenses and prisms constituting each optical system cause changes in focal length and distortion from the design values. Therefore, understanding these manufacturing tolerances during the image conversion process enables appropriate image conversion. Furthermore, understanding the changes in manufacturing tolerances due to temperature, as mentioned above, allows for appropriate image conversion regardless of temperature.
[0039] Next, in step S103, the camera control unit 117 checks whether the imaging mode set by the user in the camera body 110 is still image imaging mode or video imaging mode. The following describes the case where video imaging mode is set.
[0040] Next, in step S104, the camera control unit 117 obtains camera individual identification information (camera individual information 125 and camera manufacturing error information 126) from the memory unit 118. The camera individual information includes information such as the model of the camera body 110, the physical size of the image sensor 111, the imaging area size, the number of pixels, and the pixel pitch. At this time, the camera control unit 117 obtains the imaging area size and the number of pixels of the image sensor 111 according to the imaging mode confirmed in step S103. This is because the imaging area size and the number of pixels may differ between still image capture and video capture.
[0041] Furthermore, camera manufacturing error information is acquired as individual information for each camera body 111 during the manufacturing process of the camera body 110 and written to the memory unit 118. The camera manufacturing error information includes information indicating the mounting error of the image sensor 111 in the camera body 110 (positional misalignment or tilt of the image sensor 111 relative to the camera mount 122), as well as information such as the color and brightness of the image sensor 111.
[0042] Next, in step S105, the camera control unit 117 acquires the posture information of the camera body 110 at the start of imaging from the posture detection unit 123. This posture information is also used to perform appropriate image conversion processing.
[0043] Next, in step S106, the camera control unit 117 processes the lens individual identification information, camera individual identification information, temperature information, focus position information, and attitude information acquired up to this point into an image to generate an initial information image (first information image).
[0044] Furthermore, in step S107, the camera control unit 117 sends the generated initial information image to the image processing unit 113, which generated the first frame image as the captured image, and causes the image processing unit 113 to add the initial information image to the first frame image to generate a first frame image for recording. Then, the first frame image for recording is recorded on the recording medium through the recording unit 118.
[0045] Next, in step S108, the camera control unit 117 acquires information about the camera body 110's posture during video capture through the posture detection unit 123. Changes in posture during video capture occur, for example, when the user holds the camera body 110 in their hand while taking images.
[0046] Next, in step S109, if the camera control unit 117 detects a change in the orientation of the camera body 110 from the first frame, it generates an in-imaging information image (second information image) with updated orientation information relative to the initial information image. After this, the camera control unit 117 continues to generate in-imaging information images with updated orientation information for each unit frame or whenever there is a change in orientation.
[0047] Then, in step S110, the camera control unit 117 generates a subsequent frame image for recording by adding the updated imaging information image to the frame image corresponding to the updated imaging information image among the subsequent frame images that are sequentially generated after the first frame image.
[0048] Next, we will explain how to add information images to captured images. Figure 4 shows an captured image 401 generated by capturing a subject (streetscape) with the imaging device 100 of this embodiment. As mentioned above, the right eye and left eye optical systems 201R and 201L are circular fisheye lenses, and the captured image 401 is an image in which circular fisheye images 402 and 403, which serve as subject images, are arranged side by side.
[0049] Furthermore, since the subject image formed on the image sensor 111 by each optical system is an inverted image, the image processing unit 113 performs an inversion process on the captured image. In the inversion process, the image is inverted point-symmetrically with respect to the center of the captured image 401 (image sensor 111). Therefore, when using the imaging lens 200 of this embodiment, an image is generated in which the circular fisheye image 402 corresponding to the right eye optical system 201R is located on the left, and the circular fisheye image 403 corresponding to the left eye optical system 201L is located on the right (i.e., left and right are swapped). However, since an image transformation process is performed on the captured image afterward, the swapping of the circular fisheye images 402 and 403 does not pose a problem.
[0050] On the other hand, in the captured image 401 obtained using the rectangular image sensor 111, there are regions around the left and right circular fisheye images 402 and 403 where the subject image (circular fisheye images 402 and 403) does not exist. In the following explanation, the region of the captured image 401 where the subject image exists will be referred to as the first region 404, and the region where the subject image does not exist (i.e., outside the first region) will be referred to as the second region 405.
[0051] Figure 5 shows an example in which a two-dimensional barcode, as described above, is added (combined) as an information image (initial information image and information image during acquisition) 501 to the second region 405 of the captured image 401. Since the second region 405 is a region that is not referenced in the image conversion process, adding information other than the subject image to it does not affect the image conversion process.
[0052] As mentioned above, the information image includes an initial information image containing information that does not change during imaging, and an in-imaging information image containing information that can change moment by moment during imaging, such as posture. Figure 6 shows an example in which the initial information image 502 and the in-imaging information image 503 are added to two locations in the second region 405.
[0053] Furthermore, as mentioned above, the information image may also be a sequence of numbers. Figure 7 shows an example in which the initial information image 504 as a sequence of numbers and the information image 505 being captured are added to two locations in the second region 405.
[0054] Furthermore, Figure 8 shows an example in which the initial information image 506 and the imaging information image 507 are added to the first region 404, rather than the second region 405. As shown in Figure 1, the right and left eye optical systems 201R and 201L have a field of view of 180° or more with the front lens surface 211A being convex. Therefore, a part of the left eye optical system 201L is reflected in the leftmost region 406R of the circular fisheye image 402 corresponding to the right eye optical system 201R, and a part of the right eye optical system 201R is reflected in the rightmost region 406L of the circular fisheye image 403 corresponding to the left eye optical system 201L. In this way, the regions 406R and 406L of the first region 404 in which optical systems that are not the subject are reflected are not referenced in the image conversion process, so information images 506 and 507 may be added to them.
[0055] Figure 9(a) shows an example in which initial information images 508R and 508L are added to the first frame image, which does not contain the subject image (only the second region) among the frame images sequentially generated during video capture. Here, the initial information images 508R and 508L are the initial information images corresponding to the right eye and left eye optical systems 201R and 201L, respectively. Subsequent frame images from the second frame onward include the first and second regions, and the second region has an imaging in progress information image 509 corresponding to each frame added to it. By adding the initial information images 508R and 508L to the first frame image, the initial information images are recorded even if the camera body 110's power is cut off during imaging, making it impossible to continue imaging. Therefore, the information contained in the initial information images can be used in editing the captured video up to a certain point. Also, since only the imaging in progress information images are added to subsequent frame images without adding the initial information images, the range of information images added to subsequent frame images can be minimized.
[0056] Figure 9(b) shows an example in which intermediate frame images up to the final frame image include first and second regions, with the second region having an imaging information image 509 corresponding to each frame added, and the final frame image, which does not contain the subject image, has an initial information image 508 and an imaging information image 509 added. In this example, if imaging becomes impossible to continue during imaging, the initial information image cannot be recorded, but because no information image appears at the beginning of the captured video, it is easier to identify and classify the captured video, such as what was captured in the video, during subsequent video editing. In this example as well, the range of information images added to intermediate frame images can be minimized.
[0057] Next, the image transformation process for the captured image will be described. The captured image subject to the image transformation process includes a circular fisheye image as the subject image and an information image. In this embodiment, the equidistant projection method is used as the projection method for the fisheye lens. In the equidistant projection method, as shown in Figure 10, the angle θ1 at which the fisheye lens 601 looks at the subject 603 (the angle of the fisheye lens 601 with respect to the optical axis 602) is approximately proportional to the distance r1 from the center of the imaging plane 604 (optical axis 602).
[0058] The image processing unit 113 generates a transformed image by performing a mapping based on equirectangular projection on the captured image generated by imaging using the equidistant projection method, as shown in Figure 11. Specifically, it plots the pixels for each azimuth angle in the captured image at positions proportional to the azimuth angle in the X direction of the transformed image, and plots the pixels for each elevation angle (for example, θ1 in Figure 10) at positions proportional to the angle of attack in the Y direction of the transformed image.
[0059] VR images displayed on VR viewing devices such as head-mounted displays are generally created using equirectangular projection. The VR viewing device extracts the area to be displayed to the user (display area) from the VR image as a converted image, and converts the image of this display area into a perspective projection image. The display area is selected according to the direction the user wearing the VR viewing device is facing. This makes it possible for the user to see images similar to the outside world they would observe in the real world.
[0060] However, since the right and left eye optical systems 201R and 201L, and the image sensor 111 each have manufacturing tolerances relative to their design specifications, if the captured image is converted using equirectangular projection without considering these manufacturing tolerances, shifts and distortions will occur in the left and right images after conversion. For example, a vertically standing building will be converted into a distorted image.
[0061] Furthermore, if there is a vertical misalignment between the left and right images, the images observed by the user's left and right eyes will not fuse into a good stereoscopic image, causing so-called VR sickness. Also, if the left and right images are misaligned horizontally, the parallax when the user views the stereoscopic image with their left and right eyes will not be accurately represented, resulting in discrepancies between the perceived size and distance of the subject and what is actually seen with the eyes, which leads to user discomfort. For this reason, by performing appropriate corrections using the aforementioned lens and camera manufacturing error information during image conversion processing, it is possible to generate VR images that are less likely to cause VR sickness or discomfort.
[0062] The flowchart in Figure 12 shows the image conversion process from captured images to VR images performed by the image processing unit 113. The image processing unit 113 performs this process according to the image conversion application (program).
[0063] In step S701, the image processing unit 113 reads the captured image generated from the digital imaging signal as the target for conversion.
[0064] Next, in step S702, the image processing unit 113 reads the information images (initial information image and information image during acquisition) that are attached to the captured image.
[0065] Next, in step S703, the image processing unit 113 acquires lens individual identification information, camera individual identification information, temperature information, focus position information, and orientation information from the information image. It also acquires location information where the video was captured from a GPS device connected to the camera. The image processing unit 113 then uses the acquired information to determine the shift in the center position of the left and right image circles, the distortion of the left and right subject images due to aberrations, and the differences between the left and right subject images due to changes in orientation. Furthermore, the image processing unit 113 generates transformed coordinate information in order to perform image transformation using equirectangular projection while correcting the above-mentioned shift in the center position and image distortion. Specifically, it calculates the coordinates after transforming the coordinates of all pixels in the subject image (circumferential fisheye image) using equirectangular projection and generates transformed coordinate information.
[0066] Next, in step S704, the image processing unit 113 converts the subject image into an equirectangular projection image based on the conversion coordinate information to generate a converted image (VR image). In video capture, this image conversion is performed for each frame image.
[0067] Next, in step S705, the image processing unit 113 outputs the VR image to be recorded on a recording medium via the recording unit 116, or to be displayed on a VR viewing device.
[0068] According to this embodiment, by including the information necessary for image conversion processing in the information image attached to the captured image, image conversion processing to a VR image can be performed using the information image even if the metadata of the captured image is lost. In other words, the information used for processing the captured image can be stored stably. [Examples]
[0069] In Example 1, the image processing unit 113 was described as performing image conversion processing on an captured image using an information image, but other image processing methods may also be used.
[0070] The flowchart in Figure 13 shows the output image generation process as an image processing operation performed by the image processing unit 113 in Embodiment 2. Steps S801 and S802 are the same as steps S701 and S702 in Figure 12.
[0071] In step S803, the image processing unit 113 generates transformed coordinate information as mesh data from the information image read in step S802. This transformed coordinate information is the coordinate information (mesh data) of the intersection points of the mesh when the area with a field of view of approximately 180° in this embodiment is considered as a hemispherical dome and the dome is divided into meshes of 20 vertical x 20 horizontal lines, 40 vertical x 40 horizontal lines, 80 vertical x 80 horizontal lines, etc.
[0072] Next, in step S804, the image processing unit 113 adds the mesh data generated in step S803 to the captured image read in step S801 as metadata. The captured image with the mesh data added as metadata is then displayed as a VR image by a VR viewing device that supports this format. [Examples]
[0073] The flowchart in Figure 14 shows the image transformation process performed by the image processing unit 113 in Embodiment 3, which generates transformation coordinate information in a time series. Step S901 is the same as step S701 in Figure 12.
[0074] In step S902, the image processing unit 113 reads the initial information image attached to the captured image. Subsequently, in step S903, the image processing unit 113 reads the in-capture information image attached to the captured image. For example, as shown in Figures 6 to 9, by attaching the initial information image and the in-capture information image to different positions in the captured image, it becomes possible to read the initial information image and the in-capture information image in chronological order from their respective positions.
[0075] Next, in step S904, the image processing unit 113 corrects for manufacturing errors and the like using the initial information image read in step S902, and generates time-series transformed coordinate information that corrects for camera shake and changes in posture during imaging using the imaging information image read in step S903.
[0076] Next, in step S905, the image processing unit 113 converts the subject image into an equirectangular projection image based on the time-series conversion coordinate information to generate a converted image. By using the time-series conversion coordinate information, it is possible to perform appropriate image conversion that reflects information that changes over time. In video capture, this image conversion is performed for each frame image.
[0077] The next step, S906, is the same as step S705 in Figure 12.
[0078] In this embodiment as well, as explained in Embodiment 2 (Figure 13), it is also possible to generate an image with mesh data added as metadata to the captured image without performing image conversion on the captured image. In this case, mesh data that changes in response to time-series changing information such as posture may be added, or data that adds time-series changing posture information to fixed mesh data corresponding to the right eye and left eye optical systems 201R and 201L, respectively, may be added. In the latter case, by appropriately updating the mesh data from the time-series changing posture information in the VR viewing device, it becomes possible to display a VR image that also responds to time-series changes in posture. [Examples]
[0079] The flowchart in Figure 15 shows the imaging process performed by the image processing unit 113 in Embodiment 4. In this embodiment, mesh data and an information image are added to the captured image during the imaging process.
[0080] In step S1001, the image processing unit 113 reads the design information and manufacturing error information of the imaging lens 200 and the camera body 110 from the lens control unit 303 (storage unit 304) and the storage unit 118 of the camera body 110.
[0081] Next, in step S1002, the image processing unit 113 generates mesh data from the design information and manufacturing error information read in step S1001.
[0082] In step S1003, the image processing unit 113 generates an information image containing design information and manufacturing error information using the camera control unit 117.
[0083] Next, in step S1004, the image processing unit 113 generates a recording image by adding mesh data and an information image to the captured image.
[0084] Then, in step S1005, the image processing unit 113 causes the recording image to be recorded onto the recording medium via the recording unit 116.
[0085] It is possible to view the captured images read from the camera body 110 as VR images on a VR viewing device, allowing for a simple check of the captured images. On the other hand, with captured images, especially videos, after the aforementioned simple check, editing may be performed, such as fine-tuning the image to color and brightness, or adding transition effects for titles and scene changes. In this case, metadata may be lost when the image is encoded. Metadata is especially often lost when the captured image is a RAW image, or when video is encoded after being treated as a still image frame by frame. When metadata is lost, manufacturing error information data is also lost, making it difficult to generate VR images afterward. Therefore, by recording information images simultaneously with the captured image, as in this embodiment, it is possible to avoid data loss of manufacturing error information during adjustments and editing, and to generate VR images.
[0086] Furthermore, information images generated within a video may be compressed using various codecs during adjustment and editing. In this process, for example, a 2D barcode as an information image may also be compressed, potentially rendering the barcode unreadable. To prevent this, as shown in Figure 7, the information image can be constructed using a sequence of numbers, allowing the information to be read even with some compression. Alternatively, areas containing information images can be left uncompressed, or only the information image can be extracted and saved before encoding. These measures make it possible to generate VR images even if metadata is lost. [Examples]
[0087] Example 5 describes how to handle the information image in the converted image. Figure 16 shows an example of the converted image 801 after it has been converted from a circular fisheye image to an equirectangular projection image. The information image is used when converting from the circular fisheye image to the converted image. For this reason, the subject image may be recorded again in the region 802 corresponding to the reflection regions 406R and 406L of the optical system shown in Figure 8 in the converted image 801.
[0088] Furthermore, since region 802 is an area unnecessary for displaying the VR image, the information image 803 may be stored here. By leaving the information image 802 within the converted image 801, it becomes possible to use this information image 803 when, for example, processing the converted image 801 back into a circular fisheye image.
[0089] Furthermore, by including address information of a server (external storage device) on the Web or cloud in an information image using a 2D barcode, it becomes possible to access that server. The flowchart in Figure 17 shows an example in which the camera body 110 (camera control unit 117 and image processing unit 113), which has generated an image with an information image as shown in Figure 5, obtains information used for image conversion processing from a server accessed via the information image and then performs image conversion processing.
[0090] In step S1101, the camera control unit 117, which has started imaging, obtains the individual identification numbers (serial numbers, etc.) of the camera body 110 and the imaging lens 200.
[0091] Next, in step S1102, the camera control unit 117 generates an information image that includes the acquired individual number and server address information.
[0092] Then, in step S1103, the image processing unit 113 adds the generated information image to the captured image to create a recording image, which is then recorded on the recording medium via the recording unit 116. The information image added to the captured image is basically the initial information image, but it may also include attitude information and other information that changes over time.
[0093] Subsequently, the image processing unit 113 obtains information from the accessed server using the address information of the information image attached to the recorded captured image, and performs image conversion processing using this information.
[0094] Specifically, in step S1104, the image processing unit 113 first reads the captured image with an information image containing address information.
[0095] Next, in step S1105, the image processing unit 113 accesses a server corresponding to the address information contained in the information image via the camera control unit 117, and obtains design information and manufacturing error information corresponding to the individual numbers of the camera body 110 and the imaging lens 200 from the server. The server obtains individual design information and manufacturing error information for each camera body 110 and imaging lens 200 during the manufacturing process (step S1201), and stores this information associated with their respective individual numbers (step S1202).
[0096] Next, in step S1106, the image processing unit 113 generates transformed coordinate information based on the design information and manufacturing error information of the camera body 110 and imaging lens 200 acquired from the server. This coordinate transformation information is used to convert the circular fisheye image in the captured image into an equirectangular projection image while appropriately correcting it according to the design information and manufacturing error information.
[0097] Subsequently, in steps S1107 and S1108, the image processing unit 113 performs the transformation of the transformed coordinate information image and outputs the transformed image, similar to steps S704 and S705 in Figure 12.
[0098] According to this embodiment, it is possible to generate good VR images using design information and manufacturing error information stored on the server. Furthermore, for example, it may be possible to access an image distribution platform on the cloud from the information image, and have the platform perform image conversion of the captured image to a VR image and distribution of the VR image without any time lag.
[0099] In the above embodiments, image processing was described for cases where a two-lens stereo imaging lens is used for imaging. However, similar image processing may also be performed when using an imaging lens other than a stereo imaging lens, which produces an image with a region outside the subject image where the subject image does not exist, such as a monocular fisheye lens. (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.
[0100] The embodiments described above are merely representative examples, and various modifications and changes can be made to each embodiment when implementing the present invention. [Explanation of Symbols]
[0101] 110 Camera body 113 Image Processing Unit 117 Camera Control Unit 125 Camera Individual Information 126 Camera Manufacturing Error Information 200 Stereo Imaging Lens 305 Lens Individual Information 306 Lens Manufacturing Error Information 401 Acquired Images 501-509 Information Images
Claims
1. An image processing device that processes a captured image generated by imaging, An acquisition means for acquiring information regarding an imaging device used for the imaging; a generating means for generating an information image as an image including information about the imaging device; an information image processing device comprising: an adding device for adding the information image to the captured image and recording the image;
2. 2. The image processing apparatus according to claim 1, wherein the information image is a bar code, a numeric string, or luminance information according to a recording bit number of the captured image.
3. 3. The image processing apparatus according to claim 1, wherein the information about the imaging device includes at least one of design information of the imaging device, information about an imaging element, and manufacturing error information.
4. 4. The image processing device according to claim 1, wherein the information relating to the imaging device includes address information for acquiring at least one of design information of the imaging device, information on the imaging element, and manufacturing error information from an external storage device via communication.
5. 5. The image processing apparatus according to claim 1, wherein the information about the image capturing apparatus includes at least one of temperature information, focus position information, and attitude information about the image capturing apparatus.
6. The image processing device according to any one of claims 1 to 5, characterized in that the addition means adds to the captured image, as the information image, a first information image including information about the imaging device at the start of the imaging, and a second information image including information about the imaging device that has changed after the imaging has started.
7. the captured image has a first region in which a subject image is present and a second region in which the subject image is not present, 7. The image processing apparatus according to claim 1, wherein the adding means adds the information image to the second area.
8. When the captured image is a frame image of a video, The generating means updates the information image for each unit frame or each time there is a change in information about the imaging device, 8. The image processing apparatus according to claim 1, wherein the adding means adds the updated information image to a corresponding frame image.
9. the imaging device includes an imaging lens and a camera body having an imaging element and to which the imaging lens is detachably attached; 9. The image processing apparatus according to claim 1, wherein the acquisition unit acquires information about the imaging device from each of the imaging lens and the camera body.
10. The image processing device according to claim 1 , wherein the imaging device has a circular fisheye lens as an imaging lens.
11. 11. The image processing device according to claim 1, further comprising a processing means for acquiring information about the imaging device through the information image added to the recorded captured image, and performing image processing on the captured image using the information.
12. 12. The image processing device according to claim 1, wherein the processing means performs the image processing on the subject image without performing the image processing on an area of the captured image to which the information image is added.
13. The imaging lens of the imaging device has two optical systems arranged in parallel with each other, 13. The image processing apparatus according to claim 11, wherein the processing means performs the image processing on subject images obtained through the two optical systems, among the captured images.
14. An image processing device according to any one of claims 1 to 13, and an imaging element for capturing an object image formed by an imaging lens.
15. An image processing method for processing a captured image generated by imaging, comprising the steps of: obtaining information about an imaging device used for the imaging; generating an information image as an image including information about the imaging device; and adding the information image to the captured image and recording the information image.
16. A program for causing a computer to execute a process according to the image processing method of claim 15.
