Imaging apparatus

The imaging device addresses parallax errors by associating lens unit state and environmental temperature data with captured images, enabling efficient and precise parallax correction for improved stereoscopic viewing.

JP2024162501APending Publication Date: 2024-11-21CANON KK
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023078054
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-10
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

Existing imaging devices struggle with high parallax errors between multiple image regions, requiring lengthy parameter determination for accurate parallax correction, which is inefficient and inaccurate when parallax errors vary between images.

Method used

An imaging device that associates information about the lens unit's mounting state and environmental temperature with captured images, allowing for precise determination of parallax correction parameters and reducing the time needed for parameter updates.

Benefits of technology

Enables high-precision parallax correction processing with reduced determination time by using stored information to decide when to update parameters, ensuring accurate stereoscopic viewing without discomfort.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024162501000001_ABST
    Figure 2024162501000001_ABST
Patent Text Reader

Abstract

To provide a technique that can achieve both determination of a parameter allowing adjustment processing such as parallax correction processing (processing of adjusting at least one of the position and size of each image area) to be performed with high accuracy, and a reduction in the time required for determination of the parameter.SOLUTION: An imaging apparatus of the present invention has: imaging means that picks up images of a plurality of image areas through a plurality of optical systems; acquisition means that acquires determination information that is information related to at least one of a mounting state of a lens unit having the plurality of optical systems and the temperature around the lens unit, the information used for determination as to whether to update a parameter of processing of adjusting at least one of the position and size of each of the plurality of image areas; and control means that stores an image including the plurality of image areas and the determination information in a storage unit in association with each other.SELECTED DRAWING: Figure 5
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an imaging device, and more particularly to a technique for correcting parallax between a plurality of image regions. [Background technology]

[0002] An imaging device has been proposed that has one lens mount (and one imaging element) and can capture two image areas at once via two optical systems. For example, two image areas with parallax are captured at once for stereoscopic vision.

[0003] The disparity between two image regions may contain an error (parallax error). To achieve a natural stereoscopic view, it is necessary to reduce the disparity error by performing a parallax correction process that corrects the disparity between the two image regions. However, it takes a long time to determine (generate and update) the parameters for the parallax correction process (parallax correction parameters).

[0004] Patent Document 1 discloses a technique for performing image processing using the same parameters for each of a series of images (multiple images) obtained by continuous shooting, shooting with fixed shooting conditions for each frame (fixed exposure), or shooting with a shooting time interval between each frame equal to or less than a predetermined value. The technique disclosed in Patent Document 1 makes it possible to reduce the time required to determine parameters. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2000-98514 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, the parallax error may be (largely) different between a series of images (multiple images). Therefore, if the technology disclosed in Patent Document 1 is used to perform parallax correction processing using the same parallax correction parameters for each of the series of images, the parallax correction processing may not be performed with high accuracy for at least one of the series of images, and the parallax error may not be reduced sufficiently.

[0007] The present invention aims to provide a technology that can determine parameters that can perform adjustment processes such as parallax correction processes (adjustment processes of at least one of the position and size of each image area) with high accuracy, while reducing the time required to determine the parameters. [Means for solving the problem]

[0008] The imaging device of the present invention is characterized by having an imaging means for imaging multiple image areas through multiple optical systems, an acquisition means for acquiring judgment information which is information regarding at least one of the mounting state of a lens unit having the multiple optical systems and the ambient temperature of the lens unit, and which is information used to determine whether to update parameters of an adjustment process for at least one of the position and size of each of the multiple image areas, and a control means for storing an image including the multiple image areas and the judgment information in a memory unit in association with each other. Effect of the Invention

[0009] According to the present invention, it is possible to determine parameters that can perform adjustment processes such as parallax correction processes (adjustment processes of at least one of the position and size of each image area) with high accuracy while reducing the time required to determine the parameters. [Brief description of the drawings]

[0010] [Figure 1] FIG. [Diagram 2] FIG. 1 is a block diagram of an imaging device. [Diagram 3] FIG. 1 is a block diagram of a personal computer (PC). [Figure 4] 4 is a flowchart showing the operation of the imaging apparatus. [Diagram 5] 13 is a flowchart of a startup state process. [Figure 6] 1 is a flowchart showing the operation of a PC. [Figure 7] 13 is a flowchart of an update determination process. [Figure 8] FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0012] 1(a) to 1(c) are schematic diagrams showing captured images obtained by an imaging device equipped with a lens unit having two optical systems (for example, a twin-lens unit compatible with VR180). The captured images shown in FIGS. 1(a) to 1(c) include two image regions (two regions marked with "AB" in FIGS. 1(a) to 1(c)) captured through the two optical systems, respectively. The two optical systems are aligned in the left-right direction of the lens unit, and it is assumed that the two image regions are aligned in the left-right direction. The two image regions are regions for stereoscopic vision, and there is parallax between the two image regions. Each of the two optical systems includes a fisheye lens, and each of the two image regions is a fisheye image region (a region of a fisheye image (circular fisheye image), an image circle).

[0013] FIG. 1(a) shows an ideal captured image without parallax error. In the captured image of FIG. 1(a), two image areas of the same size are arranged in the left-right direction (horizontal direction), and a line 101 passing through the centers of the two image areas is parallel to the horizontal direction. FIG. 1(b) shows a captured image in which a parallax error occurs due to the mounting state of the lens unit to the imaging device. For example, when the lens unit is loosened relative to the imaging device (lens mount), the mounting state of the lens unit to the imaging device changes. In FIG. 1(b), the lens unit is mounted obliquely to the imaging device. Therefore, the two image areas are arranged obliquely, and a line 102 passing through the centers of the two image areas is inclined obliquely. FIG. 1(c) shows a captured image in which a parallax error occurs due to the temperature around the lens unit (the temperature around the imaging device, the environmental temperature). In FIG. 1(c), a deviation occurs between the characteristics of the left optical system and the characteristics of the right optical system due to the environmental temperature. Therefore, the sizes of the two image areas are different. In FIG. 1(c), a difference 103 (size difference) occurs between the radius of the left image region (image circle) and the radius of the right image region (image circle).

[0014] To achieve a natural stereoscopic view, it is necessary to perform a parallax correction process to correct the parallax between two image regions and reduce the parallax error, but it takes a long time to determine (generate and update) the parameters for the parallax correction process (parallax correction parameters).

[0015] In a situation such as that shown in FIG. 1(b), if the mounting state of the lens unit relative to the imaging device does not change, the generated parallax correction parameters can be reused. Also, in a situation such as that shown in FIG. 1(c), if the environmental temperature does not change, the generated parallax correction parameters can be reused. Based on this knowledge, in this embodiment, when a captured image is stored in the storage unit (when shooting), information regarding at least one of the mounting state and the environmental temperature is stored in the storage unit in association with the captured image. This information is used as judgment information for judging whether or not to update the parallax correction parameters. For example, when a captured image stored in the storage unit is edited or displayed, the judgment information associated with the captured image is used to determine whether or not the parallax correction parameters are updated. It is determined whether or not to update the difference correction parameters. By doing so, it is possible to determine parameters that can perform adjustment processing such as disparity correction processing (adjustment processing of at least one of the position and size of each image region) with high accuracy, while reducing the time required to determine the parameters.

[0016] The number of optical systems, the arrangement of the optical systems, the type of lens included in the optical systems, the number of image areas, the arrangement of the image areas, and the shape of the image areas are not particularly limited. For example, three or more image areas may be captured through three or more optical systems, respectively. The image areas may be equirectangular image areas.

[0017] FIG. 2 is a block diagram showing a configuration of an imaging device 200 as an example of an electronic device according to this embodiment. The imaging device 200 is, for example, a digital camera with a detachable lens unit. Note that the electronic device according to this embodiment is not limited to a digital camera, and may be, for example, a smartphone or a tablet terminal. The electronic device according to this embodiment is not limited to an imaging device, and may be, for example, a personal computer (PC) that receives captured images and information from an imaging device and stores them in a storage unit. In this way, various electronic devices can be used as the electronic device according to this embodiment.

[0018] The control unit 201 is, for example, a central processing unit (CPU) and controls the entire imaging device 200. The read only memory (ROM) 202 stores various data (for example, programs and parameters) that do not need to be changed. The random access memory (RAM) 203 temporarily stores various data (for example, programs and parameters).

[0019] The storage device 204 stores various data. For example, the storage device 204 is a solid state drive (SSD), a hard disk drive (HDD), or a flash storage that is fixed (built-in) to the imaging device 200. The storage device 204 may be an optical disk, a magnetic card, an optical card, an IC card, or a memory card that is detachable from the imaging device 200.

[0020] The operation unit 205 includes an operation member such as a button or a touch panel, and accepts an operation by a user (user operation). The operation unit 205 may be a receiving unit that receives, by wire or wireless, an operation signal corresponding to a user operation on an operation device such as a remote controller from the operation device. The display unit 206 displays an image based on various image data such as image data stored in advance in the imaging device 200, image data generated by the imaging device 200, and image data shared by the imaging device 200.

[0021] The imaging unit 207 is an image sensor such as a Charge Coupled Device (CCD). When a lens unit having two optical systems (for example, a twin lens unit compatible with VR180) is attached to the imaging device 200, the imaging unit 207 captures two image areas via the two optical systems. As a result, captured images such as those shown in FIGS. 1(a) to 1(c) are obtained. The sensor unit 208 includes various sensors such as a motion sensor, an environmental temperature sensor, and a detachment sensor (a sensor that detects attachment and detachment of the lens unit to and from the imaging device 200). A system bus 209 is connected to each unit of the imaging device 200 to enable communication within the imaging device 200.

[0022] FIG. 3 is a block diagram showing the configuration of a personal computer (PC) 300 as an example of an information processing device according to this embodiment. Note that the information processing device according to this embodiment is not limited to a personal computer, and may be, for example, a smartphone or a tablet terminal. The information processing device according to this embodiment may also be a display device. As the information processing device according to the embodiment, various information processing devices can be used.

[0023] The control unit 301 is, for example, a central processing unit (CPU) and controls the entire PC 300. The read only memory (ROM) 302 stores various data (for example, programs and parameters) that do not need to be changed. The random access memory (RAM) 303 temporarily stores various data (for example, programs and parameters).

[0024] The storage device 304 stores various data. For example, the storage device 304 is a solid state drive (SSD), a hard disk drive (HDD), or a flash storage that is fixed (built-in) to the PC 300. The storage device 304 may be an optical disk, a magnetic card, an optical card, an IC card, or a memory card that is removable from the PC 300.

[0025] The operation unit 305 includes an operation member such as a button or a touch panel, and accepts an operation by a user (user operation). The operation unit 305 may be a receiving unit that receives, from an operation device such as a remote controller, an operation signal corresponding to a user operation on the operation device, by wire or wirelessly. The display unit 306 displays an image based on various image data such as image data stored in advance in the PC 300, image data generated by the PC 300, and image data shared by the PC 300. The communication unit 307 is a communication interface for connecting to and communicating with an external device by wire or wirelessly. The system bus 308 is connected to each unit of the PC 300 to enable communication within the PC 300.

[0026] An overview of the operation of the imaging device 200 will be described. When the imaging device 200 is in a power-off state (standby state in which electricity is being applied), the imaging device 200 collects motion information and lens attachment / detachment information as information related to the attachment state of the lens unit relative to the imaging device 200. The motion information is information based on the detection result of a motion sensor, and the attachment / detachment information is information based on the detection result of a attachment / detachment sensor. Although the motion information and the attachment / detachment information are not particularly limited, in this embodiment, the motion information is information on the number of movements of the imaging device 200 that are greater than a threshold value. And the attachment / detachment information is information on the number of attachments and detachments of the lens unit relative to the imaging device 200. Even when the imaging device 200 is in a power-on state (start-up state), the imaging device 200 collects motion information and attachment / detachment information. When capturing an image, the imaging device 200 acquires information on the shooting date and time (shooting date and time information, current date and time information) and information on the environmental temperature detected by the environmental temperature sensor (environmental temperature information). Then, the imaging device 200 stores the captured image and the judgment information (photographing date and time information, attachment / detachment information, motion information, and environmental temperature information) in the storage device 204 in association with each other.

[0027] An overview of the operation of the PC 300 will be described. The PC 300 can sequentially perform a process of associating a captured image with a parallax correction parameter for a plurality of images. The process of associating a captured image with a parallax correction parameter is not particularly limited, but in this embodiment, the parallax correction parameter is associated with an image by performing a parallax correction process using the parallax correction parameter on the image. For the first captured image, the PC 300 generates a parallax correction parameter based on the first captured image. The PC 300 holds the generated parallax correction parameter. For the second and subsequent captured images, the PC 300 switches between using the held parallax correction parameter or updating (generating) the parallax correction parameter based on the captured image based on the above-mentioned determination information.

[0028] 4 is a flowchart showing the details of the operation of the imaging device 200. The operation of FIG. 4 is performed by the control unit 201 loading a program stored in the ROM 202 into the RAM 203 and executing it. For example, when a battery is inserted into the imaging device 200 and power is turned on in the imaging device 200 (the imaging device 200 goes into a standby state), the control unit 201 starts the operation shown in FIG.

[0029] In S401 , the control unit 201 drives the motion sensor and the attachment / detachment sensor included in the sensor unit 208 .

[0030] In S402, the control unit 201 determines whether or not a movement of the imaging device 200 greater than a threshold has occurred based on the output signal of the motion sensor. If a movement of the imaging device 200 has occurred, the process proceeds to S403, and if not, the process proceeds to S404. The threshold is, for example, the minimum magnitude of movement that may cause rattling of the lens unit relative to the imaging device 200 (lens mount section) (movement that may change the mounting state of the lens unit relative to the imaging device 200). There is no particular limitation on the method of determining the threshold, but the threshold is determined in advance based on information such as the size of the imaging device 200, the size of the lens unit, and the strength of the lens mount section, for example.

[0031] In S403, the control unit 201 adds 1 to a count value (motion count value, motion information) indicating the number of movements of the imaging device 200 that are greater than the threshold value. The initial value of the motion count value (the motion count value at the start of the operation in FIG. 4) is, for example, 0. The motion count value is held in, for example, the RAM 203.

[0032] In S404, the control unit 201 determines, based on the output signal of the attachment / detachment sensor, whether or not the lens unit has been attached or detached to the imaging device 200. If the lens unit has been attached or detached, the process proceeds to S405, and if not, the process proceeds to S406.

[0033] In S405, the control unit 201 adds 1 to a count value (detachment count value, detachment information) indicating the number of times the lens unit has been attached to or detached from the imaging device 200. The initial value of the detachment count value (the detachment count value at the start of the operation in FIG. 4) is, for example, 0. The detachment count value is held in, for example, the RAM 203.

[0034] In S406, the control unit 201 determines whether or not the user has instructed the imaging device 200 to start up (power on), for example, whether or not the user has pressed a power button included in the operation unit 205. If the user has instructed the imaging device 200 to start up, the process proceeds to S407, and if not, the process proceeds to S402.

[0035] In S407, the control unit 201 performs an activation state process, the details of which will be described later with reference to FIG.

[0036] In S408, the control unit 201 determines whether or not power supply has ended. Power supply ends when the battery is removed or the remaining battery charge becomes 0. If power supply has ended, the operation of FIG. 4 ends, and if not, the process proceeds to S402.

[0037] Fig. 5 is a flowchart of the startup state process (S407 in Fig. 4). The startup state process in Fig. 5 is realized by the control unit 201 loading a program stored in the ROM 202 into the RAM 203 and executing the program.

[0038] In S501, similarly to S402 in Fig. 4, the control unit 201 determines whether or not a motion of the imaging device 200 greater than a threshold has occurred based on the output signal of the motion sensor. If a motion of the imaging device 200 has occurred, the process proceeds to S502, and if not, the process proceeds to S503.

[0039] In S502, similarly to S403, the control unit 201 adds 1 to the motion count value. do.

[0040] In S503, similarly to S404, the control unit 201 determines, based on the output signal of the attachment / detachment sensor, whether or not the lens unit has been attached or detached to the imaging device 200. If the lens unit has been attached or detached, the process proceeds to S504, and if not, the process proceeds to S505.

[0041] In S504, similarly to S405, the control unit 201 adds 1 to the attachment / detachment count value.

[0042] In S505, the control unit 201 determines whether or not the user has instructed the start of shooting. For example, the control unit 201 determines whether or not the shutter button included in the operation unit 205 has been pressed, or whether or not the video shooting button included in the operation unit 205 has been pressed. If the start of shooting has been instructed, the process proceeds to S506, and if not, the process proceeds to S501.

[0043] In S506, the control unit 201 acquires information (lens information) about the lens unit from the lens unit attached to the imaging device 200. For example, the lens information is identification information capable of identifying at least the type of the lens unit, and indicates at least one of the type, model number, serial number, and name of the lens unit. The lens information may indicate various design values ​​of the lens unit, or may indicate various individual values ​​of the lens unit.

[0044] In S507, similarly to S501, the control unit 201 determines whether or not a motion of the imaging device 200 greater than the threshold has occurred based on the output signal of the motion sensor. If a motion of the imaging device 200 has occurred, the process proceeds to S508, and if not, the process proceeds to S509.

[0045] In S508, similarly to S502, the control unit 201 adds 1 to the motion count value.

[0046] In S509, the control unit 201 acquires a captured image from the imaging unit 207, and stores the captured image in the storage device 204 in a file format.

[0047] In S510, the control unit 201 acquires information on the environmental temperature (environmental temperature information) from the environmental temperature sensor included in the sensor unit 208. Then, the control unit 201 stores the acquired environmental temperature information in the storage device 204 as part of the metadata of the captured image stored in S509. The environmental temperature detected by the environmental temperature sensor is the temperature around the imaging device 200, and is the temperature around the lens unit attached to the imaging device 200. Note that the environmental temperature information may be acquired from the lens unit. The metadata may be stored in a file of the captured image, or may be stored in a file different from the file of the captured image.

[0048] In S511, the control unit 201 stores the lens information acquired in S506 in the storage device 204 as part of the metadata of the captured image stored in S509.

[0049] In S512, the control unit 201 stores the motion count value and the attachment / detachment count value held in the RAM 203 in the storage device 204 as part of the metadata of the captured image stored in S509.

[0050] In S513, the control unit 201 stores information on the shooting date and time (shooting date and time information, current date and time information, and date and time information when the captured image was stored in the storage device 204) as part of the metadata of the captured image stored in S509 in the storage device 204. The shooting date and time information can be acquired by various known methods.

[0051] In S514, the control unit 201 determines whether or not the user has instructed the camera to end shooting. For example, the control unit 201 determines whether or not the shutter button has been released, or whether or not the video shooting button has been pressed again. If the user has instructed the camera to end shooting, the process proceeds to S515, and if not, the process proceeds to S507. By repeating the processes of S507 to S514, the camera can shoot not only one still image, but also multiple frames such as continuous shooting, time lapse shooting, and video shooting.

[0052] In S515, the control unit 201 determines whether or not the user has instructed the imaging device 200 to be stopped (powered off), for example, whether or not the power button included in the operation unit 205 has been pressed. If the user has instructed the imaging device 200 to be stopped, the start-up state process in FIG. 5 is terminated, and if not, the process proceeds to S501.

[0053] FIG. 6 is a flowchart showing the details of the operation of the PC 300. The operation of FIG. 6 is realized by the control unit 301 expanding a program stored in the ROM 302 into the RAM 303 and executing it. Here, it is assumed that the storage device 204 is removed from the imaging device 200 and attached to the PC 300 as the storage device 304. When an image file (captured image obtained by the imaging device 200) stored in the storage device 304 is specified (selected) by the user, the control unit 301 starts the operation of FIG. 6. The user can specify multiple captured images at once, and when multiple captured images are specified, the control unit 301 performs the processes of S601 to S614 for each specified captured image. The specified multiple captured images are, for example, multiple frames obtained by one continuous shooting, time lapse shooting, or video shooting. The specification of a video file can be interpreted as the specification of multiple frames of a video. The specified multiple captured images may be multiple still images obtained by individual still image shooting.

[0054] In S601, the control unit 301 reads out a captured image designated by a user from the storage device 304 (image acquisition).

[0055] In S602, the control unit 301 reads (acquires information) the metadata of the captured image read in S601 from the storage device 304. The metadata includes the environmental temperature information, lens information, motion count value, attachment / detachment count value, and shooting date and time information stored in S510 to S513 of FIG.

[0056] In S603, the control unit 301 acquires shooting date and time information from the metadata read in S602.

[0057] In S604, the control unit 301 acquires lens information from the metadata read in S602.

[0058] In S605, the control unit 301 determines whether or not a predetermined lens unit was attached to the imaging device 200 when the captured image read out in S601 was captured, based on the lens information acquired in S604. The predetermined lens unit is, for example, a compound eye lens unit having multiple optical systems. Here, the control unit 301 determines whether or not a twin-lens unit (a lens unit having two fisheye lenses) compatible with VR180 was attached to the imaging device 200. If the predetermined lens unit was attached, the process proceeds to S606, and if not, the process proceeds to S615.

[0059] In S606, the control unit 301 acquires sensor information (ambient temperature information, motion count value, and attachment / detachment count value) from the metadata read in S602.

[0060] In S607, the control unit 301 determines whether or not the captured image read out in S601 is the first image. If it is the first image, the process proceeds to S608, and if not, the process proceeds to S611.

[0061] In S608, the control unit 301 generates a parallax correction parameter based on the captured image read in S601. Then, the control unit 301 performs a parallax correction process using the generated parallax correction parameter on the captured image read in S601. The parallax correction parameter can be generated by various known methods. Also, the parallax correction process can be performed by various known methods.

[0062] In S609, the control unit 301 stores in the RAM 303 the disparity correction parameters generated in S608 or the disparity correction parameters (updated disparity correction parameters) generated in S613 (described later).

[0063] In S610, the control unit 301 stores in the RAM 303 the shooting date and time information acquired in S603 and the sensor information acquired in S606 in association with the parallax correction parameters stored in S609.

[0064] In S611, the control unit 301 performs an update determination process (a process for determining whether or not to update the parallax correction parameter). When updating the parallax correction parameter, the reset flag is set to 1 (on), and when not updating (reusing) the parallax correction parameter, the reset flag is set to 0 (off). Details of the update determination process will be described later with reference to FIG. 7.

[0065] In S612, the control unit 301 determines whether or not the reset flag is set to 1 (ON). If the reset flag is set to 1, the process proceeds to S613, and if not, the process proceeds to S614.

[0066] In S613, the control unit 301 generates a parallax correction parameter based on the captured image read in S601. This updates the parallax correction parameter used in the parallax correction process. The control unit 301 then performs the parallax correction process using the generated parallax correction parameter on the captured image read in S601. Thereafter, the processes of S609 and S610 are performed, and the retained information (parallax correction parameter, shooting date and time information, and sensor information) is updated.

[0067] In S614, the control unit 301 applies the parallax correction process to the captured image read out in S601 by using the held parallax correction parameters.

[0068] In S615, the control unit 301 determines whether or not the processes of S601 to S614 have been performed for all the specified captured images. If the processes of S601 to S614 have been performed for all the captured images, the operation in FIG. 6 ends, and if not, the operation proceeds to S601.

[0069] Fig. 7 is a flowchart of the update determination process (S611 in Fig. 6). The update determination process in Fig. 6 is realized by the control unit 301 loading a program stored in the ROM 302 into the RAM 303 and executing it.

[0070] In the update determination process of Fig. 7, it is determined whether or not to update the parallax correction parameters using the determination information (photographing date and time information, attachment / detachment count value, motion count value, and environmental temperature information) shown in Fig. 8. When the parallax correction parameters are generated (updated), the generated (updated) parallax correction parameters are held in S609, and the determination information acquired in S603 and S606 is The result is stored in S610. After that, the processes of S603 and S606 are performed again to obtain new determination information. In the update determination process of Fig. 7, the obtained new determination information is compared with the stored determination information to determine whether or not to update the parallax correction parameter.

[0071] In S701, the control unit 301 sets a reset flag, which indicates whether or not to update the parallax correction parameter, to 0 (off).

[0072] In S702, the control unit 301 compares the captured date and time information held with the new captured date and time information. The captured date and time information held indicates the captured date and time of the captured image used for updating the held parallax correction parameter, and the new captured date and time information indicates the captured date and time of the captured image to be processed. The control unit 301 determines whether the time from the captured date and time indicated by the held captured date and time information to the captured date and time indicated by the new captured date and time information is longer than a threshold value. Assume that the threshold value is one hour, and the captured date and time indicated by the held captured date and time information is 3:00. In this case, the control unit 301 determines whether the captured date and time indicated by the new captured date and time information satisfies the condition "before 2:00 or after 4:00". If the time between the two captured dates and times is long, there is a high possibility that the mounting state (mounting state of the lens unit with respect to the imaging device 200) or the environmental temperature at the time of shooting is (largely) different between the two captured images, and there is a high possibility that the parallax error is (largely) different. Therefore, if the time between the two captured dates and times is longer than the threshold value, it is determined that the parallax correction parameter is to be updated, and the process proceeds to S706. If not, proceed to S703.

[0073] In S703, the control unit 301 compares the retained detachment count value with the new detachment count value. The retained detachment count value indicates the number of detachments (attachment and detachment of the lens unit to the imaging device 200) during the period until the captured image used for updating the retained parallax correction parameter is captured. The new detachment count value indicates the number of detachments during the period until the captured image to be processed is captured. The control unit 301 determines whether the retained detachment count value is different from the new detachment count value. If the two detachment count values ​​are different, the lens unit is detached from the imaging device 200 during the period from the capture of one of the two captured images to the capture of the other. The attachment state of the lens unit to the imaging device 200 may change due to the attachment and detachment of the lens unit to the imaging device 200. Therefore, if the two detachment count values ​​are different, there is a high possibility that the attachment state (attachment state of the lens unit to the imaging device 200) at the time of capture is (largely) different between the two captured images, and there is a high possibility that the parallax error is (largely) different. Therefore, if the two attachment / detachment count values ​​are different, it is determined that the parallax correction parameters are to be updated, and the process proceeds to S706, otherwise the process proceeds to S704.

[0074] In S704, the control unit 301 compares the stored motion count value with the new motion count value. The stored motion count value indicates the number of movements (movements greater than a threshold) of the imaging device 200 during the period until the captured image used to update the stored parallax correction parameter is captured. The new motion count value indicates the number of movements of the imaging device 200 during the period until the captured image to be processed is captured. The control unit 301 determines whether the stored motion count value is different from the new motion count value. If the two motion count values ​​are different, a movement of the imaging device 200 greater than a threshold occurs during the period from the capture of one of the two captured images to the capture of the other. The mounting state of the lens unit relative to the imaging device 200 may change due to a movement of the imaging device 200 greater than the threshold. Therefore, if the two motion count values ​​are different, there is a high possibility that the mounting state (mounting state of the lens unit relative to the imaging device 200) at the time of capture is (largely) different between the two captured images, and there is a high possibility that the parallax error is (largely) different. Therefore, if the two motion count values ​​are different, it is determined that the parallax correction parameters are to be updated, and the process proceeds to S706, otherwise the process proceeds to S705.

[0075] In S705, the control unit 301 compares the retained environmental temperature information with the new environmental temperature information. The retained environmental temperature information indicates the environmental temperature at the time of capturing the captured image used for updating the retained parallax correction parameter. The new motion count value indicates the environmental temperature at the time of capturing the captured image to be processed. The control unit 301 determines whether or not the difference between the environmental temperature indicated by the retained environmental temperature information and the environmental temperature indicated by the new environmental temperature information is greater than a threshold value. Assume that the threshold value is 5°C, and the environmental temperature indicated by the retained environmental temperature information is 20°C. In this case, the control unit 301 determines whether or not the environmental temperature indicated by the new environmental temperature information satisfies the condition "lower than 15°C or higher than 25°C." If the difference between the two environmental temperatures is large, there is a high possibility that the parallax error differs (largely). Therefore, if the difference between the two environmental temperatures is greater than the threshold value, it is determined that the parallax correction parameter is to be updated, and the process proceeds to S706. If not, it is determined that the parallax correction parameter is not to be updated (is to be reused), and the update determination process of FIG. 7 is terminated.

[0076] In S706, the control unit 301 sets a reset flag, which indicates whether or not to update the parallax correction parameter, to 1 (ON).

[0077] As described above, according to this embodiment, a captured image including a plurality of image regions captured through a plurality of optical systems is stored in a storage unit. At that time, information on at least one of the mounting state of the lens unit with respect to the imaging device and the environmental temperature is stored in the storage unit in association with the captured image. Then, using this information, it is determined whether or not to update parameters of an adjustment process such as a parallax correction process (adjustment process of at least one of the position and size of each image region). In this way, it is possible to determine parameters that can perform the adjustment process with high accuracy while reducing the time required to determine the parameters.

[0078] The above-described embodiment (including the modified examples) is merely an example, and the present invention also includes configurations obtained by appropriately modifying or changing the configurations of the above-described embodiment within the scope of the gist of the present invention. The present invention also includes configurations obtained by appropriately combining the configurations of the above-described embodiment.

[0079] For example, an example has been described in which it is determined whether or not a specific lens unit is attached to an imaging device based on lens information (S605 in FIG. 6), but it may also be determined whether or not a specific lens unit is attached to an imaging device by analyzing the captured image.

[0080] The judgment information is not limited to the information shown in FIG. 8. The judgment information may not include at least any of the shooting date and time information, the attachment / detachment count value, the motion count value, and the environmental temperature information shown in FIG. 8. The judgment information may include information on the shooting mode (a mode in which a captured image is stored in a storage unit). There may be no or small changes in the parallax error between multiple captured images obtained in a shooting mode (e.g., a shooting mode in which continuous shooting is performed) in which shooting (storing of images) is performed repeatedly at time intervals shorter than a predetermined time interval. Therefore, when a parallax correction process (association of parallax correction parameters, processes S601 to S614 in FIG. 6) is performed on a series of images obtained in such a shooting mode, the parallax correction parameters may not be updated. In other words, the parallax correction parameters generated for the first captured image may be reused for the second and subsequent captured images.

[0081] The imaging device may also include a storage device capable of continuing to hold information even when not powered. At least one of the motion count value and the attachment / detachment count value may be managed by such storage. The judgment information may include information regarding the power being turned on and off in the imaging device. When the battery runs out (power is turned off) and the battery is replaced (power is turned on again), the attachment state of the lens unit to the imaging device changes, The parallax error may change, so an update of the parallax correction parameters may be performed for such a situation.

[0082] The determination information may also include identification information such as the type, model number, serial number, and name of the lens unit used for shooting. When the lens unit used for shooting is replaced with another lens unit, the parallax error may change. In such a case, the parallax correction parameter may be updated.

[0083] The selection order when selecting the multiple captured images one by one as the target of the parallax correction process (association of the parallax correction parameters, the process of S601 to S614 in FIG. 6) is not particularly limited. The selection order may be the order of the shooting date and time, or it may not be the order of the shooting date and time. If the selection order is the order of the shooting date and time, determination information including at least one of the following four pieces of information may be stored at the time of shooting. Information indicating whether the time between the previous shooting date and time and the current shooting date and time is longer than a threshold (information indicating whether a predetermined time has passed between the previous shooting and the current shooting) - Information indicating whether the lens unit was removed from the imaging device between the previous shooting and the current shooting - Information indicating whether any movement of the imaging device occurred between the previous capture and the current capture that is greater than the threshold value - Information indicating whether the difference between the environmental temperature at the time of the previous shooting and the environmental temperature at the time of the current shooting is greater than a threshold value

[0084] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0085] The disclosure of the present embodiment includes the following configuration, method, program, and medium. (Configuration 1) an imaging means for imaging a plurality of image areas via a plurality of optical systems, respectively; an acquisition means for acquiring judgment information, the judgment information being information relating to at least one of an attachment state of a lens unit having the plurality of optical systems and an ambient temperature of the lens unit, and being information used to determine whether or not to update a parameter for an adjustment process of at least one of a position and a size of each of the plurality of image regions; a control means for storing an image including the plurality of image regions and the judgment information in a storage unit in association with each other; An imaging device comprising: (Configuration 2) The determination information includes information on the date and time when the image was stored in the storage unit. 2. The imaging device according to configuration 1, (Configuration 3) The determination information includes information on the number of times a lens unit is attached to and detached from the imaging device during a period until the image is stored in the storage unit. 3. The imaging device according to configuration 1 or 2. (Configuration 4) The determination information includes information on the number of times the imaging device moves greater than a threshold during a period until the image is stored in the storage unit. 4. The imaging device according to any one of configurations 1 to 3. (Configuration 5) The determination information includes information on the temperature around the lens unit when the image is stored in the storage unit. 5. The imaging device according to any one of configurations 1 to 4. (Configuration 6) The determination information includes information on a mode for storing the image in the storage unit. 6. The imaging device according to any one of configurations 1 to 5, (Configuration 7) The multiple image regions are two image regions for stereoscopic viewing. 7. The imaging device according to any one of configurations 1 to 6, (Configuration 8) The plurality of optical systems are two optical systems arranged in the left-right direction of the lens unit. 8. The imaging device according to any one of configurations 1 to 7, (Configuration 9) Each of the optical systems includes a fisheye lens. 9. The imaging device according to any one of configurations 1 to 8, (Configuration 10) Each of the plurality of image regions is a fisheye image region. 10. The imaging device according to any one of configurations 1 to 9. (Configuration 11) an image acquisition means for acquiring the image stored in the storage unit by the imaging device according to any one of configurations 1 to 10; an information acquisition means for acquiring the judgment information stored in the storage unit in association with the image by the imaging device; A determination means for determining parameters of the adjustment process; means for associating said parameters with said image; a storage means for storing the updated parameters when the determining means updates the parameters based on the image; having The determining means switches between updating the parameters based on the image and using the stored parameters as the current parameters based on the judgment information. 23. An information processing apparatus comprising: (Configuration 12) Associating the parameter with the image means performing the adjustment process using the parameter on the image. 12. The information processing device according to configuration 11. (Configuration 13) The storage means stores the updated parameters and the acquired judgment information, The determining means compares the acquired judgment information with the stored judgment information, and switches between updating the parameters based on the image and using the stored parameters as the current parameters. 13. The information processing device according to configuration 11 or 12. (Configuration 14) the determination information includes information on the date and time when the image was stored in the storage unit, The determining means updates the parameters based on the image when a time period from a date and time when the image used to update the held parameters was stored in the storage unit to a date and time when the acquired image was stored in the storage unit is longer than a threshold value. 14. The information processing device according to configuration 13. (Configuration 15) the determination information includes information on the number of times a lens unit is attached to and detached from the imaging device during a period until the image is stored in the storage unit, The determining means updates the parameters based on an image when the number of times of attachment and detachment during a period until an image used for updating the held parameters is stored in the storage unit differs from the number of times of attachment and detachment during a period until an acquired image is stored in the storage unit. 15. The information processing device according to any one of configurations 11 to 14. (Configuration 16) the determination information includes information on the number of times the imaging device moves greater than a threshold during a period until the image is stored in the storage unit, The determining means updates the parameters based on an image when the number of times of the movement during a period until an image used for updating the held parameters is stored in the storage unit differs from the number of times of the movement during a period until an acquired image is stored in the storage unit. 16. The information processing device according to any one of configurations 11 to 15. (Configuration 17) the determination information includes information about the temperature around the lens unit when the image was stored in the storage unit, The determining means updates the parameters based on the image when a difference between the temperature when the image used for updating the held parameters is stored in the storage unit and the temperature when the acquired image is stored in the storage unit is greater than a threshold value. 17. The information processing device according to any one of configurations 11 to 16. (Configuration 18) the determination information includes information on a mode for storing the image in the storage unit, The determining means uses the parameters stored in the memory as the current parameters when the mode is a mode in which images are repeatedly stored at time intervals shorter than a predetermined time interval. 18. The information processing device according to any one of configurations 11 to 17. (Method 1) A method for controlling an imaging device having an imaging unit that captures images of a plurality of image areas via a plurality of optical systems, comprising the steps of: an acquisition step of acquiring judgment information, the judgment information being information regarding at least one of an attachment state of a lens unit having the plurality of optical systems and an ambient temperature of the lens unit, and being information used to determine whether or not to update a parameter of an adjustment process of at least one of a position and a size of each of the plurality of image regions; a control step of storing the image including the plurality of image regions and the determination information in a storage unit in association with each other; 13. A method for controlling an electronic device comprising: (Method 2) An image acquiring step of acquiring the image stored in the storage unit by the imaging device according to any one of configurations 1 to 10; an information acquisition step of acquiring the judgment information stored in the storage unit in association with the image by the imaging device; A determination step of determining parameters of the adjustment process; associating said parameters with said image; a holding step of holding the updated parameters when the parameters are updated based on the image in the determining step; having The determining step switches between updating the parameters based on the image or using the stored parameters as the current parameters based on the determination information. 23. A method for controlling an information processing apparatus comprising: (Program 1) A program for causing a computer to function as each of the means of the imaging device according to any one of configurations 1 to 10. (medium 1) 11. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the imaging device according to any one of configurations 1 to 10. (Program 2) A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 11 to 18. (medium 2) 19. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the information processing device according to any one of configurations 11 to 18. [Explanation of symbols]

[0086] 200: Imaging device 201: Control unit 300: Personal computer (PC) 301: Control unit 302: RAM

Claims

1. an imaging means for capturing images of a plurality of image areas via a plurality of optical systems, respectively; an acquisition means for acquiring determination information, which is information relating to at least one of the mounting state of a lens unit having the plurality of optical systems and the ambient temperature of the lens unit, and which is information used to determine whether or not to update parameters for adjustment processing of at least one of the position and size of each of the plurality of image regions; a control means for storing the image including the plurality of image regions and the determination information in a storage unit in association with each other; An imaging device comprising:

2. The determination information includes information on the date and time when the image was stored in the storage unit.

2. The imaging device according to claim 1.

3. The determination information includes information on the number of times a lens unit is attached to and detached from the imaging device during a period until the image is stored in the storage unit.

2. The imaging device according to claim 1.

4. The determination information includes information on the number of times the imaging device moves greater than a threshold during a period until the image is stored in the storage unit.

2. The imaging device according to claim 1.

5. The determination information includes information about the ambient temperature of the lens unit when the image is stored in the storage unit.

2. The imaging device according to claim 1.

6. The determination information includes information on a mode for storing the image in the storage unit.

2. The imaging device according to claim 1.

7. The plurality of image regions are two image regions for stereoscopic viewing.

2. The imaging device according to claim 1.

8. The plurality of optical systems are two optical systems arranged in the left-right direction of the lens unit.

2. The imaging device according to claim 1.

9. Each of the plurality of optical systems includes a fisheye lens.

2. The imaging device according to claim 1.

10. Each of the plurality of image regions is a fisheye image region.

2. The imaging device according to claim 1.

11. an image acquisition unit that acquires the image stored in the storage unit by the imaging device according to claim 1; an information acquisition unit that acquires the determination information stored in the storage unit in association with the image by the imaging device; a determination means for determining parameters for the adjustment process; associating means for associating said parameters with said image; a storage means for storing the updated parameters when the determining means updates the parameters based on the image; and The determining means switches between updating the parameters based on the image or using the stored parameters as the current parameters based on the determination information.

1. An information processing device comprising:

12. Associating the parameter with the image means performing the adjustment process using the parameter on the image.

12. The information processing apparatus according to claim 11,

13. the storage means stores the updated parameters and the acquired determination information; The determining means compares the acquired judgment information with the stored judgment information, and switches between updating the parameters based on the image or using the stored parameters as the current parameters.

12. The information processing apparatus according to claim 11,

14. the determination information includes information on the date and time when the image was stored in the storage unit, The determining means updates the parameters based on the image when a time period from a date and time when the image used to update the held parameters was stored in the storage unit to a date and time when the acquired image was stored in the storage unit is longer than a threshold value.

14. The information processing apparatus according to claim 13,

15. the determination information includes information on the number of times a lens unit is attached to and detached from the imaging device during a period until the image is stored in the storage unit, The determining means updates the parameters based on an image when the number of times of detachment during a period until an image used for updating the held parameters is stored in the storage unit differs from the number of times of detachment during a period until an acquired image is stored in the storage unit.

12. The information processing apparatus according to claim 11,

16. the determination information includes information on the number of times the imaging device moves greater than a threshold during a period until the image is stored in the storage unit; The determining means updates the parameters based on the image when the number of times of movement during a period until the image used to update the held parameters is stored in the storage unit differs from the number of times of movement during a period until the acquired image is stored in the storage unit.

12. The information processing apparatus according to claim 11,

17. the determination information includes information about the ambient temperature of the lens unit when the image was stored in the storage unit, The determining means updates the parameters based on the image when a difference between the temperature when the image used to update the held parameters is stored in the storage unit and the temperature when the acquired image is stored in the storage unit is greater than a threshold value.

12. The information processing apparatus according to claim 11,

18. the determination information includes information about a mode for storing the image in the storage unit, When the mode is a mode in which images are repeatedly stored at time intervals shorter than a predetermined time interval, the determining means reuses the stored parameters as the current parameters.

12. The information processing apparatus according to claim 11,

19. An imaging device having an imaging means for capturing images of a plurality of image areas via a plurality of optical systems, respectively. A control method comprising: an acquisition step of acquiring determination information, which is information relating to at least one of the mounting state of the lens unit having the plurality of optical systems and the ambient temperature of the lens unit, and is information used to determine whether or not to update parameters for adjustment processing of at least one of the position and size of each of the plurality of image regions; a control step of storing the image including the plurality of image regions and the determination information in a storage unit in association with each other; 10. A method for controlling an imaging device, comprising:

20. an image acquisition step of acquiring the image stored in the storage unit by the imaging device according to claim 1; an information acquisition step of acquiring the determination information stored in the storage unit in association with the image by the imaging device; a determining step of determining parameters for the adjustment process; associating said parameters with said image; a holding step of holding the updated parameters when the parameters are updated based on the image in the determining step; and The determining step switches between updating the parameters based on the image or using the stored parameters as the current parameters based on the determination information.

2. A method for controlling an information processing apparatus comprising:

21. A program for causing a computer to function as each of the means of the imaging device according to any one of claims 1 to 10.

22. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the imaging device according to any one of claims 1 to 10.

23. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 11 to 18.

24. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the information processing device according to any one of claims 11 to 18.

Citation Information

Patent Citations

  • Method and device for printing

    JP2000098514A