Information processing device, information processing method, imaging device, program, and storage medium

Abstract

To provide an information processing apparatus that can acquire information for performing relative alignment between a chart and an imaging apparatus.SOLUTION: An information processing apparatus comprises: a first acquisition unit that acquires a photographed image obtained by photographing a chart for calibration; a second acquisition unit that acquires a reference image being an image to be a reference of the chart; a third acquisition unit that acquires information on the distortion aberration of a lens used for photographing for the photographed image; a first creation unit that creates a pseudo image being an image reflecting the distortion aberration on the reference image based on the reference image and the information on the distortion aberration; and a second creation unit that creates a composite image obtained by compositing the photographed image with the pseudo image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing technique for acquiring information for aligning a camera and a chart. [Background technology]

[0002] In recent years, a technology called VFX (Visual Effects) has been attracting attention in the field of image production. VFX is intended to create unrealistic screen effects (visual effects), and is achieved by compositing CG (Computer Graphics) with live-action images. When compositing CG and live-action images, if you try to do so without taking into account the amount of lens distortion, depending on the imaging lens, a difference will occur between the live-action image with lens distortion aberration and the CG image without lens distortion aberration. For this reason, in post-production, it is necessary to first correct the lens distortion aberration in the live-action image, compose it with the CG image, and then perform distortion processing on the composite image.

[0003] Distortion aberration can be corrected according to the image height by acquiring distortion information as metadata from the camera (lens). However, depending on the imaging lens, it may not be possible to acquire distortion information. When distortion information cannot be acquired, the amount of lens distortion is generally calculated by photographing a grid-shaped calibration chart in which black and white are arranged repeatedly during actual shooting, and then using a dedicated application from the captured image in post-production. This method of photographing a calibration chart requires accurate alignment (position and orientation) of the chart and camera. At current shooting sites, the alignment of the chart and camera is performed visually by the user, and cannot be performed accurately.

[0004] Patent document 1 discloses a method in which a reference image that serves as a basis for adjusting the angle of view is stored in a field of view adjustment device (camera), and a moving image from a camera whose field of view has not been adjusted is synthesized with the reference image and displayed on a display device.

[0005] Patent Document 2 discloses a method assuming a fisheye lens, storing grid-like chart information (model data) of the subject in an imaging device, and estimating the amount of lens distortion from a captured image and the chart information. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2014-155086 A [Patent Document 2] Patent No. 6859442 Summary of the Invention [Problem to be solved by the invention]

[0007] However, the above-mentioned Patent Document 1 does not take into consideration the distortion of the lens attached to the camera that captures the moving image. Therefore, when the lens distortion aberration is large, the deviation between the superimposed reference image and the captured moving image becomes large at the peripheral angle of view (high image height), which makes alignment difficult.

[0008] Furthermore, the above-mentioned Patent Document 2 discloses a method of estimating the position and orientation of an imaging device by acquiring reference image information (chart model data) and lens distortion parameters estimated from a captured image, assuming a fisheye lens. However, it does not mention a method of aligning the chart and the imaging device.

[0009] The present invention has been made in consideration of the above-mentioned problems, and an object of the present invention is to provide an information processing device that can acquire information for performing relative alignment between a chart and an imaging device. [Means for solving the problem]

[0010] The information processing device according to the present invention comprises: An information processing device provided in an imaging device,A chart having a grid pattern for correcting distortion of a photographic lens is attached to the photographic lens. The above a first acquisition means for acquiring a captured image obtained by capturing an image with an imaging means in an imaging device from the imaging means, a second acquisition means for acquiring a reference image which is an image serving as a reference for the chart, a third acquisition means for acquiring information on distortion aberration of the photographing lens, a first generation means for generating a pseudo image which is an image in which the distortion aberration is reflected in the reference image based on the reference image and the information on the distortion aberration, and a second generation means for generating a composite image by combining the captured image and the pseudo image. a calculation means for calculating a degree of deviation of an image based on the photographed image and the pseudo image, and further calculating an evaluation value based on the degree of deviation; The position of the photographed image is aligned with the position of the pseudo image. For A moving indicator of the imaging device a movement indicator for allowing a user to move the imaging device and adjust the position and orientation of the imaging device with respect to the chart; of , based on the evaluation value a third generating means for generating a display control means for displaying the moving indicator in a superimposed manner on the composite image; The present invention is characterized by comprising: Effect of the Invention

[0011] According to the present invention, it is possible to provide an information processing apparatus capable of acquiring information for performing relative alignment between a chart and an imaging device. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram showing the configuration of an imaging apparatus according to a first embodiment of the present invention. [Diagram 2] 4 is a flowchart showing a calibration process of the imaging device. [Diagram 3] 13 is a flowchart showing a simple alignment process. [Figure 4] 6A to 6C are diagrams illustrating a display screen during simple alignment processing. [Diagram 5] 11 is a flowchart showing a position alignment process. [Figure 6] 5A to 5C are diagrams for explaining a process of estimating a lens distortion amount. [Figure 7] 6A to 6C are diagrams illustrating a display screen during alignment processing. [Figure 8] 10 is a flowchart showing alignment processing according to the second embodiment. [Figure 9] 10A to 10C are views for explaining a process of calculating a degree of deviation between a captured image and a pseudo image in the second embodiment. [Figure 10] FIG. 13 is a diagram illustrating a display screen of an index based on the deviation degree. [Figure 11] 13 is a flowchart showing a calibration process according to the third embodiment. [Figure 12] 13 is a flowchart showing a view angle adjustment process according to the third embodiment. [Figure 13A] 13 is a flowchart showing a process of calculating a difference between a chart and a captured image in the third embodiment. [Figure 13B] 13 is a flowchart showing a process of calculating a difference between a chart and a captured image in the fourth embodiment. [Figure 14] 13 is a flowchart showing a process of calculating a camera movement amount in the third embodiment. [Figure 15] 13A to 13C are diagrams for explaining calculation of the amount of movement of a camera in the third embodiment. [Figure 16] 13A to 13C are diagrams for explaining calculation of the amount of movement of a camera in the third embodiment. [Figure 17] 13A to 13C are diagrams for explaining calculation of the amount of movement of a camera in the third embodiment. [Figure 18] 13A to 13C are diagrams illustrating camera displays during and after angle of view adjustment in the third embodiment. [Figure 19] 13A to 13C are views for explaining a process of calculating a difference between a chart and a captured image in the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0014] <First embodiment> (Configuration of imaging device) The configuration of an image capturing apparatus, which is a first embodiment of an information processing apparatus according to the present invention, will be described below.

[0015] 1, the imaging device 100 is configured to include an imaging control device 130 and an imaging lens 110 that is replaceably attached to the imaging control device 130. Examples of the imaging device 100 include, but are not limited to, a video camera or a still camera that can capture an image of a subject and record video or still image data on various recording media. Hereinafter, the imaging device 100 will be described as a camera 100.

[0016] The arithmetic device 136 controls the entire imaging control device 130. In addition, via the electrical contact unit 150, the arithmetic device 136 transmits to the lens control unit 121 commands for driving the lens group and the aperture in the imaging lens 110, commands for transmitting lens information (optical information, etc.) held in the imaging lens 110 to the imaging control device 130, and the like.

[0017] The imaging lens 110 is configured as a lens unit equipped with an imaging optical system including a fixed lens group 111, a zoom lens 112, an aperture 113, an image stabilization lens 114, and a focus lens 115. Drivers for driving the lenses and apertures are connected to a lens control unit 121 via a bus 122 and controlled by the lens control unit 121. The lens control unit 121 controls the various lenses and apertures via a zoom driver 116, an aperture driver 117, an image stabilization driver 118, and a focus driver 119 in accordance with commands from a calculation device 136.

[0018] The aperture driver 117 drives the aperture 113 to adjust the aperture diameter of the aperture 113, thereby adjusting the amount of light during shooting. The zoom driver 116 drives the zoom lens 112 to change the focal length. The image stabilization driver 118 drives the image stabilization lens 114 in accordance with the shake of the imaging lens 110, thereby reducing image blur caused by the shake. The focus driver 119 drives the focus lens 115 to control the focus state. Although the various lenses 111, 112, 114, 115 are simply shown as single lenses in FIG. 1, they are usually composed of multiple lenses.

[0019] The electrical contact unit 150 has electrical contacts (imaging control device side terminal / imaging lens side terminal) respectively corresponding to two communication lines used for communication between the imaging lens 110 and the imaging control device 130. The lens control unit 121 communicates with the imaging control device 130 via the electrical contact unit 150, and drives and controls the zoom driving unit 116, the aperture driving unit 117, and the focus driving unit 119 in response to operation information from the lens operation unit 120. The lens control unit 121 also communicates with the imaging control device 130 via the electrical contact unit 150, and receives commands from the arithmetic unit 136. Furthermore, the lens control unit 121 transmits lens information (optical information, etc.) held in the imaging lens 110 based on a transmission request from the imaging control device 130 (hereinafter, communication between the lens control unit 121 and the arithmetic unit 136 will be referred to as lens communication).

[0020] The lens operation unit 120 is provided with operation members such as a zoom operation ring, a focus operation ring, an aperture operation ring, and an operation switch for turning on / off the in-lens image stabilization. When any of these operation members is operated by the user, an operation instruction signal is output to the lens control unit 121, and the lens control unit 121 performs control according to the corresponding operation.

[0021] The subject image formed on the imaging element 131 by the light beam passing through the imaging optical system of the imaging lens 110 is converted into an electric signal by the imaging element 131. The imaging element 131 is a photoelectric conversion element that converts the subject image (optical image) into an electric signal by photoelectric conversion. The electric signal obtained by photoelectrically converting the subject image formed on the imaging element 131 is processed as an image signal (image data) by the imaging signal processing unit 132.

[0022] The imaging element control unit 133 receives instructions from the calculation device 136 regarding the accumulation time of the imaging element 131 and the value of the gain to be output from the imaging element 131 to the imaging signal processing unit 132 , and controls the imaging element 131 .

[0023] The image data output from the imaging signal processing unit 132 is sent to the imaging element control unit 133 and temporarily stored in a volatile memory 138. The image data is then subjected to correction and compression processing in the image processing unit 137, and then recorded on a recording medium 143 such as a memory card.

[0024] In parallel with this, the display control unit 141 performs a process of reducing and enlarging the image data stored in the volatile memory 138 to an optimal size for the display unit 140 such as a display mounted on the imaging control device 130 based on a command from the arithmetic device 136. The image data processed to the optimal size is temporarily stored again in an area of ​​the volatile memory 138 different from the area before processing. Furthermore, the display control unit 141 superimposes imaging information such as exposure settings on the image data with characters, icons, etc. The image data on which various information is superimposed is transmitted to the display unit 140, whereby the image is displayed. This allows the user to observe the captured image in real time (hereinafter, an image that can be observed in real time is referred to as a live view image). The display control unit 141 also controls the process of superimposing the captured image and the pseudo image performed in this embodiment.

[0025] Based on a command from the arithmetic unit 136, the image stabilization control unit 135 controls the image sensor 131 in a direction in which image blur caused by camera shake is corrected via the image stabilization driving unit 134. The image stabilization driving unit 134 can also be driven in conjunction with the image stabilization driving unit 118 of the imaging lens 110, and can perform image stabilization over a wider range than can be achieved by a single unit.

[0026] The operation unit 142 is an operating member with which the user issues instructions to each unit in the imaging control device 130, and includes, for example, an operation switch, an operation ring, an operation lever that control imaging operations and focus adjustment operations, or a touch panel installed on the display unit 140. Instructions related to the operating conditions of the camera 100 input by the user operating the operation unit 142 are transmitted to the arithmetic device 136. Then, the arithmetic device 136 transmits commands to each unit based on the operation instruction signal.

[0027] The volatile memory 138 is used not only for temporarily storing the image data described above, but also for storing temporary data used in processing in each section of the imaging control device 130, lens information obtained from the imaging lens 110, and the like.

[0028] Non-volatile memory 139 stores a control program necessary for the operation of camera 100. When camera 100 is started up by a user's operation (when the camera transitions from a power-off state to a power-on state), the control program stored in non-volatile memory 139 is read (loaded) into a part of volatile memory 138. Arithmetic unit 136 controls the operation of camera 100 in accordance with the control program loaded into volatile memory 138. Non-volatile memory 139 is also writable, and includes within it a chart information storage unit 139a and a lens distortion information storage unit 139b that store chart information and distortion information used in this embodiment.

[0029] The storage medium 143 is a readable / writable memory card such as an SD card, and is used to store captured images (moving images or still images), metadata associated with the images, etc. It is also envisioned that the storage medium 143 may be used in place of the chart information storage unit 139a and the lens distortion information storage unit 139b in the non-volatile memory 139.

[0030] (Outline of the calibration process) Next, an overview of the calibration process will be described with reference to Fig. 2. In this embodiment, in order to correct the distortion aberration of the imaging lens 110, it is necessary to actually capture an image of a calibration chart for aberration correction with the camera 100. The calibration process refers to an operation of aligning the calibration chart with the camera 100, and then capturing an image of the calibration chart with the camera 100 to obtain information on the distortion aberration of the imaging lens 110.

[0031] The following description is based on the assumption that the above-mentioned imaging control is started before the calibration process is executed, and a live view image can be observed. This calibration operation itself is performed after the user selects to execute the calibration process from a menu displayed on the display unit 140 via the operation unit 142. The description is based on the assumption that the captured image of the camera 100 is always in focus with respect to the chart. The method of focusing may be manual or autofocus, and is not particularly limited.

[0032] First, in step S201, the arithmetic unit 136 starts the calibration process, and the process proceeds to step S202.

[0033] In step S202, the arithmetic unit 136 turns on the calibration mode. Specifically, the arithmetic unit 136 changes the flag for the calibration mode stored in the volatile memory 138 to the on state.

[0034] In step S203, the arithmetic device 136 performs a simple alignment process between the camera 100 and the calibration chart. This simple alignment process roughly aligns the center of the optical axis of the camera 100 with the center of the calibration chart and adjusts the shooting angle of view for initial alignment. The details of this simple alignment process will be described later.

[0035] In step S204, the arithmetic device 136 determines whether the simple alignment process in step S203 is complete. If the simple alignment process is complete, the process proceeds to step S205, and if it is not complete (an error occurs during the simple alignment or the process is stopped by the user), the process proceeds to step S208.

[0036] In step S205, the arithmetic device 136 performs an alignment process between the camera 100 and the calibration chart. In this alignment process, detailed alignment is performed while checking even the peripheral (high image height) parts of the image, in contrast to the simple alignment process in step S203, until an installation state is reached where image shooting can be performed to calculate the amount of lens distortion. The details of this alignment process will be described later.

[0037] In step S206, the arithmetic device 136 determines whether the alignment process in step S205 is complete. If the alignment process is complete, the process proceeds to step S207, and if the alignment process is not complete (an error occurs during alignment or the process is stopped by the user), the process proceeds to step S208.

[0038] In step S207, the arithmetic unit 136 captures an image of the calibration chart in the installation state of the camera 100 and the chart after the positioning process has been completed. The arithmetic unit 136 stores the captured image in the storage medium 143, and the process proceeds to step S208.

[0039] In step S208, the arithmetic unit 136 turns off the calibration mode, and then proceeds to step S209 to end the calibration process.

[0040] In an actual shooting scene, it is assumed that shooting will be performed using multiple imaging lenses or multiple focal lengths in the case of a zoom lens. In this case, the processing of steps S201 to S209 in FIG. 2 is repeatedly performed for each imaging lens or each focal length.

[0041] By performing the above process, it is possible to obtain an image for calculating the amount of lens distortion after aligning the calibration chart and the camera with high precision. In the post-production work of VFX compositing, the amount of lens distortion is calculated from this captured image by using a dedicated application, and is used for various compositing processes.

[0042] (Simple alignment process) Next, the detailed operation of the simplified alignment process in step S203 in Fig. 2 will be described with reference to Fig. 3. The process of the flowchart in this embodiment is executed by the arithmetic unit 136 based on a computer program loaded from the non-volatile memory 139 to the volatile memory 138 when the camera 100 is started up. The operations of the subsequent flowcharts are similar.

[0043] First, in step S301, the arithmetic unit 136 starts the simple alignment process.

[0044] In step S302, the calculation device 136 reads out the captured image during the above-mentioned imaging control from the volatile memory 138, and temporarily stores it in another area of ​​the volatile memory 138. In addition, the calculation device 136 obtains chart information used for alignment from the chart information storage unit 139a.

[0045] In step S303, the arithmetic unit 136 generates a pseudo image based on the chart information acquired in step S302. The pseudo image is an image before distortion processing or an image without distortion processing. Here, distortion processing refers to a process of intentionally distorting a chart image that does not originally have lens distortion aberration in accordance with the lens distortion aberration, and converting it into an image that reflects the lens distortion aberration (amount of distortion). Hereinafter, the pseudo image of the chart before distortion processing or without distortion processing is called a reference image.

[0046] The chart information is information necessary for generating a reference image or a pseudo image. For example, the chart information is the calibration chart image itself that is the subject, the size of one grid of the calibration chart, etc. If this information is associated with the focal length of the imaging lens, it is possible to automatically perform processing according to the focal length even when the imaging lens is replaced or the focal length of a zoom lens is changed. In addition, it is assumed that the user himself stores the chart information in the chart information storage unit 139a or the storage medium 143 in the imaging control device 130 before the calibration process is performed.

[0047] However, if no chart information is stored or if the chart information differs greatly from the actual captured image, the following may be done. That is, a chart information generation mode is provided, and chart information is generated by a user's selection or input on the imaging control device 130. Alternatively, chart information is generated by detecting the size of one grid portion in the center of the captured image. This is then stored as new chart information in the chart information storage unit 139a.

[0048] Here, an example of generating a reference image based on chart information will be described, assuming that a chart image corresponding to the focal length is stored in the non-volatile memory 139a.

[0049] First, the arithmetic unit 136 reads out chart information from the non-volatile memory 139a. At the same time, the focal length of the imaging lens 110 attached to the imaging control device 130 is acquired by lens communication. Alternatively, for a lens that does not support lens communication, the user may set the focal length for the imaging control device 130 by himself.

[0050] Since the read chart information is a chart image corresponding to the focal length, if the focal length differs from that acquired by lens communication, the chart image needs to be processed according to the difference in focal length. Therefore, the calculation device 136 notifies the image processing unit 137 of the focal length of the imaging lens and instructs the image processing unit 137 to perform enlargement / reduction processing according to the difference in focal length (magnification). Based on the instruction, the image processing unit 137 performs enlargement / reduction processing on the read chart image. Next, the calculation device 136 temporarily stores the generated reference image in the volatile memory 138, and proceeds to step S304.

[0051] In step S304, the arithmetic device 136 performs a process of superimposing the reference image on the captured image using the display unit 140 and the display control unit 141. First, the display control unit 141 reads out the captured image and the reference image temporarily stored in the volatile memory 138. Then, image synthesis is performed so that the reference image is displayed superimposed on the upper side of the acquired captured image. Hereinafter, this process is called a superimposition process. Also, the image that has been subjected to the superimposition process is called a superimposed image (synthesized image).

[0052] At this time, if the reference image is simply superimposed, the captured image below cannot be observed, and it is difficult to observe the difference between the captured image and the reference image. Therefore, the display control unit 141 performs necessary image processing on the reference image. For example, the reference image is subjected to a transparent processing so that the captured image can be observed through the reference image, a processing is performed to convert the black part of the pseudo image corresponding to the black of the chart grid part of the captured image to another color such as red, or a pattern processing is performed. Then, the display control unit 141 temporarily stores the superimposed image in the volatile memory 138.

[0053] Next, the display control unit 141 notifies the display unit 140 of the temporary storage destination of the superimposed image, and commands it to update the display. The display unit 140 reads out the temporarily stored superimposed image from the volatile memory 138 and displays it. By observing the superimposed image displayed on the display unit 140, the user can check the degree of discrepancy between the captured image and the pseudo image. In other words, it becomes possible to observe the positional deviation between the calibration chart and the camera. Then, the user changes the position or attitude of the camera 100 or the calibration chart to minimize this positional deviation, and performs positional alignment.

[0054] An example of a display screen during simple alignment will be described with reference to Fig. 4. Fig. 4(a) to Fig. 4(e) show the screen being displayed on the display unit 140. Note that the calibration chart itself is a chart in which black and white grids are repeatedly arranged as described above, but hereinafter, the black grid part of the captured image will be represented by hatching with diagonal lines slanting downward to the right, and the black grid part of the reference image or pseudo image will be represented by hatching with diagonal lines slanting upward to the right.

[0055] FIG. 4(a) shows a state where only the captured image 410 is displayed. Reference numerals 401 and 402 denote a vertical marker and a horizontal marker that are shown to intersect at the optical center so that the center of the captured image can be easily recognized. Also, a black rectangular marker 403 is shown so that the center position of the calibration chart can be easily recognized, but a marker may actually be placed at the center position of the calibration chart, or only the rectangular portion including the center position may be set to a color other than black and white. In addition, when performing the superimposition process, the display method or color may be changed so that the grid portion including the center position on the reference image side is matched to the marker or color of the calibration chart so that the center alignment can be easily recognized.

[0056] Next, Fig. 4(b) shows a state in which a reference image 411 is superimposed on the state in which only the captured image 410 in Fig. 4(a) is displayed. Simple alignment is performed based on this superimposed reference image 411. In this example, the user changes the position and orientation of the camera 100 so that the center of the optical axis of the camera 100 and the center of the calibration chart are aligned.

[0057] Fig. 4(c) shows a state where the center positions of the captured image 410 and the reference image 411 are aligned, but the captured image 410 and the reference image 411 are not yet aligned (their sizes are not the same). From this state, the user changes the position so that the camera 100 approaches the chart and the display state of Fig. 4(d) is reached. By executing the above process, the simplified alignment of the camera 100 and the chart is completed.

[0058] As described above, the reference image can be enlarged / reduced according to the focal length acquired by lens communication or set by the user. However, depending on the imaging lens, the focal length itself may not be known, and the reference image may not be enlarged / reduced. In such a case, the user operates the operation unit 140 to enlarge / reduc the reference image 411 from the state in which the sizes do not match as shown in FIG. 4(c), and continues the operation until it can be determined by visual inspection that the captured image 410 and the reference image 411 match (until the state reaches the same as FIG. 4(d)).

[0059] Next, in step S305, the arithmetic device 136 judges whether or not to continue the simple alignment. This is for the purpose of judging whether or not the user has completed the simple alignment. For example, a button-shaped user interface such as "OK" / "CANCEL" is further superimposed on the superimposed image, and "OK" is assigned to completion and "CANCEL" to cancellation. Then, when the user decides to complete or cancel the simple alignment, he / she selects either "OK" (complete) or "CANCEL" (cancel) via the operation unit 142. Next, the arithmetic device 136 judges whether or not to select whether or not to continue the alignment process. If completion or cancellation is not selected, the process returns to step S302, and the process of steps S302 to S304 is continued until completion or cancellation is selected, or until the process is forcibly terminated via the operation unit 142. If completion or cancellation is selected, the process proceeds to step S306.

[0060] In step S306, the arithmetic device 136 determines whether "Complete" or "Cancel" was selected in step S305, that is, whether the simple alignment is completed or not. If "Complete" was selected in step S305, the arithmetic device 136 advances the process to step S307 and sets the result of the simple alignment to "OK". If "Cancel" was selected, the arithmetic device 136 advances the process to step S308 and sets the result of the simple alignment to "CANCEL".

[0061] In step S309, the calculation device 136 temporarily stores the result of the simple alignment set in either step S307 or S308 in the volatile memory 138. Additionally, the calculation device 136 stores the reference image at the time of completion of the simple alignment in the chart information storage unit 139a as chart information.

[0062] In step S310, the calculation device 136 ends the simple registration process.

[0063] (Alignment process) Next, a registration process that performs registration with higher accuracy than the simple registration process will be described with reference to FIG.

[0064] First, in step S501, the arithmetic unit 136 starts the alignment process.

[0065] In step S502, the arithmetic unit 136 reads out the chart information from the chart information storage unit 139a in the non-volatile memory 139.

[0066] In step S503, the calculation device 136 reads out the captured image during the above-mentioned imaging control from the volatile memory 138, and temporarily stores the image in another area of ​​the volatile memory 138.

[0067] In step S504, the arithmetic unit 136 performs a lens distortion amount acquisition process, and the process proceeds to step S504.

[0068] An example of the estimation process among the lens distortion amount acquisition processes will be described with reference to Fig. 6. Fig. 6(a) is a diagram illustrating grid information of the calibration chart (information for identifying each grid and the positions of the four corners of the grid).

[0069] Here, there are 17 grids arranged horizontally and 12 vertically, including black and white, and to identify the positions of the four corners of the grid, numbers 0 to 17 are assigned horizontally (i0 to i17) and numbers 0 to 11 are assigned vertically (j0 to j11). Next, to identify each grid, consecutive numbers 0 to 186 are assigned from the top left (g0 to g186). Therefore, the coordinates of the four corners of the top left grid g0 can be expressed as top left (xi0j0, yi0j0), top right (xi1j0, yi1j0), bottom left (xi0j1, yi0j1), and bottom right (xi1j1, yi1j1).

[0070] 6(b), the coordinates of the lattice section 601 of the captured image are set as upper left (Cxi0j0, Cyi0j0), upper right (Cxi1j0, Cyi1j0), lower left (Cxi0j1, Cyi0j1), and lower right (Cxi1j1, Cyi1j1) so that the coordinates of the captured image and the reference image can be identified. The coordinates of the four corners of the lattice section 602 on the reference image side are set as upper left (Rxi0j0, Ryi0j0), upper right (Rxi1j0, Ryi1j0), lower left (Rxi0j1, Ryi0j1), and lower right (Rxi1j1, Ryi1j1).

[0071] Next, the estimation process performed based on the above grid information will be described. First, lens distortion is generally modeled by the following formula: xd, yd are coordinates with distortion, xu, yu are coordinates without distortion, and K1 to K5 are coefficients that represent the amount of lens distortion.

[0072] xd=(1+K1r 2 +K2r 4 +K5r 6 )xu+2K3xuyu+K4(r 2 +2xu 2 ) yd=(1+K1r 2 +K2r 4 +K5r 6 )yu+K3(r 2 +2yu 2 )+2K4xuyu r 2 =xu 2 +yu 2 The amount of lens distortion calculated using these formulas is stored in the lens distortion information storage unit 139b. At this time, the focal length of the imaging lens, the lens ID, the serial number, etc. may be stored in association with the amount of lens distortion. In this way, the information can be reused when performing calibration again.

[0073] In the above equation, K3 and K4 represent tangential distortion aberration. However, since this can often be ignored in practical use, a simplified model such as the following equation is often used.

[0074] xd=(1+K1r 2 +K2r 4 +K5r 6 )xu yd=(1+K1r 2 +K2r 4 +K5r 6 )yu The coordinates of the upper left corner of the grid g=0 in FIG. 6 are as follows:

[0075] Cxi0j0=(1+K1r 2 +K2r 4 +K5r 6 )Rxi0j0 Cyi0j0=(1+K1r 2 +K2r 4 +K5r 6 )Ryi0j0 r 2 =Rxi0j0 2 +Ryi0j0 2 Using this relational expression, the distortion amounts K1, K2, and K5 are searched for so that the coordinates of the four corners of each grid (g = 0 to 186) of the captured image and the reference image match or are close to each other. If the search results in matching or close coordinates, it can be determined that the estimation is complete.

[0076] In this embodiment, only the four corners of each grid are matched, but when the coordinates between the four corners of each grid are further divided in a state where they are matched to a certain extent, the accuracy of the estimated distortion amount is improved by increasing the number of coordinates to be compared. Also, in this embodiment, the estimation process is a search for a coefficient representing the distortion amount, but this is not limited to this. Next, the calculation device 136 stores the estimated lens distortion amount in the lens distortion information storage unit 139b.

[0077] In the present embodiment, the method of estimating the lens distortion amount from the captured image and the chart information stored in the imaging control device 130 has been described. However, depending on the imaging lens, the lens distortion amount corresponding to the state of each lens (such as focal length) may be stored in the imaging lens 110 or the imaging control device 130 (non-volatile memory 139). These lens distortion amounts are determined when the imaging lens is designed or manufactured. If the distortion amount is stored in the imaging lens, lens communication is performed in the lens distortion amount acquisition process in step S506, and the distortion amount at each lens position at the time of communication is acquired from the lens side. If it is stored on the imaging control device 130 side, lens communication is first performed to acquire a unique number (hereinafter, referred to as a lens ID) assigned to each imaging lens. Then, the lens ID and the lens distortion amount associated with each lens position are read from the non-volatile memory 139. A pseudo image, which will be described later, may be generated based on these lens distortion amounts.

[0078] Next, in step S505, the arithmetic device 136 generates a pseudo image from the lens distortion amount and chart information acquired in step 504. The process before the distortion process is the same as the process of generating the reference image in step S303 of FIG. 3, so a description thereof will be omitted. Next, the arithmetic device 136 reads out the reference image temporarily stored in the volatile memory 138 and reads out the lens distortion amount stored in the non-volatile memory 139b. Then, the arithmetic device 136 commands the image processing unit 137 to perform distortion process on the reference image. The image processing unit 137 generates a pseudo image reflecting the distortion from the read reference image and the lens distortion amount, temporarily stores the pseudo image in the volatile memory 138, and advances the process to step S506.

[0079] In step S506, the pseudo image generated in step S505 is displayed superimposed on the captured image. Note that the process itself is similar to the process in step S304 in Fig. 3, so the process content will not be described. The user aligns the camera 100 and the chart by observing and matching the superimposed image in which the pseudo image is superimposed on the captured image.

[0080] An example of a display screen during alignment will be described with reference to FIG. 7. FIGS. 7(a) to 7(c) show screens displayed on the display unit 140. FIG. 7(a) shows a state in which the simple alignment in steps S203 and S204 in FIG. 2 has been completed. FIG. 7(b) shows a state in which a pseudo image 712 is generated by distorting the reference image 711 and is displayed superimposed on the captured image 710 in FIG. 7(a) after the reference image 711 is superimposed on the captured image 710. FIG. 7(b) shows an example in which the reference image and the pseudo image are slightly misaligned. From this state, the position and attitude of the camera 100 are changed by the user so that the captured image and the pseudo image match as closely as possible until the alignment is completed as shown in FIG. 7(c).

[0081] During the alignment process, the amount of lens distortion is always estimated and then the pseudo image is generated. Therefore, it is assumed that the closer the positional relationship between the camera 100 and the chart is to a direct facing state, the higher the accuracy of estimating the amount of distortion and the smaller the degree of discrepancy between the captured image and the pseudo image becomes. Also, unlike the simple alignment process, it is possible to check whether the captured image and the pseudo image match up to the peripheral image height, so that the user can operate the operation unit 140 to enlarge the image position to be checked and check the degree of match. As a result, even with a lens with large distortion, it is possible to accurately align the camera 100 and the chart by matching the captured image and the pseudo image up to the peripheral image height.

[0082] Next, in step S507, the arithmetic device 136 determines whether or not to continue the alignment. Note that this process is similar to the process in step S305 in FIG. 3, and therefore the description of the process content will be omitted. At this time, if completion or cancellation is not selected, the arithmetic device 136 returns the process to step S503. Then, the process from the capture of the captured image to the superimposed display of the pseudo image in steps S503 to S506 is continued until either completion or cancellation is selected, or until it is forcibly terminated via the operation unit 142. Also, if completion or cancellation is selected, the process proceeds to step S508.

[0083] Here, the processing in steps S508 to S510 is similar to the processing in steps S306 to S308 in FIG. 3, and therefore a description thereof will be omitted.

[0084] In step S511, the calculation device 136 temporarily stores the alignment result set in either step S509 or S510 in the volatile memory 138. In addition, the calculation device 136 stores the pseudo image at the time of completion of the alignment as chart information in the chart information storage unit 139a.

[0085] In step S512, the computing device 136 completes the registration process.

[0086] When capturing and saving an image for calculating the amount of lens distortion in step S207 in FIG. 2, various information such as the alignment result, lens information at the time of completion of alignment, the amount of lens distortion, chart information, etc. may be saved in storage medium 143 together with the image itself as the image itself or as information (metadata) associated with the image.

[0087] In the present embodiment, the display of the superimposed image is updated in accordance with the period of pseudo image generation from steps S503 to S506 of the positioning process. This is to avoid a difference between the captured image displayed and the captured image when the pseudo image is generated when the processing time from steps S504 to S506 until the generation of the superimposed image is longer than one period of the imaging cycle (reading cycle of the imaging element 131). However, when the time until the generation of the superimposed image described above is shorter than one period of the imaging cycle, the superimposed image can be generated and the display updated in accordance with the imaging cycle. In addition, when the period of display update can be slow, the pseudo image may be generated in accordance with the movement of the camera 100, and then the display may be updated. The movement of the camera 100 can be detected by detecting a change in the captured image, or by providing a gyro sensor or the like in the imaging control device 130 to detect a movement such as a change in posture.

[0088] In the present embodiment, the alignment process is performed after the simplified alignment process. However, if the lens distortion amount described above is stored in the imaging lens 110 or the imaging control device 130, or if the lens distortion amount can be estimated without center alignment (the chart is included in the entire angle of view), the alignment process may be performed first.

[0089] As described above, in this embodiment, the image capture control device stores information about the chart that is the subject, and generates a pseudo image that has been subjected to distortion processing based on the information about the amount of lens distortion and the chart information. By superimposing the pseudo image on the captured image, it becomes possible to align the image capture control device and the chart with high accuracy and efficiency.

[0090] <Second embodiment> In the first embodiment, a method for accurately and efficiently aligning the imaging control device and the chart is described, in which a pseudo image that has been subjected to distortion processing is generated from chart information and lens distortion amount information stored in the imaging control device, and the pseudo image is displayed superimposed on the captured image. In the second embodiment, a method for accurately aligning the chart is described, in which a deviation degree between the captured image and the pseudo image is calculated, and information and instructions according to the calculated deviation degree are further superimposed on an image in which the pseudo image is superimposed on the captured image.

[0091] The outline of the process of the second embodiment is similar to the process of Fig. 2 described in the first embodiment, and therefore the description will be omitted. In the second embodiment, the registration process in step S205 in Fig. 2 is different from that in the first embodiment, and therefore the details of the process will be described.

[0092] First, the alignment process in the second embodiment will be described with reference to Fig. 8. Note that the processes in steps S801 to S805 are similar to the processes in steps S501 to S505 in Fig. 5 of the first embodiment, and the processes in steps S808 to S813 are similar to the processes in steps S507 to S512, so their description will be omitted.

[0093] The arithmetic unit 136 executes the processes in steps S801 to S805, and temporarily stores the captured image and the pseudo image in the volatile memory 138.

[0094] In step S806, the calculation device 136 calculates the degree of deviation from the captured image and the pseudo image. An example of a method for calculating the degree of deviation will be described later with reference to Fig. 9. Next, the calculation device 136 temporarily stores an evaluation value based on the calculated degree of deviation in the volatile memory 138.

[0095] In step S807, the arithmetic device 136 performs a process of superimposing a pseudo image on the captured image using the display unit 140 and the display control unit 141. This superimposing process is similar to the process of step S506 in FIG. 5 of the first embodiment, and therefore a description thereof will be omitted.

[0096] Next, the arithmetic device 136 generates a GUI (Graphical User Interface) serving as an indicator for encouraging the user to change the position and attitude of the camera 100 in order to match the captured image with the pseudo image, based on the evaluation value based on the deviation calculated in step S806, and further superimposes and displays the GUI on the superimposed image. An example of the display of an indicator for moving the camera 100 will be described later with reference to FIG.

[0097] In addition, in this embodiment, the deviation is represented by the distance and direction (vector) between the four corners (feature points) of each grid in the captured image and the pseudo image, but it may also be represented by the distance and direction between the center of gravity points of each grid (specific area) in the captured image and the pseudo image.

[0098] In step S808, if the user selects to complete or cancel the alignment, the arithmetic unit 136 advances the process to step S809 and executes the processes from step S809 to step S812.

[0099] Then, in step S813, the calculation device 136 ends the alignment process.

[0100] Next, an example of a method for calculating the deviation will be described with reference to Fig. 9. Here, the distance and direction (vector) between the positions of the four corners of each grid part of the captured image and the pseudo image are used. Also, the information (number) for identifying each grid and the positions of the four corners of each grid is the same as that in Fig. 6(a).

[0101] 9(a) is a diagram for explaining the calculation of the deviation degree in the upper left lattice (lattice number: g=0). The coordinates of the four corners of the lattice section 901 on the captured image side are upper left (Cxi0j0, Cyi0j0), upper right (Cxi1j0, Cyi1j0), lower left (Cxi0j1, Cyi0j1), and lower right (Cxi1j1, Cyi1j1). The coordinates of the four corners of the lattice section 902 on the pseudo image side are upper left (Rxi0j0, Ryi0j0), upper right (Rxi1j0, Ryi1j0), lower left (Rxi0j1, Ryi0j1), and lower right (Rxi1j1, Ryi1j1). The following process will be explained using the upper left as an example.

[0102] The distance and direction (vector) Vi0j0 between the upper left corner of the captured image grid 901 and the pseudo image grid 902 is calculated. At this time, Vi0j0 is expressed as (Rxi0j0-Cxi0j0, Ryi0j0-Cyi0j0). Next, the deviations of the upper right Vi1j0, the lower left Vi0j1, and the lower right Vi1j1 are calculated. Then, the deviation of the grid (grid number: g=0) is expressed as Vddg0=(Vi0j0+Vi1j0+Vi0j1+Vi1j1). This deviation calculation is performed for all grid parts or for either black or white grid parts. Next, the calculation device 136 temporarily stores the calculated deviations for each grid in the volatile memory 138.

[0103] Next, the arithmetic device 136 calculates an evaluation value based on the deviation and displays a movement index of the camera 100. An example of the calculation of the evaluation value based on the deviation will be described. For example, the sum of the deviations VddgN (N=0 to 186) of each lattice described above is calculated. In FIG. 9(b) and FIG. 9(c), there is no deviation between the captured image and the pseudo image. In FIG. 9(d) and FIG. 9(e), the pseudo image is shifted downward or to the right with respect to the captured image. Alternatively, the camera 100 may be divided into four quadrants based on the center of the optical axis, the sum is calculated for each quadrant, and the difference or sum between the quadrants may be calculated.

[0104] An example of the movement indicator display will be described with reference to Fig. 10. Fig. 10(a) shows an example in which a message indicating the movement direction of the camera 100 is further superimposed on the superimposed image. The user changes the position of the camera 100 according to the message, and fine-tunes the position until the message disappears or a message indicating completion is displayed.

[0105] 10(b) shows an example of displaying a graphical figure indicating the movement direction of the camera 100. An arrow mark indicating the movement direction is displayed, but the length and size of the arrow may be changed depending on the value of the amount of movement.

[0106] Fig. 10(c) shows an example in which a bar gauge is used to guide the movement direction and the approximate movement amount of the camera 100. Bar gauges 1003 to 1006 in Fig. 10(c) are user interfaces for guiding the movement of the camera 100. Bar gauges 1003 and 1004 indicate the horizontal direction, and bar gauges 1005 and 1006 indicate the vertical direction.

[0107] The deviations of the four quadrants described above are used as evaluation values ​​for displaying the bar gauges. The bar gauge 1003 in the horizontal direction and at the top of the image indicates the difference between the deviations of the first quadrant and the fourth quadrant, and the bar gauge 1004 in the horizontal direction and at the bottom of the image indicates the difference between the deviations of the second quadrant and the third quadrant. Similarly, the bar gauge 1005 in the vertical direction and at the left side of the image indicates the difference between the deviations of the third quadrant and the fourth quadrant, and the bar gauge 1006 in the vertical direction and at the right side of the image indicates the difference between the deviations of the first quadrant and the second quadrant.

[0108] The black triangle indicates the current state, and in Fig. 10(c) the vertical position is correct, but the horizontal position is shifted to the left. While observing the bar gauges 1003-1006, the user changes the position of the camera 100 and makes fine adjustments so that the bar gauges 1003-1006 are all in the same state. Note that the display examples of the moving indicators in Figs. 10(a)-10(c) are merely examples, and the present invention is not limited to these.

[0109] In this embodiment, whether or not the alignment is complete is determined by the user's visual confirmation and judgment. Therefore, the above-mentioned deviation degree is used to determine whether or not the alignment is complete. For example, when the evaluation value based on the deviation degree becomes equal to or less than a predetermined value, it may be determined that the alignment is complete, and the alignment process may be terminated after performing necessary termination processing.

[0110] In the outline of the calibration process described in steps S201 to S207, an image for calculating the amount of lens distortion is captured after the alignment is completed. However, when it is determined that the alignment is completed using the deviation degree described above, an image for calculating the amount of lens distortion may be automatically captured.

[0111] In the first embodiment, the lens distortion amount is stored in the imaging lens 110 or the imaging control device 130. However, although these lens distortion amounts are values ​​corresponding to each lens position, they may be discrete values ​​due to limitations such as the storage capacity of the imaging lens 110 or the imaging control device 130. In that case, depending on the lens position, it is assumed that even a pseudo image subjected to distortion processing may have a large deviation from the captured image. In this case, the process of estimating the lens distortion amount from the captured image and chart information described in the first embodiment may be executed in parallel. In that case, for example, the degree of deviation between the captured image and each of the pseudo image generated from the lens distortion amount stored in the imaging device 110 or the imaging control device 130 and the pseudo image generated from the estimated lens distortion amount is calculated. Then, the pseudo image with the smaller absolute value of the deviation is displayed.

[0112] In the above description, in the process of calculating the deviation from the captured image and the pseudo image in step S806, the deviation is calculated for the entire chart grid portion of the captured image. However, the lens distortion amount tends to increase from the optical center to the peripheral image height (high image height). Therefore, the area for calculating the deviation may be limited according to the lens distortion amount and image height. As a result, it is possible to reduce the load of the calculation process.

[0113] In addition, the pseudo image used in the processing of step S806 is an image that has been subjected to distortion processing, but since the evaluation value based on the degree of deviation is determined from the relative difference in the degree of deviation, the degree of deviation may be calculated from the captured image and a reference image that has not been subjected to distortion processing.

[0114] In addition, in the process of superimposing a pseudo image on a captured image in step S807, the contents of the superimposing process may be changed according to the degree of deviation calculated in step S806. For example, it is possible to change the transparency, change the color, or change the display method according to the degree of deviation. Specifically, in the case of changing the color, if the black part of the pseudo image corresponding to the black of the chart grid part of the captured image is red, when the degree of deviation is equal to or less than a predetermined value, the color is changed to a color that is symmetrical to the displayed color and is easily visible. In this case, the red is changed to a blue-based color (blue or cyan). By doing so, the position of the part with a large degree of deviation becomes easier to understand, making it easier to perform positioning. In addition, in the case of changing the display method, for example, when the evaluation value based on the degree of deviation meets a predetermined value, only the frame part of the grid part of the pseudo image is displayed. By doing so, the position of the part with a large degree of deviation becomes easier to understand, as in the case of changing the color.

[0115] As described above, in this embodiment, the deviation degree is calculated from the captured image and the pseudo image, and an index based on the deviation degree is further superimposed on the image in which the pseudo image is superimposed on the captured image, thereby making it possible to more accurately align the image capture device and the chart.

[0116] <Third embodiment> Next, we will explain the problems that arise when visually aligning the chart and the camera in the calibration process, and a method to reduce the effort by instructing the user on the amount and direction to move the camera and performing final fine adjustments with the camera.

[0117] When aligning the chart and the camera visually, the user may not know how far to move the camera until the misaligned state is aligned. Therefore, the user must move the camera while visually checking whether the displayed chart displayed on the LCD matches the photographed chart. Even if the user judges that they match visually, there is a possibility that a minute misalignment that cannot be seen on the LCD remains. If a minute misalignment remains, it is considered that the lens data to be acquired in the calibration process cannot be acquired in a concentric circle with the center of the screen as the origin, and there is a possibility that accurate lens information cannot be acquired. Furthermore, if the user performs a finishing operation for the angle of view adjustment so as to eliminate even the smallest misalignment in order to perform accurate angle of view adjustment, the calibration work will be hindered by increasing the work time, etc.

[0118] In response to the above problem, a method of instructing the user on the amount and direction of camera movement for aligning the chart and camera, and performing final fine adjustments with the camera will be described with reference to Figs. 11 to 19.

[0119] 11 to 14 are flowcharts for explaining the process of aligning the cameras for the calibration process in this embodiment.

[0120] FIG. 11 shows the procedure of the angle of view adjustment process and the calibration process in the calibration process.

[0121] First, when the user switches the camera 100 to a calibration mode for acquiring lens data for VFX compositing via the operation unit 142, the process of FIG. 11 starts.

[0122] In step S2101, the arithmetic unit 136 performs a field of view adjustment process, and the process proceeds to step S2102. The field of view adjustment process will be described in detail later.

[0123] In step S2102, the calculation device 136 determines whether or not the angle-of-view adjustment process in step S2101 has been completed, and if it is determined that the angle-of-view adjustment process has been completed, ends the angle-of-view adjustment process and proceeds to step S2103.

[0124] Next, in step S2103, a calibration process is performed, and the process ends.

[0125] Next, the angle of view adjustment process in step S2101 will be described in detail with reference to FIG.

[0126] First, in step S2201, the arithmetic unit 136 initializes flag information for determining whether or not the angle of view adjustment process in step S2102 described above has been completed to an incomplete state.

[0127] In step S2202, the calculation device 136 reads out model data of the chart recorded in the chart information storage unit 139a. The model data corresponds to a pseudo image or a reference image generated from the chart information described in the first and second embodiments.

[0128] In step S2203, the arithmetic unit 136 calculates a difference indicating the amount of deviation between the model data of the chart read out in step S2202 and the image of the chart captured by the camera. Details of this process will be described later with reference to FIG.

[0129] In step S2204, the arithmetic unit 136 calculates the direction and amount in which the camera 100 should be moved from the amount of deviation calculated in step S2203. Details of this process will be described later with reference to FIG.

[0130] In step S2205, the calculation device 136 determines whether the amount of displacement calculated in step S2203 or the amount of movement G of the camera 100 calculated in step S2204 based on the amount of displacement is less than a threshold value. If it is determined that it is less than the threshold value, the process proceeds to step S2206, and if it is determined that it is equal to or greater than the threshold value, the process proceeds to step S2209.

[0131] Step S2206 is the case where the arithmetic device 136 determines in step S2205 that the amount of movement G of the camera 100 is less than the threshold. When the amount of movement G of the camera 100 is small, this is processing for the case where it is determined that further fine adjustment of the camera position relative to the chart is difficult through manual adjustment by the user. Therefore, the user is not prompted to move the camera, and centering of the angle of view is performed inside the camera.

[0132] Specifically, the arithmetic device 136 calculates the drive amount of the image stabilization lens 114 based on the difference calculated in step S2203 so that the difference between the chart and the captured image is minimized. Then, an instruction to drive the image stabilization lens 114 is output to the lens control unit 121 via the electrical contact unit 150. The lens control unit 121 that receives this instruction outputs a command to drive the image stabilization lens 114 to the image stabilization drive unit 118, and the image stabilization drive unit 118 actually drives the image stabilization lens 114. Alternatively, the arithmetic device 136 calculates the drive amount of the image stabilization drive unit 134 for the image sensor 131 based on the difference calculated in step S2203, and outputs a command to the image stabilization control unit 135 to drive the image sensor 131. Then, the image stabilization drive unit 134 actually drives the image sensor 131. Alternatively, both the image stabilization lens 114 and the image sensor 131 described above may be driven. When these two components are driven, it is desirable to drive them to positions where the change in data on the amount of distortion is the smallest, compared to a state in which each of the lenses 110 to 115 and the imaging element 131 is on the optical axis.

[0133] In step S2207, since the calculation device 136 performed centering of the angle of view inside the camera 100 in step S2206, the display control unit 141 displays on the display unit 140 a message indicating that the user's operation to adjust the angle of view has been completed.

[0134] In step S2208, the arithmetic unit 136 turns on a flag indicating that the angle of view adjustment has been completed, and ends the process. Note that this flag is used in step S2102 to determine whether the angle of view adjustment has been completed.

[0135] Step S2209 is a case where it is determined in step S2205 that the amount of deviation is equal to or greater than the threshold, and the user still has to adjust the angle of view. Therefore, the display control unit 141 displays on the display unit 140 a message indicating that the user has not yet completed the angle of view adjustment. As an example, a method can be considered in which, of the movement amount G of the camera 100 calculated in step S2204, the vertical movement amount, horizontal movement amount, and movement direction with respect to the chart are displayed.

[0136] With the above process, if the positional deviation of the camera 100 with respect to the chart is large, the user can move the camera after recognizing the amount and direction of movement. Furthermore, for minute deviations that the user cannot adjust, the user does not need to make adjustments, and the time required to adjust the angle of view for calibration processing is shortened.

[0137] Next, the process of calculating the difference between the chart and the captured image, which is described in step S2203, will be described in detail with reference to FIG. 13A.

[0138] First, in step S2301, the calculation device 136 issues an instruction to the image sensor control unit 133 to acquire an image signal obtained by photographing a chart from the image sensor 131 at a predetermined cycle, and sends the acquired image signal to the image processing unit 137. The image processing unit 137 performs appropriate image processing on the image signal, and temporarily stores it in the volatile memory 138.

[0139] In step S2302, the calculation device 136 instructs the image processing unit 137 to perform image processing such as binarization of the temporarily stored chart image so as to facilitate calculation of the difference in step S2304, which will be described later. Note that the processing of the image signal is not limited to binarization.

[0140] Next, in step S2303, the calculation device 136 performs shift processing of the photographed image on the chart image that has been subjected to the image processing.

[0141] Specifically, one example of a method for searching for the position where the difference with the chart model data read out in step S2202 is smallest is to calculate the position where the difference is smallest while changing the comparison position between the captured image and the chart model data. An image is generated in which the chart image is moved horizontally to the left by N pixels and vertically upward by N pixels, and the difference is calculated. Furthermore, if it is determined as a result of this process in step S2309 described later that the difference calculation for the specified position has not been completed, the image is moved vertically to the left by N-1 pixels, vertically upward by N pixels, and so on, differently from the previous time, and the image difference is calculated again.

[0142] The shift range, which indicates within what range the image is shifted, and the shift step amount, which indicates by what step amount (thinning amount) the image is shifted within the shift range, may be set in the camera or may be given by the user. When setting in the camera, a method is conceivable in which a large shift step amount is first set to search for a position where the difference is minimum, and then a smaller shift step amount is set to search for the position where the difference is minimum. In step S2303, the image processing unit 137 changes the above-mentioned shift range by the set shift step amount each time step S2303 is executed, cuts out the image, and outputs it to the calculation device 136.

[0143] As another example of searching for the position where the difference with the model data of the chart read out in step S2202 is smallest, the following method can be considered. That is, the image captured while changing the position of the image stabilization lens 114 is compared with the model data of the chart each time, and the position of the image stabilization lens 114 where the difference is smallest is calculated. A specific method will be described later with reference to FIG.

[0144] Furthermore, as another example of searching for the position where the difference with the model data of the chart read out in step S2202 is the smallest, the following method can be considered. That is, the image captured while changing the position of the image sensor 131 is compared with the model data of the chart each time, and the position of the image sensor 131 where the difference is the smallest is calculated. A specific method will be described later with reference to FIG.

[0145] Furthermore, when implementing a method of calculating the minimum position while changing the comparison position between the captured image and the chart model data, it is possible to compare the image not only by changing the shift position in the horizontal and vertical directions but also by changing the position in the rotational direction.

[0146] The above-mentioned method of changing the comparison position and rotation amount r between the captured image and the chart model data, the method of changing the position of the image stabilization lens 114, and the method of changing the position of the image sensor 131 may be implemented by using only one of them or by combining two or more of them.

[0147] Next, in step S2304, the calculation device 136 compares the image captured and generated in step S2303 with the model data of the chart, and calculates the difference S. An example of a method for calculating the difference S will be described with reference to FIG.

[0148] 19, 2900 indicates an area representing the angle of view, 2901 indicates chart model data, and 2902 indicates the area of ​​the chart in the photographed image. The area indicated by 2910 (area where the right-upward hatching and the right-downward hatching overlap) is an area where the black parts of the chart model data 2901 and the black parts of the photographed image 2902 overlap, and is calculated as an area where there is no difference. Similarly, the area indicated by 2911 (blank area) is an area where the white parts of the chart model data 2901 and the white parts of the photographed image 2902 overlap, and is calculated as an area where there is no difference. The area indicated by 2912 (area with only right-downward hatching) is an area where the black parts of the chart model data 2901 and the white parts of the photographed image 2902 overlap, and is calculated as an area where there is no difference. Similarly, the area indicated by 2913 (area with only upward-sloping hatching) is an area where the white parts of the chart model data 2901 and the black parts of the photographed image 2902 overlap, and is the area that is calculated as the difference area.

[0149] In the above explanation, the difference area is calculated based on whether or not the areas are the same color. However, it is also possible to use a method in which an identifier such as an ID is assigned to each grid area of ​​the chart, and the difference area is calculated based on whether or not areas of the same color with the same ID overlap.

[0150] Next, in step S2305, the calculation device 136 compares the area of ​​the differential region calculated by the above-mentioned method with the stored minimum difference value. If the difference S is smaller than the minimum difference value, the process proceeds to step S2306, and if the difference S is equal to or larger than the minimum difference value, the process proceeds to step S2309.

[0151] In step S2306, the calculation device 136 re-stores the difference S as the minimum difference value.

[0152] In step S2307, the calculation device 136 stores the shift position when the difference S is minimized. Specifically, the pixel movement amount p of the chart image, the driving amount ω of the image stabilization lens 114, and the movement amount i of the image sensor 131 when the difference is minimized are stored as shift amounts.

[0153] In step S2308, the calculation device 136 instructs the display control unit 141 to perform a synthesis process on the chart model data and the captured image, and displays the result on the display unit 140. The user visually checks the misalignment between the chart model data displayed on the display unit 140 and the captured image, and moves the camera so as to correct the misalignment.

[0154] In step S2309, the arithmetic unit 136 judges whether or not all of the search locations to be searched in step S2303 have been searched. If it is judged that all of the search locations have been searched, the process ends, and if it is judged that there are still search locations to be searched, the process returns to step S2301.

[0155] By the above processing, it is possible to calculate the amount of deviation for determining the difference between the chart and the camera from the difference between the chart model data and the photographed chart image.

[0156] Next, the camera movement amount calculation process described in step S2204 will be described in detail with reference to FIG.

[0157] First, in step S2401, the arithmetic device 136 transmits and receives commands to and from the lens control unit 121, and acquires lens focal length information f and a drive amount ω of the image stabilization lens 114. In addition, it acquires position information of the focus lens 115, information related to the drive states of various lenses, and information related to the state of the aperture 113.

[0158] In step S2402, the calculation device 136 acquires subject distance information d. The subject distance information d may be calculated from the position information of the focus lens 115 described above after focusing on the chart, or the subject distance information d input by the user may be acquired. Alternatively, if the image sensor 131 is an image sensor capable of calculating the defocus amount of the image plane phase difference method, the subject distance information d may be calculated from the position information of the focus lens 115 described above and the defocus amount.

[0159] In step S2403, the calculation device 136 calculates the movement amount G and rotation amount, which are the amount by which the camera 100 should be moved, using the acquired lens focal length information f and subject distance information d, the pixel movement amount p and rotation amount r calculated in the angle of view adjustment process of FIG. 12, the drive amount ω of the image stabilization lens 114, and the movement amount i of the image sensor 131, and then ends the process.

[0160] Next, an example of calculating the movement amount G of camera 100 in the shift process of the captured image in step S2303 in Fig. 13A will be described. Fig. 15 shows a case where the shift process is performed by virtual image movement inside camera 100 and the movement amount G is calculated. Fig. 16 shows a case where the shift process is performed by image stabilization lens 114 and the movement amount G is calculated. Fig. 17 shows a case where the shift process is performed by image sensor 131 and the movement amount G is calculated. Note that in the drawings, the same or similar components are given the same reference numbers and duplicate explanations will be omitted.

[0161] 15(a), 2500 denotes a chart used for calibration, and point O denotes the center position of the chart. 2501 denotes the optical axis of the camera 100. 2502 denotes the angle of view of the camera 100. One of the purposes of this embodiment is to calculate the deviation distance G between point O and the optical axis 2501 and present the deviation distance (amount of movement) G to the user.

[0162] FIG. 15(b) shows the display on the LCD screen when shooting in the situation of FIG. 15(a).

[0163] Model data 2901 of a chart to which an image is to be aligned is displayed in the center of the screen with respect to an angle of view 2900. Also, in Fig. 15(a), the camera 100 is slightly shifted to the right with respect to the chart 2500, so in Fig. 15(b), the chart 2902 in the captured image is composited with a slight shift to the left.

[0164] Fig. 15(c) is a diagram showing an example of the shift process of the captured image performed each time in step S2303. If the captured image is not shifted, the shift will be as shown in Fig. 15(b), but if the captured image is virtually shifted to the upper left in the first process of step S2303 and combined with the chart model data, the shift will be as shown in the upper left diagram of Fig. 15(c). In this state, the difference is calculated and compared with the minimum difference value, and the minimum difference value and the shift amount that results in the minimum difference are updated as necessary.

[0165] Next, in the second process, the captured image is virtually shifted to the upper right and combined with the chart model data, resulting in a shift as shown in the upper right diagram of Fig. 15(c). In this state, the difference is calculated and compared with the minimum difference value, and the minimum difference value and the shift amount that results in the minimum difference are updated as necessary.

[0166] Next, in the third process, the captured image is virtually shifted downward and to the left and combined with the chart model data, resulting in a shift as shown in the lower left diagram in Fig. 15(c). In this state, the difference is calculated and compared with the minimum difference value, and the minimum difference value and the shift amount that results in the minimum difference are updated as necessary.

[0167] Next, in the fourth process, the captured image is virtually shifted to the lower right and combined with the chart model data, resulting in a shift as shown in the lower right diagram in Fig. 15(c). In this state, the difference is calculated and compared with the minimum difference value, and the minimum difference value and the shift amount that results in the minimum difference are updated as necessary.

[0168] After performing the above process, the pixel movement amount corresponding to the recorded minimum difference value is set as p, and the deviation distance G is calculated. The deviation distance G can be calculated using the following formula based on the relationship in FIG.

[0169] G = (d × p) / f Next, a method for calculating the deviation distance G by driving the image stabilization lens 114 will be described with reference to FIG.

[0170] 16(a), 2503 indicates the optical axis when the image stabilization lens 114 moves. As an example, the optical axis 2503 passes through point O, and the correction angle at that time is ωdeg. Also, 2504 is a line indicating the angle of view after the change when the image stabilization lens 114 moves.

[0171] Figure 16(b) shows how image stabilization lens 114 is driven when shooting in the situation of Figure 16(a). The left diagram of Figure 16(b) shows a situation in which image stabilization lens 114 holds the center 2505 of the optical axis. Meanwhile, the right diagram of Figure 16(b) shows a situation in which image stabilization lens 114 is shifted to the left by ωdeg from the center 2505 of the optical axis. The bottom diagram of Figure 16(b) shows the display on the LCD screen when shooting in the above situation.

[0172] Chart model data 2901, which is a target for aligning the image with respect to the angle of view 2900, is displayed in the center of the screen. Also, in the left diagrams of Figures 16(a) and 16(b), the camera 100 is slightly shifted to the right with respect to the chart 2500, so the chart 2902 in the captured image is composited with a slight shift to the left. On the other hand, in the right diagrams of Figures 16(a) and 16(b), the optical axis 2503 of the camera 100 passes through point O, so the centers of the chart 2902 in the captured image and the chart model data 2901 coincide with each other.

[0173] Fig. 16(c) is a diagram showing an example of the shift process of the captured image performed each time in step S2303. If the image stabilization lens 114 is not shifted, the shift will be as shown in the left diagram of Fig. 16(b), but if the captured image is shifted to the upper left in the first process of step S2303 and combined with the chart model data, the shift will be as shown in the upper left diagram of Fig. 16(c). In this state, the difference is calculated and compared with the minimum difference value, and the shift amount of the image stabilization lens 114 that results in the minimum difference is updated as necessary.

[0174] Next, in the second process, the captured image is shifted to the upper right and combined with the chart model data, resulting in a shift as shown in the upper right diagram of Fig. 16(c). In this state, the difference is calculated and compared with the minimum difference value, and the shift amount of the image stabilization lens 114 that results in the minimum difference is updated as necessary.

[0175] Next, in the third process, the captured image is shifted downward and to the left and combined with the chart model data, resulting in a shift as shown in the lower left diagram in Fig. 16(c). In this state, the difference is calculated and compared with the minimum difference value, and the shift amount of the image stabilization lens 114 that results in the minimum difference is updated as necessary.

[0176] Next, in the fourth process, the captured image is shifted to the lower right and combined with the chart model data, resulting in a shift as shown in the lower right diagram in Fig. 16(c). In this state, the difference is calculated and compared with the minimum difference value, and the shift amount of the image stabilization lens 114 that results in the minimum difference is updated as necessary.

[0177] The above process is performed, and the drive amount of the image stabilization lens 114 corresponding to the recorded minimum difference value is set as ω, and the deviation distance G is calculated. The deviation distance G can be calculated using the following formula from the relationship in FIG.

[0178] G = d tan ω In the above-mentioned method of driving the image stabilization lens 114, unlike the above-mentioned method of virtually shifting the captured image inside the camera, the image actually captured moves in conjunction with the driving of the image stabilization lens 114. If the captured image moves, it may be difficult for the user to center the camera when trying to center the camera on the chart, even if the movement direction and movement distance are displayed. Therefore, when the image stabilization lens 114 is driven to calculate the difference, in step S2308, the synthesis process is performed only on the image when the image stabilization lens 114 is at the optical center. Alternatively, the synthesis process is performed with an image in which the captured image is virtually returned by an amount equivalent to the shift process of the image stabilization lens 114, thereby preventing the user from having difficulty in aligning the camera.

[0179] Next, a method for calculating the deviation distance G by driving the imaging element 131 will be described with reference to FIG.

[0180] 17(a), 2506 indicates the optical axis when the image sensor 131 moves. As an example, 2506 illustrates a pattern in which the image sensor 131 passes through point O and the correction amount of the image sensor 131 at that time is imm. Also, 2507 indicates the angle of view after the change when the image sensor 131 moves.

[0181] Fig. 17(b) shows how the image sensor 131 is driven when shooting in the situation of Fig. 17(a). The left diagram of Fig. 17(b) shows a situation in which the image sensor 131 holds the center 2508 of the optical axis. On the other hand, the right diagram of Fig. 17(b) shows a situation in which the image sensor 131 is shifted by imm to the right from the center 2501 of the optical axis. The bottom diagram of Fig. 17(b) shows the display on the LCD screen when shooting in the above situation.

[0182] Chart model data 2901, which is a target for aligning the image with respect to the angle of view 2900, is displayed in the center of the screen. Also, in the left diagrams of Figures 17(a) and 17(b), the camera 100 is slightly shifted to the right with respect to the chart 2500, so the chart 2902 in the captured image is composited with a slight shift to the left. On the other hand, in the right diagrams of Figures 17(a) and 17(b), the optical axis 2506 of the camera 100 passes through point O, so the centers of the chart 2902 in the captured image and the chart model data 2901 coincide with each other.

[0183] Fig. 17(c) is a diagram showing an example of the captured image shift process performed each time in step S2303. If the image sensor 131 is not shifted, the shift will be as shown in the left diagram of Fig. 17(b), but if the captured image is shifted to the upper left in the first process of step S2303 and combined with the chart model data, the shift will be as shown in the upper left diagram of Fig. 17(c). In this state, the difference is calculated and compared with the minimum difference value, and the shift amount of the image sensor 131 that results in the minimum difference is updated as necessary.

[0184] Next, in the second process, the captured image is shifted to the upper right and combined with the chart model data, resulting in a shift as shown in the upper right diagram of Fig. 17(c). In this state, the difference is calculated and compared with the minimum difference value, and the shift amount of the image sensor 131 that results in the minimum difference is updated as necessary.

[0185] Next, in the third process, the captured image is shifted downward and to the left and combined with the chart model data, resulting in a shift as shown in the lower left diagram in Fig. 17(c). In this state, the difference is calculated and compared with the minimum difference value, and the shift amount of the image sensor 131 that results in the minimum difference is updated as necessary.

[0186] Next, in the fourth process, the captured image is shifted to the lower right and combined with the chart model data, resulting in a shift as shown in the lower right diagram in Fig. 17(c). In this state, the difference is calculated and compared with the minimum difference value, and the shift amount of the image sensor 131 that results in the minimum difference is updated as necessary.

[0187] The above process is performed, and the drive amount of the image sensor 131 corresponding to the recorded minimum difference value is defined as i to calculate the deviation distance G. The deviation distance G can be calculated from the relationship in FIG.

[0188] G = (d × i) / f In the above-mentioned method of driving the image sensor 131, unlike the above-mentioned method of virtually shifting the captured image inside the camera, the image that is actually captured moves in conjunction with the driving of the image sensor 131. If the captured image moves, it may be difficult for the user to center the camera when trying to center the camera on the chart, even if the movement direction and movement distance are displayed. Therefore, when the image sensor 131 is driven to calculate the difference, in step S2308, the composition process is performed only on the image when the image sensor 131 is at the optical center. Alternatively, the composition process is performed with an image in which the captured image is virtually returned by an amount equivalent to the amount of shift processing of the image sensor 131, thereby preventing the user from having difficulty in aligning the camera.

[0189] By carrying out the above process, the distance G between the camera and the center of the chart can be obtained, and the amount of movement of the camera can be notified to the user. This makes it possible to reduce the effort required for adjusting the angle of view for calibration processing.

[0190] In the above description, the number of shifts is four for ease of understanding, but in reality the number of shifts is more than four.

[0191] Next, an example of a method of displaying the amount of movement G (the amount of movement in the vertical and horizontal directions and the movement direction) of the camera 100 until the user completes the angle of view adjustment in steps S2209 and S2207 will be described with reference to FIG.

[0192] FIG. 18(a) shows the display on the display unit 140 when it is determined that the amount of deviation between the camera 100 and the calibration chart is equal to or greater than a threshold value.

[0193] In the figure, 2800 indicates an area representing the angle of view, 2801 indicates model data of the chart, and 2802 indicates the area of ​​the chart in the captured image. In Fig. 18(a), since the amount of deviation between the camera 100 and the chart is equal to or greater than a threshold, the chart 2802 in the captured image is displayed at a position that is shifted from the chart model data 2801. Also, 2803 indicates an example of a display that instructs the user on the amount and direction of movement of the camera when it is determined that the amount of deviation between the camera 100 and the captured chart is equal to or greater than a threshold (above a predetermined value). The display 2803 indicates that moving the camera 54 mm to the right and 23 mm downward will bring the center of the chart model data 2801 into alignment with the center of the position of the chart 2802 in the captured image.

[0194] FIG. 18(b) shows the display on the display unit 140 when it is determined that the amount of misalignment between the camera 100 and the chart is less than the threshold value.

[0195] Display 2804 in Fig. 18(b) displays that there is no longer any misalignment between the camera 100 and the captured chart. The user stops moving the camera position based on display 2804. However, since it is rare for the misalignment between the camera 100 and the captured chart to become completely zero, as described above in step S2206, the image stabilization lens 114 and the image sensor 131 are driven (shifted) to positions where the difference is minimized. This minimizes the difference between the chart model data 2801 and the chart area 2802 in the captured image, allowing the user to complete the alignment of the camera 100 without having to perform minute alignment.

[0196] By using the technique described above, it is easier for the user to know in which direction and by how much to move the camera compared to aligning the chart and camera visually, thereby reducing the amount of time required for alignment.

[0197] Furthermore, even if minute misalignment that cannot be seen with the naked eye remains on the LCD, the misalignment can be corrected by driving the image stabilization lens 114 and the image sensor 131, thereby achieving a further reduction in work time.

[0198] The chart pattern is not limited to the format described above, and may be any design that is represented at a predetermined interval in the horizontal and vertical directions of the chart. For example, it may be a checkerboard pattern in which black and white are reversed in adjacent rectangles among rectangular areas divided into a grid, or a barcode-like pattern.

[0199] Furthermore, as described above in the first embodiment, the chart model data may be distorted based on the amount of distortion and the amount of tilt calculated provisionally from the captured image, and then displayed or the difference may be calculated.

[0200] Furthermore, the magnification rate of the chart model data may be changed according to the focal length for display or difference calculation.

[0201] <Fourth embodiment> In the process of calculating the difference between the chart and the captured image in Fig. 13A, a case will be described with reference to Fig. 13B where the amount of deviation of the camera 100 with respect to the chart 2500 is large and the amount of movement G of the camera cannot be calculated by only one of the methods of virtual movement of the image, movement of the image stabilization lens 114, and movement of the image sensor 131. Note that in Fig. 13B, the same processes as those in Fig. 13A are assigned the same step numbers and their explanations will be omitted.

[0202] If the result of executing steps S2301 to S2309 shows that the shift position with the minimum difference is located at the edge of the range in which the search was executed, there is a possibility that the shift position with the minimum difference is located outside the search range. Therefore, in order to expand the search range, the following pattern is used for processing.

[0203] In the captured image shifting process of step S2303, if the shifting process is performed by moving the image, the search range is expanded by driving the position of either the image stabilization lens 114 or the image sensor 131 so that the image is shifted in the direction that minimizes the difference.

[0204] Furthermore, in the captured image shifting process in step S2303, if shifting process is performed by the image stabilization lens 114, the search range is expanded by driving the image sensor 131 so that the image is shifted in the direction that minimizes the difference.

[0205] Furthermore, in the captured image shifting process of step S2303, if shifting process is performed by the image sensor 131, the search range is expanded by driving the image stabilization lens 114 so that the image is shifted in the direction that minimizes the difference.

[0206] 13B, and in step S2322, either the image stabilization lens 114 or the image sensor 131 is driven and fixed in position according to the pattern described above. Then, the process returns to step S2301, and the shift position that minimizes the difference is calculated again.

[0207] When the shift process of the captured image involves image movement, the search range may be further expanded by driving both the image stabilization lens 114 and the image sensor 131.

[0208] With the above method, even if there is a large misalignment between the camera 100 and the calibration chart 2500, it is possible to calculate and present the direction and amount that the camera 100 should move, thereby reducing the labor required for the user to adjust the angle of view.

[0209] (Other embodiments) The present invention can also be realized by a process in which a program for realizing one or more functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for realizing one or more functions.

[0210] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0211] 100: imaging device (camera), 110: imaging lens, 114: image stabilization lens, 121: lens control unit, 130: imaging control device, 131: imaging element, 132: imaging signal processing unit, 133: imaging element control unit, 134: image stabilization drive unit, 136: calculation unit (CPU), 137: image processing unit, 138: volatile memory (RAM), 139: non-volatile memory (ROM), 140: display unit, 141: display control unit, 142: operation unit, 143: storage medium (memory card), 150: electrical contact unit

Claims

1. An information processing device provided in an imaging device, a first acquisition means for acquiring a captured image from an imaging means in the imaging device to which the photographing lens is attached, the captured image being obtained by photographing a chart having a grid pattern for correcting distortion aberration of the photographing lens with the imaging means; a second acquisition means for acquiring a reference image which is an image serving as a reference for the chart; a third acquisition means for acquiring information on distortion of the photographing lens; a first generating means for generating a pseudo image, which is an image in which the distortion aberration is reflected in the reference image, based on the reference image and information on the distortion aberration; a second generating means for generating a composite image by combining the photographed image and the pseudo image; a calculation means for calculating a degree of deviation of an image based on the photographed image and the pseudo image, and further calculating an evaluation value based on the degree of deviation; a third generating means for generating, based on the evaluation value, a movement index for the imaging device for aligning a position of the captured image with a position of the pseudo image, the movement index being used by a user to adjust a position and an attitude of the imaging device relative to the chart by moving the imaging device; a display control means for displaying the moving indicator in a superimposed manner on the composite image; An information processing device comprising:

2. 2. The information processing apparatus according to claim 1, further comprising output means for outputting the composite image to display means.

3. 3. The information processing apparatus according to claim 2, further comprising the display means.

4. 4. The information processing device according to claim 1, wherein the distortion information includes at least one of information on distortion estimated from the captured image and the reference image, information on distortion stored in a storage means in the photographing lens, and information on distortion stored in the imaging device to which the photographing lens is attached.

5. 5. The information processing apparatus according to claim 1, wherein the first generating means enlarges or reduces the reference image in accordance with a focal length of the photographing lens.

6. An information processing device as claimed in any one of claims 1 to 5, characterized in that the deviation includes at least one of a vector indicating the distance and direction between a feature point of the captured image and a paired feature point of the pseudo image, and a vector indicating the distance and direction between the center of gravity of a specific area of ​​the captured image and the center of gravity of a specific area that is a paired area of ​​the pseudo image.

7. 7. The information processing device according to claim 1, wherein the calculation means limits a range for calculating the degree of deviation in the captured image and the pseudo image depending on an amount of distortion aberration of the photographing lens or an image height of the captured image.

8. 8. The information processing apparatus according to claim 1, wherein the second generating means changes a color or a form of a part of the pseudo image to be combined with the photographed image in accordance with the degree of deviation.

9. 9. The information processing apparatus according to claim 1, further comprising a storage unit that stores the information on the photographing lens, the information on the distortion aberration of the photographing lens, the deviation degree, and the evaluation value in association with the photographed image.

10. 10. The information processing apparatus according to claim 1, further comprising a control unit that controls the apparatus so as to automatically capture an image when the evaluation value satisfies a predetermined condition.

11. 11. The information processing device according to claim 1, further comprising a fourth generating means for generating the chart information, the fourth generating means generating the chart information by detecting information necessary for generating the chart information from the captured image.

12. a second calculation means for calculating a difference between the captured image and the pseudo image; a control means for controlling a display means to display an amount of movement of the imaging device to which the photographing lens is attached when a difference between the photographed image and the pseudo image is equal to or greater than a predetermined value, and for causing a shift means for performing a shift operation to relatively shift the photographed image and the pseudo image to perform the shift operation so as to reduce the difference when the difference is smaller than the predetermined value; The information processing apparatus according to claim 1 , further comprising:

13. 13. The information processing device according to claim 12, wherein the shifting means includes at least one of a means for performing the shifting operation by driving an image stabilization lens provided in the photographing lens, and a means for performing the shifting operation by driving an image sensor that captures the photographed image.

14. 14. The information processing apparatus according to claim 12, wherein the second calculation means further calculates a relative rotation amount between the photographed image and the pseudo image.

15. The information processing device according to claim 13, characterized in that the control means controls the second generation means to generate the composite image based on the captured image before the shift operation is performed, when the shift operation is performed by driving the image stabilization lens or when the shift operation is performed by driving the image sensor.

16. The photographing lens; an imaging element for capturing the photographed image; An information processing device according to any one of claims 1 to 15; An imaging device comprising:

17. a first acquisition step of acquiring a captured image from an imaging means in an imaging device to which a photographing lens is attached by photographing a chart having a grid-like pattern for correcting distortion aberration of the photographing lens; a second acquisition step of acquiring a reference image that is an image serving as a reference for the chart; a third acquisition step of acquiring information on distortion aberration of the photographing lens; a first generation step of generating a pseudo image based on the reference image and information on the distortion; a second generation step of generating a composite image by combining the captured image and the pseudo image; a calculation step of calculating a degree of deviation of an image based on the captured image and the pseudo image, and further calculating an evaluation value based on the degree of deviation; a third generation step of generating, based on the evaluation value, a movement indicator for the imaging device for aligning a position of the captured image with a position of the pseudo image, the movement indicator being used by a user to adjust a position and an attitude of the imaging device relative to the chart by moving the imaging device; a display control step of displaying the moving indicator superimposed on the composite image; 13. An information processing method comprising:

18. A program for causing a computer to execute each step of the information processing method according to claim 17.

19. A computer-readable storage medium storing a program for causing a computer to execute each step of the information processing method according to claim 17.

Images