Information processing apparatus, information processing method, image pickup apparatus, storage medium

Abstract

The present application relates to an information processing apparatus, an information processing method, an image capturing apparatus, and a storage medium. The information processing apparatus includes: a first acquisition section configured to acquire a captured image obtained by capturing an image of a chart for calibration; a second acquisition section configured to acquire a reference image, which is an image of the chart used as a reference; a third acquisition section configured to acquire information about a distortion aberration of a lens used to capture the captured image; a first generation section configured to generate a pseudo image based on the reference image and the information about the distortion aberration, the pseudo image being an image obtained by reflecting the distortion aberration in the reference image; and a second generation section configured to generate a composite image obtained by compositing the captured image and the pseudo image.

Description

Technical Field

[0001] This invention relates to information processing technology for acquiring information for aligning a camera and a chart in position. Background Technology

[0002] In recent years, a technique called Visual Effects (VFX) has attracted attention in the field of image production. VFX is a technique used to achieve non-realistic visual effects by compositing computer graphics (CG) with live-action images. Here, depending on the camera lens, when CG and live-action images are composited without considering lens distortion, differences arise between the live-action image with lens distortion aberration and the CG image without it. Therefore, in post-production, it is necessary to first correct the live-action image for lens distortion aberration, then composite the corrected live-action image with the CG image, and finally perform distortion correction on the composite image.

[0003] By acquiring distortion information from the camera (lens) as metadata, distortion aberration correction based on image height becomes possible. However, depending on the camera lens, distortion information may not be available. To calculate lens distortion when distortion information is unavailable, a calibration chart with repeating black-and-white grid patterns is typically captured during live-action shooting, and then, in post-production, a specialized application is used to calculate lens distortion from the captured image. This method, utilizing images involving the captured calibration chart, requires accurate alignment (position, pose) between the chart and the camera. Currently, in live-action shooting, the alignment of the chart and the camera is done visually by the user, thus inaccurate alignment is not achievable.

[0004] Japanese Patent Application Publication No. 2014-155086 discloses a method in which a reference image, used as a reference for adjusting the viewing angle, is stored in a viewing angle adjustment device (camera), and a motion image of the camera without viewing angle adjustment is combined with the reference image and displayed on a display device.

[0005] Japanese Patent 6859442 discloses the following method: assuming a fisheye lens is used, the grid pattern information (model data) of the subject used for imaging is stored in the imaging device, and the lens distortion is estimated based on the captured image and the pattern information.

[0006] However, the aforementioned Japanese Patent Application Publication No. 2014-155086 did not take into account the distortion of the lens used to acquire moving images. Therefore, in cases where the lens distortion is large, there will be a large deviation at the peripheral viewing angle (at the high image height) between the reference image displayed in a superimposed manner and the captured moving image, making positioning difficult.

[0007] Furthermore, Japanese Patent 6859442 discloses a method that, assuming a fisheye lens is used, acquires reference image information (graphic model data) and lens distortion parameters estimated from the captured image, and then estimates the position and orientation of the camera device. However, it does not mention a method for aligning the image and the camera device in position. Summary of the Invention

[0008] The present invention was made in view of the above-mentioned problems, and the present invention provides an information processing apparatus capable of acquiring information for aligning the relative positions of a drawing and a camera device.

[0009] According to a first aspect of the present invention, an information processing apparatus is provided, comprising: a first acquisition unit configured to acquire a captured image obtained by capturing an image of a calibration diagram; a second acquisition unit configured to acquire a reference image, the reference image being an image of the diagram used as a reference; a third acquisition unit configured to acquire information related to distortion aberrations of a lens used to capture the captured image; a first generation unit configured to generate a pseudo image based on the reference image and the information related to the distortion aberrations, the pseudo image being an image obtained by reflecting the distortion aberrations in the reference image; and a second generation unit configured to generate a composite image obtained by combining the captured image and the pseudo image.

[0010] According to a second aspect of the present invention, a camera device is provided, comprising: a lens; an image sensor for capturing images; and the aforementioned information processing device.

[0011] According to a third aspect of the present invention, an information processing method is provided, comprising: performing a first acquisition, the first acquisition being for acquiring a captured image obtained by capturing an image of a calibration diagram; performing a second acquisition, the second acquisition being for acquiring a reference image, the reference image being an image of the diagram used as a reference; performing a third acquisition, the third acquisition being for acquiring information related to distortion aberrations of a lens used to capture the captured image; performing a first generation, the first generation being for generating a pseudo image based on the reference image and the information related to the distortion aberrations, the pseudo image being an image obtained by reflecting the distortion aberrations in the reference image; and performing a second generation, the second generation being for generating a composite image obtained by combining the captured image and the pseudo image.

[0012] According to a fourth aspect of the present invention, a non-transitory computer-readable storage medium is provided, which stores a computer program for causing a computer to perform the steps of an information processing method, the information processing method comprising: performing a first acquisition, the first acquisition being for acquiring a captured image obtained by capturing an image of a calibration diagram; performing a second acquisition, the second acquisition being for acquiring a reference image, the reference image being an image of the diagram used as a reference; performing a third acquisition, the third acquisition being for acquiring information related to distortion aberrations of a lens used to capture the captured image; performing a first generation, the first generation being for generating a pseudo image based on the reference image and the information related to the distortion aberrations, the pseudo image being an image obtained by reflecting the distortion aberrations in the reference image; and performing a second generation, the second generation being for generating a composite image obtained by combining the captured image and the pseudo image.

[0013] Further features of the invention will become apparent from the following description of typical embodiments with reference to the accompanying drawings. Attached Figure Description

[0014] Figure 1 This is a block diagram illustrating the structure of a camera device according to a first embodiment of the present invention.

[0015] Figure 2 This is a flowchart illustrating the calibration process of the camera equipment.

[0016] Figure 3 This is a flowchart illustrating the basic position alignment process.

[0017] Figures 4A to 4D This is an illustration of the display screen during the basic position alignment process.

[0018] Figure 5 This is a flowchart illustrating the position alignment process.

[0019] Figures 6A to 6C This is a diagram illustrating the processing used to estimate lens distortion.

[0020] Figures 7A to 7C This is an illustration of the display screen during the alignment process.

[0021] Figure 8 This is a flowchart illustrating the position alignment process of the second embodiment.

[0022] Figures 9A to 9E This is a diagram illustrating the processing used to calculate the degree of deviation based on the captured image and the pseudo image in the second embodiment.

[0023] Figures 10A to 10C This is a diagram illustrating the display screen of an indicator based on the degree of deviation.

[0024] Figure 11 This is a flowchart illustrating the calibration process in the third embodiment.

[0025] Figure 12 This is a flowchart illustrating the viewpoint alignment process in the third embodiment.

[0026] Figure 13A This is a flowchart illustrating the process for calculating the difference between the image and the captured image in the third embodiment.

[0027] Figure 13B This is a flowchart illustrating the process for calculating the difference between the image and the captured image in the fourth embodiment.

[0028] Figure 14 This is a flowchart illustrating the process for calculating camera movement in the third embodiment.

[0029] Figures 15A to 15D This is a diagram illustrating the calculation of camera movement in the third embodiment.

[0030] Figures 16A to 16D This is a diagram illustrating the calculation of camera movement in the third embodiment.

[0031] Figures 17A to 17D This is a diagram illustrating the calculation of camera movement in the third embodiment.

[0032] Figure 18A and Figure 18B This is an illustration of the camera display during and after viewpoint alignment in the third embodiment.

[0033] Figure 19 This is a diagram illustrating the processing for calculating the difference between the diagram and the captured image in the third embodiment. Detailed Implementation

[0034] The embodiments will be described in detail below with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. Multiple features are described in the embodiments, but there is no limitation to an invention requiring all of these features, and multiple features can be appropriately combined. Furthermore, in the drawings, the same reference numerals are assigned to the same or similar configurations, and redundant descriptions are omitted.

[0035] First Embodiment

[0036] Structure of camera equipment

[0037] The structure of the camera device, which is the first embodiment of the information processing apparatus of the present invention, will be described below. Figure 1 This is a block diagram showing the structure of the camera device 100.

[0038] exist Figure 1 In this embodiment, the camera device 100 is configured to include a camera control unit 130 and a camera lens 110 replaceably mounted to the camera control unit 130. Examples of the camera device 100 include, but are not limited to, video cameras and still cameras capable of capturing images of a subject and recording moving or still image data to various recording media. Hereinafter, the camera device 100 will be described as a camera 100.

[0039] The computing device 136 controls the entire camera control device 130. In addition, the computing device 136 sends commands to the lens control unit 121 via the electrical contact unit 150 for driving the lens group and aperture equipped in the camera lens 110, and commands for causing the camera lens 110 to send the lens information (optical information, etc.) it holds to the camera control device 130.

[0040] The camera lens 110 is configured as a lens unit equipped with a camera optical system, which includes a fixed lens group 111, a zoom lens 112, an aperture 113, an image stabilizing lens 114, and a focusing lens 115. Furthermore, drive units for driving the lenses and aperture are connected to and controlled by the lens control unit 121 via a bus 122. The lens control unit 121 controls the various lenses and aperture via the zoom drive unit 116, the aperture drive unit 117, the image stabilizing drive unit 118, and the focusing drive unit 119 according to commands from the computing device 136.

[0041] Aperture drive unit 117 adjusts the aperture 113 by driving it to regulate the amount of light during image capture. Zoom drive unit 116 changes the focal length by driving zoom lens 112. Image stabilization drive unit 118 responds to camera shake by driving image stabilization lens 114 to reduce image blur caused by camera shake. Focus drive unit 119 controls the focus state by driving focusing lens 115. Lenses 111, 112, 114, and 115 are... Figure 1 It is simply shown as a single lens, but is usually composed of multiple lenses.

[0042] Electrical contacts (terminals on the camera lens side / terminals on the camera control device side) are arranged in the electrical contact unit 150, corresponding to the two communication lines used in the communication between the camera lens 110 and the camera control device 130, respectively. The lens control unit 121 communicates with the camera control device 130 via the electrical contact unit 150 and controls the driving of the zoom drive unit 116, aperture drive unit 117, and focus drive unit 119 according to the operation information from the lens operation unit 120. In addition, the lens control unit 121 communicates with the camera control device 130 via the electrical contact unit 150 and receives commands from the computing device 136. Furthermore, it transmits lens information (optical information, etc.) held in the camera lens 110 based on the transmission request from the camera control device 130 side (hereinafter, the communication between the lens control unit 121 and the computing device 136 will be referred to as lens communication).

[0043] The lens operation unit 120 is equipped with operation components such as a zoom operation ring, a focus operation ring, an aperture operation ring, and an operation switch for turning the image stabilization within the lens on / off. When the user operates any of these operation components, an operation indication signal is output to the lens control unit 121, and the lens control unit 121 performs control appropriate to that operation.

[0044] The subject image formed by the light beam from the camera optical system passing through the camera lens 110 on the image sensor 131 is converted into an electrical signal by the image sensor 131. The image sensor 131 is a photoelectric conversion device that converts the subject image (optical image) into an electrical signal through photoelectric conversion. The electrical signal obtained by photoelectric conversion of the subject image formed on the image sensor 131 is processed into an image signal (image data) by the camera signal processing unit 132.

[0045] The image sensor control unit 133 receives from the computing device 136 an indication of the storage time of the image sensor 131 and the value of the gain to be output from the image sensor 131 to the camera signal processing unit 132, and controls the image sensor 131.

[0046] Image data output from camera signal processing unit 132 is sent to image sensor control unit 133 and temporarily stored in volatile memory 138. Furthermore, after being processed in image processing unit 137, such as by correction and compression, the image data is recorded to storage medium 143, such as a memory card.

[0047] In parallel, the display control unit 141, based on commands from the computing device 136, performs processing to reduce / enlarge the image data stored in the volatile memory 138 to the optimal size for the display unit 140 (such as a display mounted in the camera control device 130). The image data processed to the optimal size is then temporarily stored again in a different area of ​​the volatile memory 138 than before processing. Furthermore, the display control unit 141 overlays camera information such as exposure settings onto the image data using characters and icons. The image is displayed by sending the image data overlaid with various information to the display unit 140. Thus, the user can observe the captured image in real time (hereinafter, the image that can be observed in real time will be referred to as a live view image). The display control unit 141 also controls the processing for overlaying pseudo images onto the captured image implemented in this embodiment.

[0048] The image stabilization control unit 135, based on commands from the computing device 136, controls the image sensor 131 via the image stabilization drive unit 134 in the direction of correcting image blur caused by camera shake. The image stabilization drive unit 134 can also be driven in conjunction with the image stabilization drive unit 118 of the camera lens 110, thereby enabling image stabilization to be achieved over an even larger range than when image stabilization is performed solely by the image stabilization drive unit 134.

[0049] The operation unit 142 is an operating component that enables the user to instruct various units equipped in the camera control device 130, and includes, for example, an operation switch, operation ring, operation joystick, or touch panel mounted on the display unit 140 for controlling operations such as camera recording and focus adjustment. Instructions related to the driving conditions of the camera 100 input by the user through operation of the operation unit 142 are sent to the computing device 136. The computing device 136 then sends commands to the respective units based on the operation instruction signal.

[0050] The volatile memory 138 is used not only to temporarily store the aforementioned image data, but also to store temporary data used in the processing of each unit of the camera control device 130, as well as lens information obtained from the camera lens 110.

[0051] Non-volatile memory 139 stores the control program required for the operation of camera 100. When camera 100 is started by user operation (when camera 100 changes from power off to power on), the control program stored in non-volatile memory 139 is read (loaded) into a portion of volatile memory 138. Computing device 136 controls the operation of camera 100 according to the control program loaded in volatile memory 138. Non-volatile memory 139 is also writable and has an image information storage unit 139a and a lens distortion information storage unit 139b that respectively store image information used in this embodiment and information related to distortion aberration.

[0052] Storage medium 143 is a readable and writable memory card such as an SD card, and is used to store captured images (moving or still images) and metadata associated with the images. It is also envisioned that storage medium 143 will be used in place of image information storage unit 139a and lens distortion information storage unit 139b of non-volatile memory 139.

[0053] Overview of calibration processing

[0054] Next, we will use Figure 2 The calibration process is summarized below. In this embodiment, in order to correct the distortion aberrations of the camera lens 110, it is actually necessary to use the camera 100 to capture an image of the calibration map used for aberration correction. The calibration process refers to the operation of aligning the calibration map and the camera 100 in position, then using the camera 100 to capture an image of the calibration map and obtain information related to the distortion aberrations of the camera lens 110.

[0055] The following description assumes that the aforementioned camera control begins before the calibration process is performed, and that the user is able to observe the live view image. Furthermore, the calibration operation itself is performed after the user selects to perform the calibration process from the menu displayed on the display unit 140 via the operation unit 142. Additionally, the following description assumes that the image captured by the camera 100 is always in focus. Focusing can be done manually or via autofocus, and there are no particular limitations on the method used.

[0056] First, in step S201, the computing device 136 starts the calibration process and moves the process to step S202.

[0057] In step S202, the computing device 136 activates the calibration mode. Specifically, the computing device 136 changes the flag of the calibration mode stored in the volatile memory 138 to ON.

[0058] In step S203, the computing device 136 performs a basic position alignment process between the camera 100 and the calibration map. This basic position alignment process involves coarse alignment of the optical axis center of the camera 100 with the center of the calibration map, as well as alignment of the camera's viewing angle, during initial position alignment. This basic position alignment process will be described in detail later.

[0059] In step S204, the computing device 136 determines whether the basic position alignment process of step S203 is complete. If the basic position alignment process is complete, the computing device 136 causes the process to proceed to step S205, and if the basic position alignment process is not complete (e.g., an error occurred during basic position alignment or the user stopped the process), the process proceeds to step S208.

[0060] In step S205, the computing device 136 performs a position alignment process for the camera 100 and the calibration map. In this position alignment process, compared to the basic position alignment process in step S203, the computing device 136 performs detailed position alignment while checking the peripheral (high image height) portion of the image, and sets the camera 100 and the calibration map to a state where imaging can be performed to calculate lens distortion. This position alignment process will be described in detail later.

[0061] In step S206, the computing device 136 determines whether the position alignment process in step S205 is complete. If the position alignment is complete, the computing device 136 causes the process to proceed to step S207, and if the position alignment is not complete (an error occurred during the position alignment or the user stopped the process), the process proceeds to step S208.

[0062] In step S207, after setting up the camera 100 and the image with the position alignment completed, the computing device 136 captures an image of the calibration image. The computing device 136 saves the captured image to the storage medium 143 and causes the process to proceed to step S208.

[0063] In step S208, the computing device 136 shuts down the calibration mode. Then, the computing device 136 causes the process to proceed to step S209 and ends the calibration process.

[0064] Imagine using multiple camera lenses, or in the case of zoom lenses, to actually capture video at multiple focal lengths. In this scenario, the process would be repeated for each camera lens or each focal length. Figure 2 The processing of steps S201 to S209 in the process.

[0065] By performing the above processing, an image for calculating lens distortion can be obtained after accurate calibration mapping and camera alignment. In the post-production of VFX compositing, a dedicated application is used to calculate lens distortion based on this captured image, and this distortion is then applied in various compositing processes.

[0066] Basic position alignment processing

[0067] Next, we will use Figure 3 To explain in detail Figure 2 The basic position alignment process in step S203. In this embodiment, the flowchart processing is performed by the computing device 136 based on a computer program loaded from non-volatile memory 139 to volatile memory 138 when the camera 100 is started. This similarly applies to the operations in subsequent flowcharts.

[0068] First, in step S301, the computing device 136 begins basic position alignment processing.

[0069] In step S302, the computing device 136 reads the image captured during the aforementioned camera control from the volatile memory 138 and temporarily saves the read image to another area of ​​the volatile memory 138. Furthermore, image information for position alignment is obtained from the image information storage unit 139a.

[0070] In step S303, the computing device 136 generates a pseudo image based on the image information obtained in step S302. The pseudo image mentioned here is either the image before distortion processing or the image without distortion processing. Here, distortion processing is a process used to intentionally distort an image of a picture where the lens distortion aberration was not initially present to match the lens distortion aberration, and to convert that image into an image reflecting the lens distortion aberration (distortion variable). In the following text, the pseudo image of the picture before distortion processing or the image without distortion processing will be referred to as the reference image.

[0071] Image information is the information required to generate a reference image or pseudo-image. Examples include an image of the actual calibration map of the subject used for imaging and the size of a grid portion of the calibration map. As long as the above information is associated with the focal length of the camera lens, processing based on the focal length can be performed automatically even if the camera lens is changed or the focal length of the zoom lens is altered. Furthermore, it is assumed that the user stores the image information in the image information storage unit 139a of the camera control device 130 or in the storage medium 143 before performing calibration processing.

[0072] However, in cases where image information is not stored or the image information differs significantly from the actual captured image, the following configuration can be used. That is, an image information generation mode is provided, and image information is generated through user selection or input on the camera control device 130. Alternatively, image information can be generated by detecting the size of a grid portion in the center of the captured image. The generated image information is then saved as new image information to the image information storage unit 139a.

[0073] Here, an example of generating a reference image based on graph information will be described. This description assumes an image of a graph corresponding to the focal length stored in the non-volatile memory 139a.

[0074] First, the computing device 136 reads the image information from the non-volatile memory 139a. Then, the computing device 136 obtains the focal length of the camera lens 110 mounted on the camera control device 130 via lens communication. Alternatively, for lenses that do not support lens communication, the user can set the focal length in the camera control device 130.

[0075] The image information read is an image corresponding to a focal length. Therefore, if the focal length differs from the focal length obtained through lens communication, it is necessary to adjust the image for the difference in focal length. In view of this, the computing device 136 notifies the image processing unit 137 of the focal length of the camera lens and instructs the image processing unit 137 to perform magnification / reduction processing suitable for the difference in focal length (magnification). The image processing unit 137 performs magnification / reduction processing on the read image based on this instruction. Subsequently, the computing device 136 temporarily saves the generated reference image to the volatile memory 138 and proceeds to step S304.

[0076] In step S304, the computing device 136 uses the display unit 140 and the display control unit 141 to perform processing for overlaying a reference image onto the captured image. First, the display control unit 141 reads the captured image and the reference image temporarily stored in the volatile memory 138. Then, image synthesis is performed so that the reference image is displayed as an overlay on the acquired captured image. Hereinafter, this process will be referred to as overlay processing. Furthermore, the image obtained through overlay processing will be referred to as an overlaid image (synthesized image).

[0077] At this point, if the reference image is simply overlaid, the captured image below cannot be observed, and the difference between the captured image and the reference image is also difficult to discern. Therefore, the display control unit 141 performs the necessary image processing on the reference image. Examples include: performing transparency processing on the reference image so that the captured image can be observed through the reference image; performing processing to convert the black grid portions of the pseudo-image corresponding to the black grid portions of the image in the captured image into another color such as red; and performing pattern processing. Then, the display control unit 141 temporarily saves the overlaid image to the volatile memory 138.

[0078] Next, the display control unit 141 notifies the display unit 140 of the temporary storage destination of the overlay image and instructs the display unit 140 to update the display. The display unit 140 reads the overlay image temporarily stored in the volatile memory 138 and displays the overlay image. By observing the overlay image displayed on the display unit 140, the user can check the degree to which the captured image deviates from the pseudo-image. In other words, the user can observe the misalignment between the calibration map and the camera. Then, the user changes the position or orientation of the camera 100 or the calibration map to minimize the misalignment and performs alignment.

[0079] Will use Figures 4A to 4D Here is an example of the display screen during basic position alignment. Figures 4A to 4D The image displayed on display unit 140 is shown. Note that the calibration image itself is a repeating arrangement of black and white grid portions as described above, and in the following text, the black grid portions of the captured image will be represented by downward-sloping shading lines, and the black grid portions of the reference image or pseudo-image will be represented by upward-sloping shading lines.

[0080] Figure 4A The diagram shows only the captured image 410. Reference numerals 401 and 402 represent vertical and horizontal marks, respectively, intersecting at the optical center to make the center of the captured image easily visible. Furthermore, a black rectangular mark 403 is shown to make the center of the calibration map easily visible, but the mark can actually be placed at the center of the calibration map, or the rectangular portion including the center position can be set to a color other than black or white. Additionally, when implementing overlay processing, the display method or color can be changed to make center alignment easily visually confirmed by aligning the grid portion including the center position of the reference image with the marks or colors of the calibration map.

[0081] then, Figure 4B Show relative to only show Figure 4AThe state of the captured image 410 and the state of the reference image 411 displayed in an overlay manner are described. Basic position alignment is performed based on the reference image 411 displayed in an overlay manner. In this example, the user changes the position or orientation of the camera 100 so that the optical axis center of the camera 100 is aligned with the center of the calibration map.

[0082] Figure 4C This shows a state where the center positions of the captured image 410 and the reference image 411 are aligned, but the captured image 410 and the reference image 411 are not yet matched (they do not match in size). From this state, the user changes the position of the camera 100 so that the camera 100 is closer to the image and achieves... Figure 4D The display status. By performing the above process, the basic position alignment of the camera 100 and the image is completed.

[0083] Furthermore, as mentioned above, the reference image can be magnified / reduced based on the focal length obtained through lens communication or set by the user. However, depending on the camera lens, since the focal length itself is unknown, it may be impossible to magnify / reduce the reference image. In this case, from Figure 4C If the sizes do not match, the user operates the operation unit 142 and zooms in / out on the reference image 411 until he or she can visually determine that the captured image 410 matches the reference image 411 (until a match is achieved). Figure 4D Until a similar state is reached.

[0084] Next, in step S305, the computing device 136 determines whether to continue basic position alignment. The purpose is to determine whether the user has completed basic position alignment. For example, a button-like user interface "OK" and "Cancel" are further displayed on the overlay image, where "OK" is assigned to the user's completion of basic position alignment, and "Cancel" is assigned to the user's desire to stop basic position alignment. If the user decides that basic position alignment is complete or decides to stop, he or she selects "OK" (complete) or "Cancel" (stop) respectively via the operation unit 142. Subsequently, the computing device 136 determines whether to continue the basic position alignment process. If no completion or stop is selected, the computing device 136 returns the process to step S302 and continues the process from steps S302 to S304 until completion or stop is selected, or until the basic position alignment process is forcibly terminated via the operation unit 142. If completion or stop is selected, the computing device 136 proceeds to step S306.

[0085] In step S306, the computing device 136 determines whether the "complete" or "stop" option was selected in step S305, that is, whether the basic position alignment is finished. If the "complete" option was selected in step S305, the computing device 136 proceeds to step S307 and sets the basic position alignment result to "OK". Conversely, if the "stop" option was selected, the computing device 136 proceeds to step S308 and sets the basic position alignment result to "cancel".

[0086] In step S309, the computing device 136 temporarily saves the basic position alignment result set in step S307 or S308 to the volatile memory 138. Additionally, the computing device 136 saves the reference image at the end of the basic position alignment as image information to the image information storage unit 139a.

[0087] In step S310, the computing device 136 completes the basic position alignment process.

[0088] Position alignment processing

[0089] After that, it will be used Figure 5 This describes a position alignment process used to achieve higher accuracy than the basic position alignment process.

[0090] First, in step S501, the computing device 136 begins position alignment processing.

[0091] In step S502, the computing device 136 reads the graph information from the graph information storage unit 139a in the non-volatile memory 139.

[0092] In step S503, the computing device 136 reads the image captured during the above-mentioned camera control from the volatile memory 138 and temporarily saves the read image to another area of ​​the volatile memory 138.

[0093] In step S504, the computing device 136 performs lens distortion acquisition processing and proceeds to step S504.

[0094] Now will be used Figures 6A to 6C This section illustrates an example of estimation processing as part of the lens distortion acquisition process. Figure 6A It is a diagram illustrating the grid information of the calibration plot (used to identify the position of each grid section and the four corners of each grid section).

[0095] Here, the number of black and white grid sections arranged horizontally and vertically are 17 and 11 respectively. To identify the positions of the four corners of each grid section, numbers 0 to 17 (i0 to i17) are given horizontally and numbers 0 to 11 (j0 to j11) are given vertically. Next, to identify each grid section, numbers 0 to 186 (g0 to g186) are given sequentially from the top left. Therefore, the coordinates of the four corners of the top-left grid section g0 can be represented as top-left (x... i0j0 ,y i0j0 ), top right (x) i1j0 ,y i1j0 ), lower left (x) i0j1 ,y i0j1 ) and bottom right (x) i1j1 ,y i1j1 ).

[0096] In addition, such as Figure 6B As shown, in order to identify the coordinates of the captured image and the reference image, the coordinates of the four corners of the grid portion 601 of the captured image are given as the top left (Cx). i0j0 Cy i0j0 ), top right (Cx) i1j0 Cy i1j0 ), bottom left (Cx) i0j1 Cy i0j1 ) and bottom right (Cx) i1j1 Cy i1j1 ), and the coordinates of the four corners of the grid portion 602 of the reference image are given as the top left (Rx i0j0 ,Ry i0j0 ), top right (Rx) i1j0 ,Ry i1j0 ), bottom left (Rx) i0j1 ,Ry i0j1 ) and bottom right (Rx i1j1 ,Ry i1j1 ).

[0097] Next, the estimation process based on the grid information of each grid section will be explained. First, lens distortion aberration is typically modeled using the following formula. In the formula, xd and yd are the coordinates with distortion, xu and yu are the coordinates without distortion, and K1 to K5 are coefficients representing the amount of lens distortion.

[0098] xd=(1+K1r 2 +K2r 4 +K5r 6 )xu+2K3xuyu+K4(r 2 +2xu 2 )

[0099] yd=(1+K1r 2+K2r 4 +K5r 6 )yu+K3(r 2 +2yu 2 )+2K4xuyu

[0100] r 2 =xu 2 +yu 2

[0101] The lens distortion calculated using these formulas is saved to the lens distortion information storage unit 139b. At this point, the camera lens's focal length, lens ID, and serial number can be saved together with the lens distortion. This allows the information to be reused during recalibration.

[0102] In the above equation, K3 and K4 represent tangential distortion aberrations that are usually negligible in practice, so a simplified model like the one below is often used.

[0103] xd=(1+K1r 2 +K2r 4 +K5r 6 )xu

[0104] yd=(1+K1r 2 +K2r 4 +K5r 6 )yu

[0105] Figure 6A The top-left coordinate of the lattice portion g=0 is as follows.

[0106] Cx i0j0 =(1+K1r) 2 +K2r 4 +K5r 6 Rx i0j0

[0107] Cy i0j0 =(1+K1r) 2 +K2r 4 +K5r 6 Ry i0j0

[0108] r 2 =Rx i0j0 2 +Ry i0j0 2

[0109] Using these relationships, a search is performed for the distorted variables K1, K2, and K5 to ensure that the coordinates of the four corners of each grid portion (g = 0 to 186) in the captured image and the reference image are consistent or close to the values. If consistent or close coordinates are obtained as the search result, the estimation can be considered complete.

[0110] This embodiment has been described by simply making the four corners of each grid portion consistent. However, by further dividing the coordinates between the four corners of each grid portion after achieving a certain degree of consistency, and by further increasing the number of coordinates compared, the accuracy of the estimated distortion is improved. Furthermore, in this embodiment, the estimation process is described in terms of searching for coefficients representing the distortion, but the invention is not limited thereto. Afterwards, the computing device 136 saves the estimated lens distortion to the lens distortion information storage unit 139b.

[0111] Furthermore, in this embodiment, a method for estimating lens distortion based on captured images and image information stored in the camera control device 130 is described. However, depending on the camera lens, lens distortion corresponding to the state (focal length, etc.) of each lens can be stored in the camera lens 110 or in the camera control device 130 (non-volatile memory 139). These lens distortions are determined during the design or manufacture of the camera lens. When the distortions are stored in the camera lens, lens communication is performed in the lens distortion acquisition process of step S504, and the distortions of each lens position at the time of communication are acquired from the lens side. When the distortions are stored on the camera control device 130 side, first lens communication is performed, and a unique number assigned to each camera lens (hereinafter referred to as lens ID) is acquired. Then, the lens distortions associated with the lens ID and each lens position are read from the non-volatile memory 139. The pseudo-images described later can also be generated based on these lens distortions.

[0112] Next, in step S505, the computing device 136 generates a pseudo-image based on the lens distortion and image information obtained in step S504. This is because the processing prior to distortion processing... Figure 3 The reference image generation process in step S303 is similar, so its description will be omitted. Next, the computing device 136 reads the reference image temporarily stored in the volatile memory 138 and reads the lens distortion stored in the non-volatile memory 139b. Then, the computing device 136 instructs the image processing unit 137 to perform distortion processing on the reference image. The image processing unit 137 generates a pseudo-image reflecting the distortion aberration based on the read reference image and lens distortion, temporarily stores the generated pseudo-image in the volatile memory 138, and proceeds to step S506.

[0113] In step S506, the image processing unit 137 overlays the pseudo-image generated in step S505 onto the captured image and displays the overlaid image. Note that due to the processing itself and Figure 3 The processing in step S304 is similar, so the description of the processing will be omitted. The user aligns the camera 100 and the image by observing the superimposed image and making the captured image and the pseudo-image consistent.

[0114] Now will be used Figures 7A to 7C An example illustrating the display screen during positioning alignment. Figures 7A to 7C The image displayed on the display unit 140 is shown. Figure 7A Show Figure 2 The basic position alignment is completed in steps S203 and S204. Figure 7B The following state is shown: displayed in an overlay manner from the reference image 711. Figure 7A The state on the captured image 710 is generated by distorting the reference image 711 to obtain a pseudo image 712, and the generated pseudo image 712 is displayed in an overlay manner. Figure 7B This example shows a slightly misaligned captured image and a pseudo-image. The user can change the position or orientation of camera 100 from this state until a desired alignment is achieved. Figure 7C The position is aligned until the desired state is achieved, making the captured image and the pseudo-image as close as possible to each other.

[0115] After continuously estimating lens distortion during the ongoing alignment process, a pseudo-image is generated. Therefore, assuming that the camera 100 and the image are approximately in opposite positional relationships, the accuracy of distortion estimation will be greater, and the deviation between the captured image and the pseudo-image will be smaller. Furthermore, unlike the basic alignment process, the alignment of the captured image and the pseudo-image can be checked up to the peripheral image height. Therefore, the user can operate the operation unit 142 to magnify the image position he or she wants to check and examine the degree of alignment between the captured image and the pseudo-image. As a result, even with a lens exhibiting high distortion, accurate alignment of the camera 100 and the image can be achieved by aligning the captured image and the pseudo-image up to the peripheral image height.

[0116] Next, in step S507, the computing device 136 determines whether to continue position alignment. Note that because this process is related to... Figure 3The processing in step S305 is similar, so the description of the processing content will be omitted. At this time, if no completion or stop is selected, the computing device 136 returns the processing to step S503. The processing from acquiring the captured image to overlaying the pseudo-image, as described in steps S503 to S506, continues until completion or stop is selected, or until the position alignment processing is forcibly terminated via the operation unit 142. Furthermore, if completion or stop is selected, the computing device 136 proceeds to step S508.

[0117] Here, due to the processing in steps S508 to S510 and Figure 3 The processes in steps S306 to S308 are similar, so their descriptions will be omitted.

[0118] In step S511, the computing device 136 temporarily saves the position alignment result set in step S509 or S510 to the volatile memory 138. Additionally, the computing device 136 saves the pseudo-image at the end of the position alignment as image information to the image information storage unit 139a.

[0119] In step S512, the computing device 136 completes the position alignment process.

[0120] Note that when in Figure 2 In step S207, when processing is performed to capture and save an image for calculating lens distortion, various information (such as position alignment results, lens information at the end of position alignment, lens distortion and image information, etc.) can be saved together with the image itself as the image itself or as information (metadata) associated with the image to the storage medium 143.

[0121] Furthermore, this embodiment has been described from the perspective of updating the display of the overlay image to match the cycle of pseudo-image generation in steps S503 to S506 of the position alignment process. This is to avoid the situation where the processing time spent on overlay image generation in steps S504 to S506 is longer than a time period equivalent to one camera cycle (readout cycle of image sensor 131), resulting in a different captured image than the captured image at the time of pseudo-image generation. However, when the time spent on overlay image generation is shorter than the time period of one camera cycle, overlay image generation and display updates can be performed to match the camera cycle. Furthermore, where the display update cycle can be delayed, the display update can be performed after a pseudo-image is generated based on the movement of the camera 100. Note that the movement of the camera 100 can be detected using methods such as detecting changes in the captured image or providing a gyroscope sensor in the camera control device 130 to detect movements such as changes in posture.

[0122] Furthermore, this embodiment describes the process of performing position alignment after basic position alignment processing. However, if the lens distortion is stored in the camera lens 110 or camera control device 130, or if the lens distortion (shown in the entire field of view) can be estimated even without center position alignment, position alignment processing can be performed first.

[0123] As described above, in this embodiment, the image information of the subject being photographed is stored in the camera control device, and a pseudo-image obtained by distortion processing is generated based on this image information and information related to lens distortion. Then, by displaying the pseudo-image on the captured image in a superimposed manner, the positional alignment of the camera device and the image can be performed accurately and efficiently.

[0124] Second Embodiment

[0125] In the first embodiment, a method is described for accurately and efficiently aligning the position of the camera device and the image by generating a pseudo-image obtained through distortion processing based on image information stored in the camera control device and information related to lens distortion, and displaying the generated pseudo-image on top of the captured image in a superimposed manner. In the second embodiment, a method is described for accurately aligning the position by calculating the degree of deviation between the captured image and the pseudo-image, and further superimposing information and indications suitable for the calculated degree of deviation onto an image obtained by displaying the pseudo-image on top of the captured image in a superimposed manner.

[0126] Since the overview of the processing in the second embodiment is similar to that described in the first embodiment... Figure 2 The processing involved will therefore be omitted from the description. Due to the second embodiment... Figure 2 The position alignment process in step S205 differs from that in the first embodiment, so its processing content will be described.

[0127] First, refer to Figure 8 The position alignment process in the second embodiment is explained. Note that the processes in steps S801 to S805 are similar to those in the first embodiment. Figure 5 The processes in steps S501 to S505 and steps S808 to S813 are similar to those in steps S507 to S512, so their descriptions will be omitted.

[0128] The computing device 136 performs the processing steps S801 to S805 and temporarily saves the captured image and the pseudo image to the volatile memory 138.

[0129] In step S806, the computing device 136 calculates the degree of deviation based on the captured image and the pseudo image. This will be used later. Figures 9A to 9EAn example illustrating the method for calculating the degree of deviation is provided. Next, the computing device 136 temporarily saves the evaluation value based on the calculated degree of deviation to the volatile memory 138.

[0130] In step S807, the computing device 136 uses the display unit 140 and the display control unit 141 to perform processing for overlaying a pseudo-image onto the captured image. This overlay process is similar to that in the first embodiment. Figure 5 The process in step S506 will therefore be omitted from the description.

[0131] Subsequently, in order to make the captured image and the pseudo-image consistent based on the evaluation value of the degree of deviation calculated in step S806, the computing device 136 generates a graphical user interface (GUI) as an indicator to prompt the user to change the position or posture of the camera 100, and further displays the generated GUI on the overlay image in an overlay manner. This will be used later. Figures 10A to 10C The description shows an example of an indicator used to move the camera 100.

[0132] Furthermore, in this embodiment, the degree of deviation is represented by the distance and direction (vector) of the four corners (feature points) of the corresponding grid portion of the captured image and the pseudo image, but it can also be represented by the distance and direction of the centroid of the corresponding grid portion (specific area) of the captured image and the pseudo image.

[0133] In step S808, if the user selects to complete or stop position alignment, the computing device 136 causes the processing to proceed to step S809 and executes the processing in steps S809 to S812.

[0134] Then, the computing device 136 completes the position alignment process in step S813.

[0135] Next, we will use Figures 9A to 9E This example illustrates a method for calculating the degree of deviation. Here, the distance and direction (vector) of the four corners of the corresponding grid portions of the captured image and the pseudo-image are used. Furthermore, the information (numbering) used to identify the position of each grid portion and the positions of its four corners is similar to... Figure 6A .

[0136] Figure 9A This is a graph used to illustrate the calculation of the deviation of the upper left grid section (grid section number: g=0). The coordinates of the four corners of grid section 901 on the image-taking side are the upper left (Cx... i0j0 Cy i0j0 ), top right (Cx) i1j0 Cy i1j0 ), bottom left (Cx) i0j1 Cy i0j1 ) and bottom right (Cx) i1j1Cy i1j1 Furthermore, the coordinates of the four corners of the grid portion 902 on the pseudo-image side are the top left (Rx). i0j0 ,Ry i0j0 ), top right (Rx) i1j0 ,Ry i1j0 ), bottom left (Rx) i0j1 ,Ry i0j1 ) and bottom right (Rx i1j1 ,Ry i1j1 The following explanation will use the top left corner as an example to illustrate the subsequent processing.

[0137] Calculate the distance and direction (vector) V between the top left of the grid portion 901 on the captured image side and the top left of the grid portion 902 on the pseudo-image side. i0j0 At this time, V i0j0 By (Rx i0j0 -Cx i0j0 ,Ry i0j0 -Cy i0j0 () indicates. Next, calculate the upper right V. i1j0 Lower left V i0j1 and the bottom right V i1j1 The degree of deviation. Then, the degree of deviation of the lattice part (lattice part number: g=0) is determined by Vdd. g0 =(V i0j0 +V i1j0 +V i0j1 +V i1j1 The deviation is indicated by (). The deviation is calculated for all grid portions or black or white grid portions. Then, the computing device 136 temporarily saves the deviation calculated for each grid portion to the volatile memory 138.

[0138] Next, the computing device 136 calculates an evaluation value based on the degree of deviation and displays an indicator for moving the camera 100. An example of calculating the evaluation value based on the degree of deviation will be explained. For example, the degree of deviation Vdd of the aforementioned grid portion is calculated. gN The sum of (N = 0 to 186). In Figure 9B or Figure 9C In this process, the result will be that there is no deviation between the captured image and the fake image. Furthermore, in Figure 9D or Figure 9E In this case, the result will be a false image that is misaligned to the lower or right side relative to the captured image. Furthermore, the image can be divided into four quadrants with reference to the optical axis center of the camera 100. A sum can be calculated for each quadrant, and further calculations can be made of the differences between quadrants or the sums of quadrants.

[0139] Will use Figures 10A to 10C An example illustrating the movement indicator is shown. Figure 10AAn example is shown where a message relating to the direction of movement of camera 100 is further displayed overlaid on a superimposed image. The user changes the position of camera 100 based on this message and fine-tunes the position until the message disappears or a message indicating that the alignment is complete is displayed.

[0140] Figure 10B An example is shown that displays a graphic indicating the direction of movement of camera 100. Arrow markers indicating the direction of movement are displayed, and the length or size of the arrows can be changed according to the value of the movement.

[0141] Figure 10C An example is shown where the direction of movement and approximate amount of movement of camera 100 are displayed as a guide with a bar gauge. Figure 10C Bar gauges 1003 to 1006 are user interfaces used to guide the movement of camera 100. Bar gauges 1003 and 1004 represent the horizontal direction, and bar gauges 1005 and 1006 represent the vertical direction.

[0142] As evaluation values ​​for displaying the bar gauges, the degree of deviation in the four quadrants is used. The horizontal bar gauge 1003 at the top of the image represents the difference in deviation between the first and fourth quadrants, and the horizontal bar gauge 1004 at the bottom of the image represents the difference in deviation between the second and third quadrants. Similarly, the vertical bar gauge 1005 on the left side of the image represents the difference in deviation between the third and fourth quadrants, and the vertical bar gauge 1006 on the right side of the image represents the difference in deviation between the first and second quadrants.

[0143] The black triangle represents the current state, where Figure 10C This shows a state where the vertical position is aligned but the horizontal position is misaligned to the left. While observing bar gauges 1003 to 1006, the user changes the position of camera 100 and makes minor adjustments until all bar gauges 1003 to 1006 are in the same position. Note that... Figures 10A to 10C The example of the movement indicator shown is illustrative, and the invention is not limited thereto.

[0144] Furthermore, in this embodiment, the completion of position alignment is determined by the user's visual confirmation and judgment. Therefore, a configuration is adopted that uses the aforementioned degree of deviation to determine the completion of position alignment. For example, a configuration may be adopted as follows: if the evaluation value based on the degree of deviation decreases to less than or equal to a predetermined value, the position alignment is determined to be complete, and the position alignment process ends after implementing the required termination process.

[0145] Furthermore, after completing the alignment, the calibration process outlined in steps S201 to S207 will be explained from the perspective of capturing images used to calculate lens distortion. However, a configuration can be adopted whereby images used to calculate lens distortion are automatically captured when the alignment is determined to be complete using the aforementioned degree of deviation.

[0146] Furthermore, in the first embodiment, the case where lens distortion variables are stored in the camera lens 110 or the camera control device 130 is described. These lens distortion variables are values ​​corresponding to the positions of each lens, but due to limitations such as the storage capacity of the camera lens 110 or the camera control device 130, these values ​​can actually be discrete values. In this case, depending on the lens position, it is assumed that even the pseudo-image obtained through distortion processing will deviate significantly from the captured image. In this case, a configuration can be adopted whereby processing for estimating lens distortion variables based on the captured image and image information described in the first embodiment is performed in parallel. In this case, for example, the degree of deviation of the captured image from the pseudo-image generated based on the lens distortion variables stored by the camera device 100 or the camera control device 130 and the pseudo-image generated based on the estimated lens distortion variables are calculated. Then, the pseudo-image with the smaller absolute value of the deviation is displayed.

[0147] Furthermore, the above explanation of calculating the degree of deviation from all grid portions of the image in the captured image illustrates the processing in step S806 for calculating the degree of deviation based on the captured image and the pseudo-image. However, as the image height increases from the optical center to the periphery (at higher image heights), lens distortion tends to increase. Therefore, the area for calculating the degree of deviation can be limited based on lens distortion or image height. As a result, the computational processing load can be reduced.

[0148] Furthermore, although the pseudo-image used in this process of step S806 is a distorted image, the evaluation value based on the degree of deviation is determined according to the relative difference in the degree of deviation. Therefore, a configuration that calculates the degree of deviation based on the captured image and the undistorted reference image can be adopted.

[0149] Furthermore, in the process of overlaying the pseudo-image onto the captured image in step S807, the content of the overlay process can be changed according to the degree of deviation calculated in step S806. For example, it is conceivable to change the opacity, color, or display method according to the degree of deviation. Specifically, in the case of changing the color, if the display color of the black grid portion of the pseudo-image corresponding to the black grid portion of the image in the captured image is red, the color of the grid portion with a deviation greater than or equal to a predetermined value is changed to a contrasting color that is easily visible relative to the display color. In this case, red is changed to blue (blue or cyan). This makes it easy to identify the position of the portion with a large degree of deviation, thereby further promoting positional alignment. Furthermore, in the case of changing the display method, a configuration can be adopted such that, for example, only the frame portion of the grid portion of the pseudo-image whose evaluation value based on the degree of deviation meets the predetermined value is displayed. Similar to the case of changing the color, this makes it easy to identify the position of the portion with a large degree of deviation.

[0150] As described above, in this embodiment, the degree of deviation is calculated based on the captured image and the pseudo-image, and an indicator based on the degree of deviation is further superimposed on the image obtained by superimposing the pseudo-image on the captured image. This allows for more accurate alignment of the camera device and the image.

[0151] Third Embodiment

[0152] Next, we will explain the problems that arise during the calibration process when visually aligning the image and camera positions. Furthermore, we will explain methods for instructing the user on the amount and direction of camera movement, and methods for reducing time and effort by utilizing the camera for final fine-tuning.

[0153] When visually aligning the image and camera, the user may not know how much further to move the camera from a misaligned state to achieve alignment. Therefore, the user needs to move the camera while visually checking the alignment between the image being captured and the image displayed on the LCD. Furthermore, even if the user visually judges that the camera and... Figure 1 However, there is also the possibility that minute misalignments, invisible to the LCD, may still exist. When these minute misalignments persist, it is conceivable that the lens data expected to be obtained during calibration cannot be acquired using concentric circles with the origin centered on the image, and there is a possibility that accurate lens information cannot be obtained. Furthermore, when users dedicate themselves to aligning the viewing angle to eliminate these minute misalignments and thus accurately align the viewing angle, the calibration process will be hampered due to the increased time spent on viewing angle alignment and other factors.

[0154] In response to the above issues, the following will be used Figures 11 to 19This describes methods for instructing users on the amount and direction of camera movement to align the image and camera in position, as well as methods for making final fine adjustments using the camera.

[0155] Figures 11 to 14 This is a flowchart illustrating the process of aligning the camera in position for calibration in this embodiment.

[0156] Figure 11 This illustrates the viewpoint alignment and calibration process during the calibration procedure.

[0157] First, when the user changes the camera 100 to a calibration mode for acquiring lens data for VFX compositing via the operation unit 142, the process begins. Figure 11 The processing.

[0158] In step S2101, the computing device 136 performs viewpoint alignment processing and proceeds to step S2102. The viewpoint alignment processing will be described in detail later.

[0159] In step S2102, the computing device 136 determines whether the viewpoint alignment process in step S2101 is completed, and if it determines that the viewpoint alignment process is completed, it ends the viewpoint alignment process and causes the process to proceed to step S2103.

[0160] Next, in step S2103, the computing device 136 performs the calibration process and ends the process.

[0161] Next, we will use Figure 12 The viewpoint alignment process in step S2101 is explained in detail.

[0162] First, in step S2201, the computing device 136 initializes the flag information used to determine whether the viewpoint alignment process is completed in step S2102 above to an incomplete state.

[0163] In step S2202, the computing device 136 reads the model data of the graph recorded in the graph information storage unit 139a. The model data corresponds to the pseudo image or reference image generated according to the graph information described in the first and second embodiments.

[0164] In step S2203, the computing device 136 calculates the difference representing the misalignment based on the model data of the image read in step S2202 and the image of the image captured by the camera. This will be used later. Figure 13A and Figure 13B Please explain the process in detail.

[0165] In step S2204, the computing device 136 calculates the direction and amount in which the camera 100 should be moved based on the misalignment calculated in step S2203. This will be used later. Figure 14 Please explain the process in detail.

[0166] In step S2205, the computing device 136 determines whether the misalignment amount calculated in step S2203 or the movement amount G of the camera 100 calculated based on the misalignment amount in step S2204 is less than a threshold. If it is determined to be less than the threshold, the process proceeds to step S2206; if it is determined to be greater than or equal to the threshold, the process proceeds to step S2209.

[0167] If the computing device 136 determines in step S2205 that the movement amount G of the camera 100 is less than a threshold, then proceed to step S2206. The case where the movement amount G of the camera 100 is small is handled when it is determined that any further fine adjustments to the camera position relative to the image made manually by the user would be difficult. Therefore, instead of having the user move the camera, the center position of the viewing angle is aligned within the camera itself.

[0168] Specifically, the computing device 136 calculates the driving amount of the image stabilizing lens 114 based on the difference calculated in step S2203, minimizing the difference between the image and the captured image. Then, the computing device 136 outputs an instruction to the lens control unit 121 via the electrical contact unit 150 to drive the image stabilizing lens 114. Upon receiving this instruction, the lens control unit 121 outputs a command to the image stabilizing drive unit 118 to drive the image stabilizing lens 114, and the image stabilizing drive unit 118 actually drives the image stabilizing lens 114. Alternatively, the computing device 136 calculates the driving amount of the image stabilizing drive unit 134 relative to the image sensor 131 based on the difference calculated in step S2203, and outputs a command to the image stabilizing control unit 135 to drive the image sensor 131. Then, the image stabilizing drive unit 134 actually drives the image sensor 131. Alternatively, both the image stabilizing lens 114 and the image sensor 131 can be driven. When driving these two components, it is desirable to drive them to a position that minimizes the data variation of the distortion, compared to the state of lenses 110 to 115 and image sensor 131 on the optical axis.

[0169] Since the computing device 136 achieves center alignment of the viewing angle inside the camera 100 in step S2206, in step S2207, the display control unit 141 displays on the display unit 140 that the user's viewing angle alignment operation has been completed.

[0170] In step S2208, the computing device 136 activates a flag indicating that the viewpoint alignment is complete and ends the process. Note that this flag is used to determine whether the viewpoint alignment is complete in step S2102.

[0171] If, in step S2205, it is determined that the misalignment amount is greater than or equal to a threshold, and there is still an amount that requires user-assisted viewpoint alignment, then step S2209 is performed. Therefore, the display control unit 141 displays on the display unit 140 that the user's viewpoint alignment is incomplete. As an example, a conceivable method involves displaying the vertical movement, horizontal movement, and direction of movement relative to the image in the movement amount G of the camera 100 calculated in step S2204.

[0172] As a result of the above processing, even when the camera 100 is significantly misaligned with the image, the user can move the camera 100 after confirming the amount and direction of movement. Furthermore, for minor misalignments that the user cannot eliminate, adjustments are no longer required, and the time spent on angle alignment during calibration is reduced.

[0173] Next, we will use Figure 13A The process for calculating the difference between the image and the captured image in step S2203 is explained in detail.

[0174] First, in step S2301, the computing device 136 instructs the image sensor control unit 133 to acquire image signals obtained from the image sensor 131 through the captured image at predetermined intervals, and sends the acquired image signals to the image processing unit 137. The image processing unit 137 performs appropriate image processing on the image signals and temporarily stores the resulting image signals in the volatile memory 138.

[0175] In step S2302, the computing device 136 instructs the image processing unit 137 to perform image processing (such as binarizing the image of the temporarily stored image) to facilitate the calculation of the difference in step S2304 described later. Note that the processing of the image signal is not limited to binarization.

[0176] Next, in step S2303, the computing device 136 performs image shifting processing on the image of the image that has undergone image processing.

[0177] Specifically, an example of a method for searching for the position that minimizes the difference between the captured image and the model data of the image read in step S2202 includes a method for calculating the minimum difference position while changing the comparison position between the captured image and the model data of the image. The difference is calculated by generating an image that is horizontally shifted N pixels to the left and vertically shifted N pixels upwards. Furthermore, if, as a result of this process, the difference calculation for the predetermined position determined later in step S2309 is not completed, the image is moved in a different manner than before (e.g., horizontally shifted N-1 pixels to the left and vertically shifted N pixels upwards), and the difference between the images is calculated again.

[0178] The shift range associated with the range of image shift and the shift step size associated with the step size (interval rejection amount) of shifting the shift range can be set in the camera or provided by the user. When these variables are set in the camera, a possible method involves: first setting a large shift step size and searching for the position with the smallest difference, then setting a smaller shift step size and searching for the position with the smallest difference again. In step S2303, the image processing unit 137 changes the set shift step size for the aforementioned shift range each time step S2303 is executed, extracts the image, and outputs the extracted image to the computing device 136.

[0179] Furthermore, as another example of a method for searching for the position with the smallest difference from the model data of the image read in step S2202, a method such as the following can be conceived. That is, each time the image captured while changing the position of the image stabilizing lens 114, is compared with the model data of the image, and the position of the image stabilizing lens 114 with the smallest difference is calculated. This will be used later. Figures 16A to 16D Explain the specific methods.

[0180] Furthermore, as another example of a method for searching for the position with the smallest difference from the model data of the image read in step S2202, a method such as the following can be conceived. That is, each time the image captured while changing the position of the image sensor 131 is compared with the model data of the image, and the position of the image sensor 131 with the smallest difference is calculated. This will be used later. Figures 17A to 17D Explain the specific methods.

[0181] Furthermore, in the case of implementing a method that calculates the minimum difference position while changing the comparison position of the captured image and the model data of the figure, the comparison can also be performed after the position is changed not only horizontally and vertically but also in the rotation direction.

[0182] It is possible to implement only one of the above-mentioned methods for changing the comparison position and rotation amount r of the model data of the captured image and the image, the method for changing the position of the image stabilizing lens 114, and the method for changing the position of the image sensor 131, or to combine multiple of these methods.

[0183] Next, in step S2304, the computing device 136 compares the image captured and generated in step S2303 with the model data of the graph, and calculates the difference S. This will be used... Figure 19 An example illustrating how the difference S is calculated.

[0184] Figure 19 In the accompanying drawings, reference numeral 2900 indicates the region representing the viewpoint, reference numeral 2901 indicates the model data of the graph, and reference numeral 2902 indicates the region of the graph in the captured image. Furthermore, the region indicated by reference numeral 2910 (the region where the upward and downward shading lines overlap) is the region where the black portion of the model data 2901 of the graph overlaps with the black portion of the captured image 2902, and is calculated as a region without difference. Similarly, the region indicated by reference numeral 2911 (the blank region) is the region where the white portion of the model data 2901 of the graph overlaps with the white portion of the captured image 2902, and is calculated as a region without difference. The region indicated by reference numeral 2912 (the region with only upward shading lines) is the region where the black portion of the model data 2901 of the graph overlaps with the white portion of the captured image 2902, and is calculated as a region with difference. Similarly, the area indicated by reference numeral 2913 (the area with only downward shading) is the area where the white portion of the model data 2901 of the figure overlaps with the black portion of the captured image 2902, and is the portion that is calculated as the difference region.

[0185] Note that in the above description, the difference region is calculated based on whether the colors are the same. However, the following method can be used: provide each grid part of the graph with an identifier such as an ID, and calculate the difference region by whether areas of the same color with the same ID overlap.

[0186] Next, in step S2305, the computing device 136 compares the area of ​​each difference region calculated by the above technique with the stored minimum difference value. If the difference S is less than the minimum difference value, the computing device 136 causes the processing to proceed to step S2306, and if the difference S is greater than or equal to the minimum difference value, the computing device 136 causes the processing to proceed to step S2309.

[0187] In step S2306, the computing device 136 saves the difference S back as the minimum difference value.

[0188] In step S2307, the computing device 136 stores the shift position with the smallest difference S. Specifically, the computing device 136 stores the pixel movement p of the image with the smallest difference, the driving amount ω of the image stabilizing lens 114, and the movement amount i of the image sensor 131 as the shift amount.

[0189] In step S2308, the computing device 136 instructs the display control unit 141 to perform a process for combining the model data of the image with the captured image, and displays the combined image on the display unit 140. The user visually observes the misalignment between the model data of the image displayed on the display unit 140 and the captured image, and moves the camera to correct the misalignment.

[0190] In step S2309, the computing device 136 determines whether the search for all search positions to be searched in step S2303 has ended. Then, if it is determined that all searches have ended, the computing device 136 ends the process, and if it is determined that there are still search positions to be searched, the computing device 136 returns the process to step S2301.

[0191] As a result of the above processing, the misalignment between the image and the camera can be calculated based on the difference between the model data of the image and the captured image of the image.

[0192] Next, we will use the technology for... Figure 14 Detailed explanation of the camera movement calculation process in step S2204.

[0193] First, in step S2401, the computing device 136 uses the lens control unit 121 to send and receive commands, and acquires the focal length information f of the lens and the drive amount ω of the image stabilizing lens 114. Furthermore, the computing device 136 acquires the position information of the focusing lens 115, information related to the drive states of various lenses, and information related to the state of the aperture 113.

[0194] In step S2402, the computing device 136 acquires the subject distance information d. The subject distance information d can be calculated based on the position information of the focusing lens 115 after focusing, or it can be acquired through user input. Alternatively, if the image sensor 131 is an image sensor capable of calculating the defocus amount using an image plane phase detection method, the subject distance information d can be calculated based on the position information of the focusing lens 115 and the defocus amount.

[0195] In step S2403, the computing device 136 uses the acquired subject distance information d and focal length information f of the lens, as well as the information obtained in the previous step. Figure 12The pixel movement p, rotation r, drive amount ω of image stabilizing lens 114, and movement amount i of image sensor 131 are calculated in the perspective alignment process to calculate the movement amount G and rotation amount as the amount that causes camera 100 to move, and the process ends.

[0196] Next, it will be explained in Figure 13A Example of calculating the movement amount G of camera 100 in step S2303 of the captured image shift processing. Figures 15A to 15D This illustrates the case where displacement processing is performed by moving a virtual image inside the camera 100 and the displacement amount G is calculated. Figures 16A to 16D This illustrates the use of image stabilizing lens 114 for displacement processing and calculation of the displacement G. Figures 17A to 17D The diagram illustrates the use of image sensor 131 for shift processing and calculation of the shift amount G. Note that in these figures, identical or similar structures are given the same reference numerals, and redundant descriptions are omitted.

[0197] exist Figure 15A In the diagram, reference numeral 2500 indicates the graph used in calibration, and point O indicates the center position of the graph. Reference numeral 2501 indicates the optical axis of camera 100. Reference numeral 2502 indicates the field of view of camera 100. One objective of this embodiment is to calculate the offset distance G between point O and optical axis 2501 and to present the offset distance (movement) G to the user.

[0198] Figure 15B Showing when in Figure 15A The display on the LCD screen when recording video under certain conditions.

[0199] Model data 2901, used as a target for aligning the image with the viewpoint 2900, is displayed in the center of the screen. Furthermore, in Figure 15A In the middle, camera 100 is slightly misaligned to the right relative to Figure 2500, therefore in Figure 15B In the image, Figure 2902 is composited with a slight misalignment to the left.

[0200] Figure 15C This is a diagram illustrating an example of the image shifting process performed each time in step S2303. If the image is not shifted, something like... Figure 15B The misalignment shown, and in the first iteration of the processing in step S2303, the captured image is virtually shifted in the upper left direction and synthesized with the model data of the image, which leads to, for example, Figure 15C The misalignment is shown in the upper left figure. In this state, the difference is calculated and compared with the minimum difference value, and if necessary, the minimum difference value and the shift amount that minimizes the difference are updated.

[0201] Next, as the second iteration of the process, the captured image is virtually shifted in the upper right direction and synthesized with the model data of the graph, which results in, for example... Figure 15C The misalignment is shown in the upper right figure. In this state, the difference is calculated and compared with the minimum difference value, and if necessary, the minimum difference value and the shift amount that minimizes the difference are updated.

[0202] Next, as the third iteration of the process, the captured image is virtually shifted in the lower left direction and synthesized with the model data of the graph, which results in, for example... Figure 15C The misalignment is shown in the lower left figure. In this state, the difference is calculated and compared with the minimum difference value, and if necessary, the minimum difference value and the shift amount that minimizes the difference are updated.

[0203] Next, as the fourth iteration of the process, the captured image is virtually shifted in the lower right direction and synthesized with the model data of the graph, which results in, for example... Figure 15C The misalignment is shown in the lower right figure. In this state, the difference is calculated and compared with the minimum difference value, and if necessary, the minimum difference value and the shift amount that minimizes the difference are updated.

[0204] Perform the above processing, and calculate the deviation distance G when the pixel movement amount corresponding to the recorded minimum difference is p. This can be based on... Figure 15D The deviation distance G can be calculated using the following formula.

[0205] G=(d×p) / f

[0206] Next, we will use Figures 16A to 16D This describes a method for calculating the deviation distance G by driving the image stabilizing lens 114.

[0207] exist Figure 16A In the figure, reference numeral 2503 denotes the optical axis when the image stabilizing lens 114 has been moved. As an example, a pattern is illustrated where the optical axis 2503 passes through point O and the correction angle at this time is ωdeg. Furthermore, reference numeral 2504 denotes a line representing the changed viewing angle when the image stabilizing lens 114 has been moved.

[0208] Figure 16B The example is Figure 16A The image stabilizing lens 114 is driven positively when recording video under certain conditions. Figure 16B The left image shows the image stabilizing lens 114 maintaining the center 2505 of the optical axis. On the other hand, Figure 16B The right figure shows the image stabilizing lens 114 shifted ωdeg to the left relative to the center 2505 of the optical axis. Figure 16B The image below shows the display on the LCD screen when recording video under the above conditions.

[0209] Model data 2901, used as a target for aligning the image with the viewpoint 2900, is displayed in the center of the screen. Furthermore, in Figure 16A and Figure 16B In the left image, camera 100 is slightly misaligned to the right relative to Figure 2500, therefore Figure 2902 in the captured image is composited with a slight misalignment to the left. On the other hand, in Figure 16A and Figure 16B In the right figure, the optical axis 2503 of the camera 100 passes through point O, so the center of the captured image 2902 coincides with the center of the model data 2901 of the figure.

[0210] Figure 16C This is a diagram illustrating an example of the image shifting process performed each time in step S2303. If the image stabilizing lens 114 is not shifted, something such as... Figure 16B The misalignment shown in the left image, and the fact that the captured image is shifted in the upper left direction and synthesized with the model data of the image in the first iteration of the processing in step S2303, leads to issues such as... Figure 16C The top left figure shows misalignment. In this state, the difference is calculated and compared with the minimum difference value, and if necessary, the minimum difference value and the displacement of the image stabilizing lens 114 that minimizes the difference are updated.

[0211] Next, as the second iteration of the process, the captured image is shifted in the upper right direction and synthesized with the model data of the graph, which results in, for example... Figure 16C The misalignment is shown in the upper right figure. In this state, the difference is calculated and compared with the minimum difference value, and if necessary, the minimum difference value and the displacement of the image stabilizing lens 114 that minimizes the difference are updated.

[0212] Next, as the third iteration of the process, the captured image is shifted in the lower left direction and synthesized with the model data of the graph, which results in, for example... Figure 16C The misalignment is shown in the lower left figure. In this state, the difference is calculated and compared with the minimum difference value, and if necessary, the minimum difference value and the displacement of the image stabilizing lens 114 that minimizes the difference are updated.

[0213] Next, as the fourth iteration of the process, the captured image is shifted in the lower right direction and synthesized with the model data of the graph, which results in, for example... Figure 16C The misalignment is shown in the lower right figure. In this state, the difference is calculated and compared with the minimum difference value, and if necessary, the minimum difference value and the displacement of the image stabilizing lens 114 that minimizes the difference are updated.

[0214] The above processing is performed, and the deviation distance G is calculated when the driving amount of the image stabilizing lens 114 corresponding to the recorded minimum difference is ω. This can be based on... Figure 16D The deviation distance G can be calculated using the following formula.

[0215] G=d·tanω

[0216] Note that in the above method of driving the image stabilizing lens 114, unlike the method described above where the image is virtually shifted inside the camera, the actually captured image moves in conjunction with the driving of the image stabilizing lens 114. If the user attempts to align the camera with the center of the image while it is moving, it is conceivable that even if the direction and distance of movement are displayed, it would be difficult to align the camera with the center. Therefore, when calculating the difference by driving the image stabilizing lens 114, the image is only processed in step S2308 when the image stabilizing lens 114 is at the optical center. Alternatively, by performing the composite processing on an image obtained by virtually restoring the captured image by an amount equivalent to the amount of shift of the image stabilizing lens 114, the difficulty for the user in aligning the camera is avoided.

[0217] Next, we will use Figures 17A to 17D This describes a method for calculating the deviation distance G by driving the image sensor 131.

[0218] exist Figure 17A In the figure, reference numeral 2506 indicates the optical axis when the image sensor 131 has been moved. As an example, a pattern is illustrated where the optical axis 2506 passes through point O and the correction amount of the image sensor 131 is i mm at this time. Furthermore, reference numeral 2507 indicates the changed viewing angle when the image sensor 131 has been moved.

[0219] Figure 17B The example is Figure 17A The image sensor 131 is driven when taking pictures under the condition of shooting. Figure 17B The left image shows the image sensor 131 maintaining the center 2508 of the optical axis. On the other hand, Figure 17B The right figure shows the image sensor 131 shifted imm to the right relative to the center 2508 of the optical axis. Furthermore, Figure 17B The image below shows the display on the LCD screen when recording video under the above conditions.

[0220] Model data 2901, used as a target for aligning the image with the viewpoint 2900, is displayed in the center of the screen. Furthermore, in Figure 17A and Figure 17BIn the left image, camera 100 is slightly misaligned to the right relative to Figure 2500, therefore Figure 2902 in the captured image is composited with a slight misalignment to the left. On the other hand, in Figure 17A and Figure 17B In the right figure, the optical axis 2506 of the camera 100 passes through point O, so the center of the captured image 2902 coincides with the center of the model data 2901 of the figure.

[0221] Figure 17C This is a diagram illustrating an example of the image shifting process performed each time in step S2303. If the image sensor 131 is not shifted, something such as... Figure 17B The misalignment shown in the left image, and the fact that the captured image is shifted in the upper left direction and synthesized with the model data of the image in the first iteration of the processing in step S2303, leads to issues such as... Figure 17C The misalignment is shown in the upper left figure. In this state, the difference is calculated and compared with the minimum difference value, and if necessary, the minimum difference value and the displacement of the image sensor 131 that minimizes the difference are updated.

[0222] Next, as the second iteration of the process, the captured image is shifted in the upper right direction and synthesized with the model data of the graph, which results in, for example... Figure 17C The misalignment is shown in the upper right figure. In this state, the difference is calculated and compared with the minimum difference value, and if necessary, the minimum difference value and the displacement of the image sensor 131 that minimizes the difference are updated.

[0223] Next, as the third iteration of the process, the captured image is shifted in the lower left direction and synthesized with the model data of the graph, which results in, for example... Figure 17C The misalignment is shown in the lower left figure. In this state, the difference is calculated and compared with the minimum difference value, and if necessary, the minimum difference value and the displacement of the imaging element 131 that minimizes the difference are updated.

[0224] Next, as the fourth iteration of the process, the captured image is shifted in the lower right direction and synthesized with the model data of the graph, which results in, for example... Figure 17C The misalignment is shown in the lower right figure. In this state, the difference is calculated and compared with the minimum difference value, and if necessary, the minimum difference value and the displacement of the image sensor 131 that minimizes the difference are updated.

[0225] The above processing is performed, and the deviation distance G is calculated when the driving amount of the image sensor 131 corresponding to the recorded minimum difference is i. This can be based on... Figure 17D The deviation distance G can be calculated using the following formula.

[0226] G=(d×i) / f

[0227] Note that in the above-described method of driving the image sensor 131, unlike the method described above which virtually shifts the image inside the camera, the actually captured image moves in conjunction with the driving of the image sensor 131. If the user attempts to align the camera with the center of the image while it is moving, even if the direction and distance of movement are displayed, it is conceivable that it would be difficult to align the camera with the center. Therefore, when calculating the difference by driving the image sensor 131, the image is only processed in step S2308 when the image sensor 131 is at the optical center. Alternatively, by performing the composite processing on an image obtained by virtually restoring the captured image by an amount equivalent to the amount of shift of the image sensor 131, the difficulty for the user in aligning the camera is avoided.

[0228] By performing the above processing, the offset distance G between the camera and the center of the image can be calculated, and the amount of camera movement can be notified to the user. Therefore, the time and effort required for angle alignment during calibration can be reduced.

[0229] Note that, for ease of understanding, the shift is described above as being performed four times, but in reality, more than four shifts can be performed.

[0230] Next, we will use Figure 18A and Figure 18B This section describes an example of a method that displays the amount of movement G (vertical / horizontal movement and direction of movement) of the camera 100 in steps S2209 and S2207 until the user's viewpoint is aligned.

[0231] Figure 18A The display shown on the display unit 140 is given when the misalignment between the camera 100 and the calibration map is determined to be greater than or equal to a threshold.

[0232] Figure 18A In the accompanying figures, reference numeral 2800 indicates the area representing the viewpoint, reference numeral 2801 indicates the model data of the figure, and reference numeral 2802 indicates the area of ​​the figure in the captured image. Figure 18A In this image, if the misalignment between the camera 100 and the image is greater than or equal to a threshold, then the image 2802 in the captured image will appear at a position misaligned with the image model data 2801. Furthermore, reference numeral 2803 indicates a display example where, if the misalignment between the camera 100 and the image in the captured image is determined to be greater than or equal to a threshold (greater than or equal to a predetermined value), the user is informed of the amount and direction of camera movement. Display 2803 indicates that when the camera moves 54mm to the right and 23mm downwards, the center of the image model data 2801 will be aligned with the center of the image in the captured image.

[0233] Figure 18BThe display shown on the display unit 140 is given when the misalignment between the camera 100 and the image is determined to be less than a threshold.

[0234] Figure 18B Display 2804 indicates that the misalignment between camera 100 and the captured image has been eliminated. The user ends the camera position movement based on display 2804. On the other hand, the complete elimination of misalignment between camera 100 and the captured image is rare, therefore, as described in step S2206 above, the image stabilizing lens 114 or image sensor 131 is driven (shifted) to the position with the smallest difference. As a result, the difference between the model data 2801 of the image and the region 2802 of the image in the captured image is minimized, and the position alignment of camera 100 can be completed without the user needing to perform minor position alignment.

[0235] Compared to visually aligning the image and camera in position, the above technique makes it easier for users to understand how much and in which direction the camera should be moved, thus reducing the time spent on position alignment.

[0236] Furthermore, even when minute misalignments invisible to the LCD still exist, a further reduction in the time spent on position alignment is achieved by correcting the misalignment by driving the image stabilizing lens 114 or the image sensor 131.

[0237] Note that the pattern of the graph is not limited to formats such as those described above, and can be any pattern that represents the horizontal and vertical directions of the graph at predetermined intervals. For example, the graph can be an interlaced grid pattern (where adjacent black and white rectangles alternate in rectangular areas divided into grid shapes), or the graph can be a barcode-like pattern.

[0238] Furthermore, as described in the first embodiment above, the display or difference calculation can be performed after the model data of the graph has been distorted based on the distortion and tilt-shift amount temporarily calculated from the captured image.

[0239] In addition, the magnification of the model data can be changed according to the focal length, and the display or difference calculation can be performed.

[0240] Fourth embodiment

[0241] Will use Figure 13B The following situations are explained: In Figure 13A In the processing for calculating the difference between the image and the captured image, the camera 100 is significantly misaligned with the image 2500, and only one of the methods involving virtually moving the image, moving the image stabilizing lens 114, and moving the image sensor 131 cannot calculate the camera movement G. Note that in Figure 13B In, with Figure 13A The same process is assigned the same step number, and its description will be omitted.

[0242] If, as a result of the processing in steps S2301 to S2309, the shift position with the smallest difference is determined to be located at the edge of the search range, there is a possibility that the shift position with the smallest difference is outside the search range. Therefore, the following processing mode is used to further expand the search range.

[0243] In the case of image shifting processing in step S2303, the search range is expanded by driving the image stabilizing lens 114 or the image sensor 131 to a position where the image is shifted in the direction of minimum difference.

[0244] Furthermore, in the case of using the image stabilizing lens 114 for shift processing in the captured image shift processing in step S2303, the search range is expanded by driving the image sensor 131 to a position where the image is shifted in the direction of minimum difference.

[0245] Furthermore, in the case of using the image sensor 131 for image shifting processing in step S2303, the search range is expanded by driving the image stabilizing lens 114 to a position where the image is shifted in the direction of minimum difference.

[0246] exist Figure 13B In step S2321, the above determination is made, and in step S2322, the position is fixed by driving the image stabilizing lens 114 or the image sensor 131 according to the above pattern. After that, the process returns to step S2301, and the shift position with the smallest difference is calculated again.

[0247] Note that in the case of shift processing by image movement during image shift processing, the search range can be further expanded by driving both the image stabilizing lens 114 and the image sensor 131.

[0248] Using the above method, even when the positions of camera 100 and the calibration-used figure 2500 are greatly misaligned, the direction and amount that camera 100 should move can be calculated and presented, thereby reducing the user's workload in aligning the viewpoint.

[0249] Other embodiments

[0250] The embodiments of the present invention can also be implemented by providing software (programs) that perform the functions of the above embodiments to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessor unit (MPU) of the system or device reads out and executes the program.

[0251] Although the invention has been described with reference to exemplary embodiments, it should be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the appended claims is to be interpreted in the broadest sense to include all such modifications, equivalent structures, and functions.

Claims

1. An information processing device, comprising: The first acquisition unit is configured to acquire a captured image obtained by capturing an image of a calibration image; The second acquisition unit is configured to acquire a reference image, which is an image of the graph used as a reference; The third acquisition component is configured to acquire information related to the distortion aberration of the lens used to capture the captured image; A first generating unit is configured to generate a pseudo image based on the reference image and information related to the distortion aberration, the pseudo image being an image obtained by reflecting the distortion aberration in the reference image; The second generation component is configured to generate a composite image obtained by combining the captured image and the pseudo image; A calculation unit is configured to calculate the degree of deviation of an image based on the captured image and the pseudo image, and further calculate an evaluation value based on the degree of deviation; as well as A third generating component is configured to generate an indicator for moving a camera device equipped with the lens, based on the evaluation value. The calculation unit limits the range of deviations used to calculate the degree of deviation in the captured image and the pseudo-image based on the distortion aberration of the lens or the image height of the captured image.

2. The information processing device according to claim 1 further includes an output component, the output component being configured to output the composite image to a display component.

3. The information processing device according to claim 2, further comprising the display component.

4. The information processing device according to claim 1, in, Information related to the distortion aberration includes at least one of the following: distortion aberration-related information estimated based on the captured image and the reference image, distortion aberration-related information stored in the lens, and distortion aberration-related information stored in the camera device on which the lens is mounted.

5. The information processing device according to claim 1, in, The first generating component magnifies or reduces the reference image based on the focal length of the lens.

6. The information processing device according to claim 1, in, The degree of deviation includes at least one of the following vectors: a vector representing the distance and direction between a feature point of the captured image and a corresponding feature point of the pseudo image, and a vector representing the distance and direction between the centroid of a specific region of the captured image and the centroid of a corresponding specific region of the pseudo image.

7. The information processing device according to claim 1, in, The second generating component changes the color or form of a portion of the pseudo-image synthesized with the captured image based on the degree of deviation.

8. The information processing device according to claim 1, further comprising: A storage unit is configured to store information related to the lens, information related to the lens's distortion aberrations, the degree of deviation, and the evaluation value in association with the captured image.

9. The information processing device according to claim 1 further includes a control component, the control component being configured to control the automatic capture of an image when the evaluation value meets a predetermined condition.

10. The information processing apparatus of claim 1, further comprising a fourth generating component configured to generate information of the image, wherein the fourth generating component generates the information of the image by detecting information required to generate the information of the image from the captured image.

11. The information processing device according to claim 1, further comprising: A computing unit is configured to calculate the difference between the captured image and the pseudo image; as well as A control unit is configured to, when the difference between the captured image and the pseudo image is greater than or equal to a predetermined value, control the display of the movement of the camera device with the lens mounted on it on a display unit, and when the difference is less than the predetermined value, cause a shifting mechanism to perform a shifting operation to shift the captured image and the pseudo image relative to each other to reduce the difference.

12. The information processing device according to claim 11, in, The shifting mechanism includes at least one of the following: a mechanism configured to perform the shifting operation by driving an image stabilizing lens provided in the lens, and a mechanism configured to perform the shifting operation by driving an image sensor for capturing the captured image.

13. The information processing device according to claim 11, in, The computing unit also calculates the relative rotation of the captured image and the pseudo image.

14. The information processing device according to claim 12, in, When the shifting operation is performed by driving the image stabilizing lens or by driving the image sensor, the control unit controls the second generating unit to generate the composite image based on the image captured before the shifting operation.

15. A camera device, comprising: Lens; An image sensor, configured to capture images; as well as The information processing device according to claim 1.

16. An information processing method, comprising: Perform a first acquisition, which is used to acquire a captured image obtained by capturing an image of a calibration image; Perform a second acquisition, the second acquisition being used to acquire a reference image, the reference image being an image of the graph used as a reference; Perform a third acquisition, the third acquisition being used to acquire information related to the distortion aberrations of the lens used to capture the captured image; Perform a first generation, which is used to generate a pseudo image based on the reference image and information related to the distortion aberration, the pseudo image being an image obtained by reflecting the distortion aberration in the reference image; Perform a second generation, which is used to generate a composite image obtained by combining the captured image and the pseudo image; The degree of image deviation is calculated based on the captured image and the pseudo image, and an evaluation value is further calculated based on the degree of deviation. as well as A third generation is performed, which is used to generate an indicator for moving the camera device equipped with the lens based on the evaluation value. In the calculation, the range of deviations used to calculate the degree of deviation in the captured image and the pseudo-image is limited based on the distortion aberration of the lens or the image height of the captured image.

17. A non-transitory computer-readable storage medium storing a computer program for causing a computer to perform the steps of an information processing method, the information processing method comprising: Perform a first acquisition, which is used to acquire a captured image obtained by capturing an image of a calibration image; Perform a second acquisition, the second acquisition being used to acquire a reference image, the reference image being an image of the graph used as a reference; Perform a third acquisition, the third acquisition being used to acquire information related to the distortion aberrations of the lens used to capture the captured image; Perform a first generation, which is used to generate a pseudo image based on the reference image and information related to the distortion aberration, the pseudo image being an image obtained by reflecting the distortion aberration in the reference image; Perform a second generation, which is used to generate a composite image obtained by combining the captured image and the pseudo image; The degree of image deviation is calculated based on the captured image and the pseudo image, and an evaluation value is further calculated based on the degree of deviation. as well as A third generation is performed, which is used to generate an indicator for moving the camera device equipped with the lens based on the evaluation value. In the calculation, the range of deviations used to calculate the degree of deviation in the captured image and the pseudo-image is limited based on the distortion aberration of the lens or the image height of the captured image.

18. A computer program product comprising a computer program that, when executed by a computer, implements the steps of the method of claim 16.

Images