Interchangeable lens, information processing device, information processing method and program

Abstract

The present technology relates to an interchangeable lens, an information processing device, an information processing method and a program capable of performing appropriate processing. A multi-eye interchangeable lens having a lens barrel, a movable part, a plurality of single-eye lenses and a light source. The movable part is configured to be movable along an optical axis relative to the lens barrel. A plurality of single-eye lenses are integrated with the movable part to be movable, and are arranged so that the emission positions of image capturing light beams emitted by each single-eye lens do not overlap. One or more light sources are integrated with a plurality of single-eye lenses movable along the optical axis and the movable part, and are arranged so that the emission position of a parallel light beam irradiated on an image sensor provided in a camera body does not overlap with the emission position of the image capturing light beam emitted by each single-eye lens. The present technology can be applied to, for example, a camera system to which an interchangeable lens or a plurality of single-eye lenses are fed.

Description

Technical Field

[0001] The present technology relates to an interchangeable lens, an information processing apparatus, an information processing method, and a program, and in particular, to an interchangeable lens, an information processing apparatus, an information processing method, and a program for enabling appropriate processing, for example. Background Art

[0002] There is proposed a technology for improving the convenience of a service using a multi-viewpoint image including a plurality of images having viewpoints different from each other (for example, see Patent Document 1).

[0003] Citation List

[0004] Patent Literature

[0005] Patent Document 1: International Publication No. 2015 / 037472 Summary of the invention

[0006] Problems to be solved by the present invention

[0007] For example, the multi-viewpoint images may be captured by a camera system including a monocular lens, which is a plurality of lenses arranged so as not to overlap each other in the optical axis direction.

[0008] However, in a camera system, when multiple monocular lenses are sent out along the optical axis direction and the focus is adjusted, etc., the area of ​​the monocular image corresponding to the image formed by the light beam focused by the monocular lens on the captured image captured by the image sensor may change before and after the sending.

[0009] When the area of ​​the monocular image changes, it may be difficult to appropriately perform processing in the camera system.

[0010] The present technology has been proposed in view of the above-mentioned circumstances, and enables appropriate processing.

[0011] Solutions to the Problem

[0012] The interchangeable lens of the present technology is an interchangeable lens as follows, comprising: a lens barrel; a movable unit configured to be movable along an optical axis relative to the lens barrel; a plurality of single-eye lenses configured to be movable integrally with the movable unit and arranged so that emission positions of imaging light emitted by the respective single-eye lenses do not overlap with each other; and one or more light sources configured to be movable integrally with the movable unit and the plurality of single-eye lenses along the optical axis and arranged so that an emission position of parallel light emitted to an image sensor provided in a camera body does not overlap with an emission position of imaging light of each of the plurality of single-eye lenses.

[0013] In the interchangeable lens of the present technology, a movable unit is configured to be movable along the optical axis relative to the lens barrel, a plurality of single-eye lenses are configured to be movable integrally with the movable unit, and emission positions of imaging light emitted by the respective single-eye lenses are arranged so as not to overlap with each other. In addition, one or more light sources are configured to be movable along the optical axis integrally with the movable unit and the plurality of single-eye lenses, and are arranged so that the emission position of parallel light emitted to an image sensor provided in a camera body does not overlap with the emission position of imaging light of each of the plurality of single-eye lenses.

[0014] The information processing device or program of the present technology is an information processing device including the following units or a program for causing a computer to operate as such an information processing device, the information processing device including: a detection unit configured to detect a light image on a captured image captured by an image sensor, the light image being a light image of parallel light emitted from a light source of a lens unit, the lens unit including: a lens barrel; a movable unit configured to be movable along an optical axis relative to the lens barrel; a plurality of single-eye lenses configured to be movable integrally with the movable unit and arranged so that emission positions of imaging light emitted by the respective single-eye lenses do not overlap with each other; and one or more light sources configured to be movable integrally with the movable unit and the plurality of single-eye lenses along the optical axis and arranged so that emission positions of parallel light emitted to an image sensor provided in a camera body do not overlap with emission positions of imaging light of each of the plurality of single-eye lenses; and a processing unit configured to perform processing according to a detection result of the detection unit.

[0015] The information processing method of the present technology is an information processing method, comprising: a detection step of detecting a light image on a captured image captured by an image sensor, the light image being a light image of parallel light emitted from a light source of a lens unit, the lens unit comprising: a lens barrel; a movable unit configured to be movable along an optical axis relative to the lens barrel; a plurality of single-eye lenses configured to be movable integrally with the movable unit and arranged so that emission positions of imaging light emitted through the respective single-eye lenses do not overlap with each other; and one or more light sources configured to be movable integrally with the movable unit and the plurality of single-eye lenses along the optical axis and arranged so that emission positions of parallel light emitted to an image sensor provided in a camera body do not overlap with emission positions of imaging light of each of the plurality of single-eye lenses; and a processing step of performing processing according to the detection result of the detection step.

[0016] In the information processing device, information processing method and program of the present technology, a light image on a captured image captured by an image sensor is detected, the light image being a light image of parallel light emitted from a light source of a lens unit, the lens unit comprising: a lens barrel; a movable unit configured to be movable along an optical axis relative to the lens barrel; a plurality of single-eye lenses configured to be movable integrally with the movable unit and arranged so that emission positions of imaging light emitted through the respective single-eye lenses do not overlap with each other; and one or more light sources configured to be movable integrally with the movable unit and the plurality of single-eye lenses along the optical axis and arranged so that emission positions of parallel light emitted to an image sensor provided in a camera body do not overlap with emission positions of imaging light of each of the plurality of single-eye lenses, and processing is performed according to the detection result.

[0017] Note that the information processing device may be an independent device, or may be an internal block configuring one device.

[0018] Furthermore, the program can be provided by being transmitted via a transmission medium or by being recorded on a recording medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a perspective view showing a configuration example of an embodiment of a camera system to which the present technology is applied.

[0020] Figure 2 is a block diagram showing an electrical configuration example of the camera system 1 .

[0021] Figure 3 2 is a diagram for describing an outline of imaging of a captured image performed using the poly-eye interchangeable lens 20 .

[0022] Figure 4 3 is a diagram showing an example of arrangement of the single-eye lenses 310 to 314 and the light sources 32L and 32R in the polycular interchangeable lens 20 , and a captured image captured using the polycular interchangeable lens 20 .

[0023] Figure 5 is a diagram for describing an attachment error when the poly-interchangeable lens 20 is attached (mounted) to the camera body 10 .

[0024] Figure 6 It is a diagram for describing a calculation method of obtaining relative optical axis center positions (dx1′, dy1′) to (dx4′, dy4′) as mounting error reflecting position information.

[0025] Figure 7 : is a block diagram showing a configuration example of the image processing unit 53 .

[0026] Figure 8 is a diagram for describing calibration performed by the camera system 1 .

[0027] Fig. 9 It is a diagram for describing generation of calibration data of a plurality of feed amounts corresponding to a plurality of reference focus positions.

[0028] Fig.10 is a diagram for describing general imaging performed by the camera system 1 .

[0029] Fig.11 3 is a diagram for describing generation, by interpolation, of calibration data for an imaging feed amount corresponding to the imaging focus position P4.

[0030] Fig.12 : is a cross-sectional view showing a configuration example of the light sources 32L and 32R.

[0031] Fig.13 2 is a cross-sectional view showing a configuration example of the polycular interchangeable lens 20 .

[0032] Fig.14 2 is a diagram for describing a first detection method of detecting a feeding amount of the feeding unit 23 .

[0033] Fig.15 : is a diagram showing an example of change in the spot size of non-parallel light which is spot light.

[0034] Fig.16 is a flowchart for describing an example of a process of detecting a feed amount by a first detection method.

[0035] Fig.17 2 is a diagram for describing a second detection method of detecting the feeding amount of the feeding unit 23 .

[0036] Fig.18 is a flowchart for describing an example of a process of detecting a feed amount by the second detection method.

[0037] Fig.19 2 is a diagram for describing a third detection method of detecting the feeding amount of the feeding unit 23 .

[0038] Fig. 20 1 is a diagram for describing a state in which a focusing point where non-parallel light as point light is focused is located at one of the front side and the depth side including the image sensor 51 when the feeding unit 23 moves from the minimum feeding state to the maximum feeding state.

[0039] Fig.21 2 is a diagram for describing a fourth detection method of detecting the feeding amount of the feeding unit 23 .

[0040] Fig. 221 is a diagram showing the irradiation position of the spot light when the feeding unit 23 is in the minimum feeding state and the irradiation position of the spot light when the feeding unit 23 is in the maximum feeding state.

[0041] Fig.23 is a diagram showing an example of a captured image in which the point light images PL′ and PR′ appear when the feed unit 23 is in the minimum feed state, and a captured image in which the point light images PL″ and PR″ appear when the feed unit 23 is in the maximum feed state.

[0042] Fig.24 is a flowchart for describing an example of a process of detecting a feed amount by a fourth detection method.

[0043] Fig.25 is a diagram showing another configuration example of the polycular interchangeable lens 20 .

[0044] Fig.26 32L and 32R is a cross-sectional view showing another configuration example of the light sources 32L and 32R.

[0045] Fig. 27 3 is a diagram showing a state in which the position of a point light image changes as the lens is tilted.

[0046] Fig.28 is a block diagram showing another electrical configuration example of the camera system 1 .

[0047] Fig.29 is a block diagram showing a configuration example of a post-processing device that performs post-processing on association information.

[0048] Fig.30 is a block diagram showing another configuration example of a post-processing device that performs post-processing on association information.

[0049] Fig.31 is a block diagram showing an electrical configuration example of a first another embodiment of a camera system to which the present technology is applied.

[0050] Fig.32 is a block diagram showing an electrical configuration example of a second another embodiment of a camera system to which the present technology is applied.

[0051] Fig.33 is a block diagram showing a configuration example of an embodiment of a computer to which the present technology is applied. DETAILED DESCRIPTION

[0052] <Embodiment of a camera system to which the present technology is applied>

[0053] Figure 1 is a perspective view showing a configuration example of an embodiment of a camera system (imaging device) to which the present technology is applied.

[0054] The camera system 1 includes a camera body 10 and a poly-interchangeable lens 20 (lens unit).

[0055] The poly-eye interchangeable lens 20 is attachable to and detachable from the camera body 10. That is, the camera body 10 includes a camera mount 11, and the poly-eye interchangeable lens 20 (the lens mount 22 thereof) is fixed (attached) to the camera mount 11 so that the poly-eye interchangeable lens 20 is mounted on the camera body 10. Note that a general interchangeable lens other than the poly-eye interchangeable lens 20 can be attached to and detached from the camera body 10.

[0056] The camera body 10 includes an image sensor 51. The image sensor 51 is, for example, a complementary metal oxide semiconductor (CMOS) image sensor, and forms an image by receiving a light beam collected by the multi-eye interchangeable lens 20 or other interchangeable lenses mounted on (the camera mount 11 of) the camera body 10 and performing photoelectric conversion. Hereinafter, an image obtained by imaging with the image sensor 51 is also referred to as a captured image.

[0057] The multi-eye interchangeable lens 20 includes a lens barrel 21, a lens mount 22, and a feed unit 23. The feed unit 23 is a movable unit configured to be movable relative to the lens barrel 21 along the optical axis of the lens barrel 21. The feed unit 23 includes five single-eye lenses 310, 311, 312, 313, and 314 as a plurality of single-eye lenses. i The feeding unit 23 is configured to be movable integrally with the feeding unit 23, and is arranged so that the feeding unit 23 is i The emission positions of the emitted imaging lights do not overlap with each other. In addition, the feeding unit 23 includes light sources 32L and 32R. The light sources 32L and 32R are configured to be aligned with the feeding unit 23 and the plurality of single-eye lenses 31. i The lens barrel 21 is integrally movable along the optical axis and is arranged so that the emission position of the illumination light emitted to the image sensor 51 provided in the camera body 10 does not overlap with the plurality of single-eye lenses 31. i The emission position of each imaging light overlaps.

[0058] The lens barrel 21 has a substantially cylindrical shape, and a lens mount 22 is formed on one bottom surface side of the cylindrical shape.

[0059] When the poly-interchangeable lens 20 is mounted on the camera body 10 , the lens mount 22 is fixed (attached) to the camera mount 11 of the camera body 10 .

[0060] The feeding unit 23 has a substantially cylindrical shape, and is accommodated in the cylindrical lens barrel 21 .

[0061] The feeding unit 23 is provided with a plurality of five lenses, namely, single-eye lenses 310, 311, 312, 313, and 314, which are arranged not to overlap in the optical axis direction of the optical axis (barrel optical axis) of the entire lens barrel 21 (when viewing). Figure 1 In the figure, five single-eye lenses 310 to 314 are arranged in the feeding unit 23 in the form of four single-eye lenses 311 to 314 forming the vertices of a square with the single-eye lens 310 as the center (as the center of gravity) on a two-dimensional plane orthogonal to the optical axis of the lens barrel (parallel to the light receiving surface (imaging surface) of the image sensor 51).

[0062] When the poly-interchangeable lens 20 is mounted on the camera body 10 , the single-eye lenses 310 to 314 condense light beams from a subject to the image sensor 51 of the camera body 10 .

[0063] Note that here, the camera body 10 is a so-called single-board camera including one image sensor 51, but as the camera body 10, a plurality of image sensors, that is, for example, a so-called three-board camera including three image sensors of red, green, and blue (RGB) may be adopted. In the three-board camera, light beams emitted from the monocular lenses 310 to 314 are respectively focused on the three image sensors using an optical system such as a prism. Note that, in addition to the three-board camera, the number of image sensors may be any number other than three, such as a two-board camera. Furthermore, the image sensor is not limited to an image sensor for RGB. All image sensors may be monochrome, or may include a color filter such as a Bayer array.

[0064] The feeding unit 23 is provided with two light sources 32L and 32R as a plurality of light sources in addition to the five single-eye lenses 310 to 314. The light sources 32L and 32R are provided at the right and left ends of the feeding unit 23, respectively, when the polycular interchangeable lens 20 is viewed from the front.

[0065] The light sources 32L and 32R are constituted by, for example, a light emitting diode (LED), a laser, or the like, and emit spot light from the front side (the side on which a light beam is incident) of the polycular interchangeable lens 20 to the rear side.

[0066] Therefore, with the poly-interchangeable lens 20 mounted on the camera body 10 , the point lights emitted by the light sources 32L and 32R are received by the image sensor 51 of the camera body 10 .

[0067] As described above, the feeding unit 23 is provided with the light sources 32L and 32R and the single-eye lenses 310 to 314 .

[0068] The feeding unit 23 is configured to be movable (slidable) in the optical axis direction of the barrel optical axis within the cylindrical barrel 21 so as to be fed out to the front side within the barrel 21 (retracted to the rear side).

[0069] Therefore, the polycular interchangeable lens 20 is configured so that the light sources 32L and 32R and the single-eye lenses 310 to 314 provided in the feeding unit 23 are integrally fed out.

[0070] As described above, since the single-eye lenses 310 to 314 and the light sources 32L and 32R are integrally sent out, the camera system 1 can perform appropriate processing.

[0071] That is, a point light image which is an image of point light emitted by the light sources 32L and 32R appears in a captured image captured by the image sensor 51 , and the attachment error of the polycular interchangeable lens 20 can be obtained using the point light image as described below.

[0072] In the poly-eye interchangeable lens 20 , the single-eye lenses 310 to 314 are provided in the feeding unit 23 , and thus the single-eye lenses 310 to 314 are fed out together with the feeding unit 23 , so that focus adjustment for telephoto shooting or macro shooting, for example, can be performed.

[0073] In this case, if the light sources 32L and 32R are provided in a portion other than the feeding unit 23 of the polycular interchangeable lens 20, the light sources 32L and 32R are not fed out even if the monocular lenses 310 to 314 are fed out. Then, the point light image using the point light emitted by such light sources 32L and 32R makes it difficult to accurately obtain the attachment error changed by the feeding out of the monocular lenses 310 to 314.

[0074] In contrast, in the case where the single-eye lenses 310 to 314 and the light sources 32L and 32R are integrally sent out, appropriate processing of accurately obtaining the attachment errors that change due to the sending out of the single-eye lenses 310 to 314 can be performed using point light images.

[0075] In addition, even if the areas of the monocular image corresponding to the images formed by the light beams respectively collected by the monocular lenses 310 to 314 on the captured image are changed due to the transmission of the monocular lenses 310 to 314, appropriate processing can be performed to accurately specify the areas of the monocular image.

[0076] Furthermore, calibration data for suppressing the influence of lens distortion due to changes in the feeding of the single-eye lenses 310 to 314 can be obtained, and further, appropriate processing of obtaining parallax information with suppressed influence of lens distortion using such calibration data can be performed.

[0077] Note that Figure 1In the embodiment, the polycular interchangeable lens 20 is provided with five single-eye lenses 310 to 314. However, the number of single-eye lenses provided in the polycular interchangeable lens 20 is not limited to five, and any number such as two, three, six or more may be adopted.

[0078] Furthermore, the plurality of single-eye lenses provided in the polycular interchangeable lens 20 may be arranged at any position on a two-dimensional plane in addition to being arranged at the center and vertex positions of a square.

[0079] Furthermore, a plurality of lenses having different focal lengths, f-numbers or other specifications may be employed as the plurality of single-eye lenses provided in the polycular interchangeable lens 20. Note that here, in order to simplify the description, a plurality of lenses having the same specification are employed.

[0080] In addition, Figure 1 In the embodiment, two light sources 32L and 32R are provided in the polyocular interchangeable lens 20. However, the number of light sources provided in the polyocular interchangeable lens 20 is not limited to two, and any number such as one or three or more may be adopted as required.

[0081] Furthermore, in the case where two light sources 32L and 32R are provided as a plurality of light sources in the polycular interchangeable lens 20, for example, the two light sources 32L and 32R may be arranged on a line connecting two farthest points on a plane where the five monocular lenses 310 to 314 are arranged, that is, on the polycular interchangeable lens 20 when the light sources are from 1 to 314. Figure 1 The light sources 32L and 32R are arranged on a circle when the substantially cylindrical feeding unit 23 is viewed from the front. In this case, when the feeding unit 23 is viewed from the front, the light sources 32L and 32R are arranged on a line passing through the center of the circle. As described below, it is desirable that the light sources 32L and 32R are arranged as far apart as possible. When the feeding unit 23 is viewed from the front, the light sources 32L and 32R can be arranged most apart by arranging the light sources 32L and 32R on a line passing through the center of the circle.

[0082] In the poly-interchangeable lens 20, five single-eye lenses 310 to 314 as a plurality of single-eye lenses are arranged so that when the poly-interchangeable lens 20 is mounted on the camera body 10, the single-eye lens 310 i The optical axis (single-eye optical axis) becomes orthogonal to the light receiving surface of the image sensor 51.

[0083] In the camera system 1 having the polycular interchangeable lens 20 mounted on the camera body 10 , the image sensor 51 images an image corresponding to an image formed on the light receiving surface of the image sensor 51 by light beams respectively condensed by the five single-eye lenses 310 to 314 .

[0084] Now, suppose that the i(where i=0, 1, 2, 3, 4) The image corresponding to the image formed by the focused light beams is called a monocular image, and the captured image captured by an image sensor 51 includes five monocular images corresponding to the five monocular lenses 310 to 314 respectively (images corresponding to the images formed by the light beams focused by the monocular lenses 310 to 314 respectively).

[0085] Single lens 31 i The monocular image is a monocular image having a monocular lens 31 i Therefore, the five monocular images of the respective monocular lenses 310 to 314 are images of different viewpoints.

[0086] Furthermore, the captured image includes a point light image (an image formed by point lights) which is an image corresponding to the point lights respectively emitted by the two light sources 32L and 32R.

[0087] here, Figure 1 The camera system 1 in the embodiment includes a camera body 10 and a multi-eye interchangeable lens 20 which is attachable to and detachable from the camera body 10, but the present technology can also be applied to a so-called lens-integrated camera system in which the multi-eye interchangeable lens 20 is fixed to the camera body 10. That is, the present technology can be applied to, for example, a lens-integrated camera.

[0088] Furthermore, one single-eye lens 31 may be configured by arranging a plurality of lenses in the optical axis direction of the barrel optical axis instead of by one lens. i .

[0089] In addition, for example, in addition to the camera body 10, some or all of the processing of the area designation unit 52, image processing unit 53, position calculation unit 57, point light image detection unit 62 and feed amount detection unit 64 of the camera body 10 to be described below can also be performed by a cloud server, a reproduction dedicated device, etc.

[0090] Furthermore, the zoom magnification can be adjusted in addition to adjusting the focus by feeding out the feeding unit 23 of the poly-interchangeable lens 20. Hereinafter, in order to simplify the description, it is assumed that the focus is adjusted by feeding out the feeding unit 23.

[0091] Note that, with respect to the camera body 10 , the surface on the side where the multi-eye interchangeable lens 20 is mounted, that is, the surface where the camera mount 11 is located is defined as the front surface.

[0092] <Electrical Configuration Example of Camera System 1>

[0093] Figure 2 It is shown Figure 1 1 is a block diagram of an example of an electrical configuration of the camera system 1.

[0094] In the camera system 1 , the poly-interchangeable lens 20 includes a storage unit 41 , a communication unit 42 , and a control unit 43 .

[0095] The storage unit 41 stores lens information which is information about the polycular interchangeable lens 20. The lens information includes individual difference reflecting position information (known reference position).

[0096] The individual difference reflecting position information is about the position of the single-lens lens 31 on a known captured image where a predetermined object located at a known distance appears. i The position information corresponding to the position of the predetermined light beam on the monocular image of the multi-eye interchangeable lens 20 is, for example, the known captured image captured by the (one) image sensor 51 when the multi-eye interchangeable lens 20 is mounted on the camera body 10. The individual difference reflecting position information can be said to be position information about the incident position of the predetermined light beam on the image sensor 51, which deviates by an amount (deviation from the designed position) different for each individual of the multi-eye interchangeable lens 20 due to manufacturing errors (manufacturing variations) during the manufacture of the multi-eye interchangeable lens 20, and is a deviation from the monocular lens 31 due to the different manufacturing errors of each individual during the manufacture of the multi-eye interchangeable lens 20 (the deviation from the designed position due to the different manufacturing errors). i As the individual difference reflecting position information, a single-lens lens 31 on a known captured image in which a predetermined object located at a known distance appears can be used. i The position itself corresponding to the predetermined light beam on the monocular image is, for example, the known captured image captured by the image sensor 51 when the polycular interchangeable lens 20 is mounted on the camera body 10.

[0097] Here, it is assumed that the single-eye lens 31 i The monocular image is formed through the monocular lens 31 i The position of the image of the light beam on the optical axis (monocular optical axis) is the optical axis center position. Note that the monocular optical axis is considered to be parallel to the optical axis of the entire lens barrel 21 (lens barrel optical axis), or is considered to be arranged at a fixed distance, but deviation occurs.

[0098] Now, suppose that, for example, i The light beam of the monocular optical axis is adopted as the monocular lens 31 i The predetermined light beam of the monocular image, the monocular lens 31 i The individual difference reflecting position information of the monocular image is the optical axis center position of the monocular image.

[0099] Note that the predetermined light beam is not limited to the light beam passing through the single eye lens 31. i That is, as the predetermined light beam, for example, a beam passing through the single-eye lens 31 can be used. iThe optical axis of the monocular is separated from the position of the predetermined distance and the light beam parallel to the optical axis of the monocular is disposed.

[0100] In addition to the single lens 31 on the known image capture i In addition to the individual difference reflecting position information of the monocular image, the lens information also includes individual difference point light position information (known light position) about the position of the point light image of the point light of the light sources 32L and 32R on the known captured image. As the individual difference point light position information, the position of the point light image of the point light of the light sources 32L and 32R on the known captured image itself can be used. Similar to the individual difference reflecting position information, the individual difference point light position information is position information including manufacturing errors that are different for each individual during the manufacture of the polycular interchangeable lens 20.

[0101] Here, a unique lens identification (ID) may be assigned to the polyocular interchangeable lens 20, and the lens ID of the polyocular interchangeable lens 20 may be adopted as the lens information to be stored in the storage unit 41. Further, in this case, a database may be prepared in which the lens ID as the lens information is associated with the individual difference reflection position information and the individual difference point light position information as the lens information other than the lens ID of the polyocular interchangeable lens 20 specified with the lens ID. In this case, the individual difference reflection position information, the individual difference point light position information, and the like of the polyocular interchangeable lens 20 associated with the lens ID may be acquired by searching the database using the lens ID as a keyword.

[0102] The communication unit 42 performs wired or wireless communication with a communication unit 56 of the camera body 10, which will be described below. Note that the communication unit 42 can also communicate with a server on the Internet, a personal computer (PC) on a wired or wireless local area network (LAN), or other external devices by any communication method as needed.

[0103] The communication unit 42 communicates with the communication unit 56 of the camera body 10 so that the lens information stored in the storage unit 41 is sent to the communication unit 56 in the following cases, for example: the multi-eye interchangeable lens 20 is mounted on the camera body 10, or the power is turned on in a state where the multi-eye interchangeable lens 20 is mounted on the camera body 10.

[0104] Furthermore, the communication unit 42 receives commands and other information sent from the communication unit 56 and provides the commands and other information to the control unit 43. The control unit 43 controls the polycular interchangeable lens 20 by feeding (moving) the feeding unit 23 according to the information from the communication unit 42, such as focus adjustment.

[0105] The camera body 10 includes an image sensor 51, an area designation unit 52, an image processing unit 53, a display unit 54, a storage unit 55, a communication unit 56, a position calculation unit 57, a control unit 61, a point light image detection unit 62, a feed amount information storage unit 63 and a feed amount detection unit 64.

[0106] The image sensor 51 is, for example, Figure 1 The CMOS image sensor described above, and the light receiving surface of the image sensor 51 is illuminated by light beams condensed by the single-eye lenses 310 to 314 of the poly-interchangeable lens 20 mounted on the camera body 10 and light beams as point lights emitted by the light sources 32L and 32R.

[0107] The image sensor 51 forms and outputs a captured image by receiving the light beams collected by the monocular lenses 310 to 314 and the light beams as the point lights emitted by the light sources 32L and 32R and performing photoelectric conversion, wherein the captured image includes the monocular images of the monocular lenses 310 to 314 (monocular images corresponding to the images formed by the light beams collected by the monocular lenses 310 to 314) and the point light images of the point lights of the light sources 32L and 32R. The captured image (another captured image) output by the image sensor 51 is provided to the area designation unit 52, the position calculation unit 57, and the point light image detection unit 62.

[0108] The captured image output by the image sensor 51 is supplied from the position calculation unit 57 to the area specification unit 52 , and installation error reflecting position information (unknown reference position) is also supplied as position information about the monocular image included in the captured image output by the image sensor 51 .

[0109] The mounting error reflecting position information is about, for example, the position of the single-eye lens 31 on a captured image (another captured image) obtained by imaging an arbitrary object by the (one) image sensor 51 in a state where the poly-interchangeable lens 20 is mounted on the camera body 10. i The installation error reflecting position information can be said to be position information about the incident position of the predetermined light beam on the image sensor 51, which is deviated due to the installation error of the polyocular interchangeable lens 20 when the polyocular interchangeable lens 20 is installed, and the installation error reflecting position information is the position information including the installation error when using the polyocular interchangeable lens 20 (the position from the monocular lens 31 caused by the installation error). i The position information of the emitted imaging light (the deviation of the emission position).

[0110] The mounting error indicates a deviation in the attachment position (mounting position) of the poly-eye interchangeable lens 20 due to the poly-eye interchangeable lens 20 being attachable to and detachable from the camera body 10. For example, the mounting error may change each time the poly-eye interchangeable lens 20 is mounted. In addition, for example, when an impact is applied to the camera system 1 having the poly-eye interchangeable lens 20 mounted on the camera body 10, the mounting error may change.

[0111] The installation error reflecting position information is position information including manufacturing error in addition to the installation error (including the position information from the single-lens lens 31 due to the manufacturing error and the installation error). i position information of the deviation of the emission position of the emitted imaging light).

[0112] Here, in the case where the captured image is a known captured image obtained by imaging a certain object located at a known distance, in the case where the center position of the optical axis on the monocular image included in the known captured image is used as the individual difference reflecting position information, for example, the center position of the optical axis on the monocular image included in a captured image (another captured image) obtained by imaging an arbitrary object (regardless of whether the distance to the object is known) can be used as the installation error reflecting position information.

[0113] The area specifying unit 52 specifies areas of the monocular images of the monocular lenses 310 to 314 on the captured image from the image sensor 51 based on the installation error reflecting position information from the position calculating unit 57, and outputs area specifying result information indicating the specifying result of each area.

[0114] That is, the area specifying unit 52 specifies, for example, a rectangular area of ​​a captured image from the image sensor 51 having a predetermined size centered on (as the center of gravity) the mounting error reflecting position information of the captured image as the area of ​​the monocular image.

[0115] Here, the area specifying unit 52 may output, for example, a set of area information indicating each area of ​​the monocular image on the entire captured image and the monocular image on the entire captured image as the area specifying result information. In addition, the area specifying unit 52 may extract (cut out) the monocular image from the captured image and output the monocular image as the area specifying result information. Note that the monocular image may be output in a set with the area information.

[0116] Hereinafter, in order to simplify the description, for example, it is assumed that the area specifying unit 52 outputs a monocular image (monocular image of the monocular lenses 310 to 314 ) extracted from the captured image as the area specifying result information.

[0117] The monocular images of the monocular lenses 310 to 314 output by the area specifying unit 52 are supplied to the image processing unit 53 .

[0118] The image processing unit 53 is a part of a processing unit that performs processing according to the detection result of the point light image detection unit 62 to be described below. The image processing unit 53 uses the monocular images of the monocular lenses 310 to 314 from the area designation unit 52 (that is, the monocular images having different viewpoints at the respective positions of the monocular lenses 310 to 314) and the feed amount of the feed unit 23 supplied from the feed amount detection unit 64, and performs image processing such as generating parallax information, and refocusing generating (reconfiguring) an image in which an arbitrary object is focused, and supplies the processing result image obtained as a result of the image processing to the display unit 54 and the storage unit 55.

[0119] Note that the image processing unit 53 can also perform general image processing such as defect correction and noise reduction. In addition, the image processing unit 53 can perform image processing on both the image to be saved (stored) in the storage unit 55 and the image to be displayed on the display unit 54 as a so-called through image.

[0120] The display unit 54 includes, for example, a liquid crystal panel, an organic electroluminescent (EL) panel, etc., and is provided on the back surface of the camera body 10. The display unit 54 displays, for example, a processing result image supplied from the image processing unit 53 as a through image. As the through image, in addition to the processing result image, a part of the entire captured image captured by the image sensor 51 or a monocular image extracted from the captured image can be displayed. In addition, the display unit 54 can display information such as menus and settings of the camera body 10, for example.

[0121] The storage unit 55 includes a memory card (not shown) or the like, and stores, for example, a processing result image supplied from the image processing unit 53 according to a user operation or the like.

[0122] The communication unit 56 performs wired or wireless communication with the communication unit 42 of the polycular interchangeable lens 20, etc. Note that the communication unit 56 can also communicate with a server on the Internet, a PC on a wired or wireless LAN, or other external devices by any communication method as needed.

[0123] For example, when the polycular interchangeable lens 20 is mounted on the camera body 10, the communication unit 56 communicates with the communication unit 42 of the polycular interchangeable lens 20 to receive lens information of the polycular interchangeable lens 20 sent from the communication unit 42, and provides the lens information to the position calculation unit 57 and the point light image detection unit 62.

[0124] Furthermore, the communication unit 56 transmits information specifying a focus (position) from the control unit 61 to the communication unit 42 , for example.

[0125] The position calculation unit 57 obtains the position of the single-eye lens 31 included in the captured image supplied from the image sensor 51 based on the individual difference reflection position information included in the lens information from the communication unit 56. i The installation error reflecting position information of the optical axis center position on the monocular image is obtained, and the installation error reflecting position information is provided to the area specifying unit 52.

[0126] Note that Figure 2 In obtaining the installation error reflecting position information of the optical axis center position on the monocular image included in the captured image provided by the image sensor 51, the position calculation unit 57 uses the individual difference point light position information in addition to the individual difference reflecting position information included in the lens information.

[0127] The control unit 61 controls the focus, etc. according to the user operation for adjusting the focus, etc. For example, the control unit 61 provides information for specifying the focus to the communication unit 56 according to the user's operation.

[0128] The point light image detection unit 62 detects the incident range of the point light emitted from the light sources 32L and 32R on the image sensor 51 , and supplies the detection result to the feed amount detection unit 64 .

[0129] That is, the point light image detection unit 62 detects the point light image on the captured image from the image sensor 51 based on the individual difference point light position information included in the lens information from the communication unit 56. In addition, the point light image detection unit 62 detects (generates) point light image information about the point light image, such as the (spot) size and position (detection light image position) of the point light image, and outputs the point light image information as the detection result of the point light image. The point light image information output by the point light image detection unit 62 is provided to the feed amount detection unit 64, which is another part of the processing unit that performs processing based on the detection result of the point light image detection unit 62. As the point light image information, in addition to information that directly indicates the size and position of the point light image on the captured image (for example, the size and position itself), information that indirectly indicates the size and position of the point light image on the captured image (for example, an image in which the point light image appears while maintaining the size and position of the point light image on the captured image) can also be used.

[0130] The feed amount information storage unit 63 stores feed amount information. The feed amount information is information in which the feed amount of the feed unit 23 is associated with the point light image information about the point light image when the feed unit 23 is sent out with the feed amount. The feed amount information may be generated in advance and stored in the feed amount information storage unit 63. In addition, for example, before the polycular interchangeable lens 20 is shipped from the factory, the feed amount information may be generated in advance and stored in the storage unit 41 as a part of the lens information. In the case where the feed amount information is stored in the storage unit 41 as a part of the lens information, the communication unit 56 communicates with the communication unit 42 to acquire the lens information stored in the storage unit 41, and provides the feed amount information included in the lens information to the feed amount information storage unit 63 and stores the information therein.

[0131] The feed amount detection unit 64 detects the feed amount of the feeding unit 23 associated with the point light image information from the point light image detection unit 62 among the feed amount information stored in the feed amount information storage unit 63 , and supplies the feed amount to the image processing unit 53 .

[0132] <Overview of Imaging Performed Using the Multi-eye Interchangeable Lens 20>

[0133] Figure 3 2 is a diagram for describing an outline of imaging of a captured image performed using the poly-eye interchangeable lens 20 .

[0134] The image sensor 51 of the camera body 10 in which the poly-interchangeable lens 20 is mounted forms a captured image, the captured image including the image corresponding to the image captured by the single-eye lens 31. i A single-eye image of an image formed when light beams are concentrated, and a point-light image of the point lights emitted by the light sources 32L and 32R.

[0135] Here, in this specification, the single-eye lens 31 i In the optical axis direction of the monocular optical axis, the direction from the rear surface side to the front surface side of the camera body 10 is defined as the z direction (axis), the direction from left to right when facing the z direction is defined as the x direction, and the direction from bottom to top is defined as the y direction.

[0136] Furthermore, in order to make the left and right sides of the object appearing in the image match the left and right sides of the object in the real space, and to make the single-eye lens 31 i The left and right sides of the position are related to the single-eye lens 31 i The position on the captured image, the position of the single-eye lens 31 and the position of the single-eye lens 31 are described below with reference to the z direction, that is, the imaging direction of the object to be captured from the rear surface side of the camera body 10. i 's position, and the left and right of an object, etc., unless otherwise specified.

[0137] Note that connecting a single lens 31 i and another monocular lens 31 j The straight line or line segment of the monocular optical axis of (i≠j) is also called a baseline, and the distance between the monocular optical axes is also called a baseline length. In addition, the angle indicating the direction of the baseline is also called a baseline angle. Here, as the baseline angle, for example, the angle formed by the x-axis and the baseline (the angle of the epipolar line (epipolar line)) is adopted.

[0138] In addition, in this specification (and claims), the feeding unit 23 is sent out broadly means that the feeding unit 23 moves in the optical axis direction of the barrel optical axis. Therefore, the feeding unit 23 includes both the movement of the feeding unit toward the front side and the movement of the feeding unit toward the depth side.

[0139] Figure 4 3 is a diagram showing an example of arrangement of the single-eye lenses 310 to 314 and the light sources 32L and 32R in the polycular interchangeable lens 20 , and a captured image captured using the polycular interchangeable lens 20 .

[0140] Figure 4 1A is a rear view showing an arrangement example of the single-eye lenses 310 to 314 and the light sources 32L and 32R in the polycular interchangeable lens 20 .

[0141] exist Figure 4 In A, Figure 1 As described above, the single eye lenses 310 to 314 are arranged so that the other four single eye lenses 311 to 314 configure the vertices of a square centered on the single eye lens 310 on a two-dimensional plane parallel to the light receiving surface of the image sensor 51 .

[0142] That is to say, Figure 4 For example, with the single eye lens 310 of the single eye lenses 310 to 314 as a reference, the single eye lens 311 is arranged at the upper right of the single eye lens 310, and the single eye lens 312 is arranged at the upper left of the single eye lens 310. In addition, the single eye lens 313 is arranged at the lower left of the single eye lens 310, and the single eye lens 314 is arranged at the lower right of the single eye lens 310.

[0143] In addition, Figure 4 In A, the light source 32L is arranged at the left end position of the multi-eye interchangeable lens 20 having a basically circular plane, and the light source 32R is arranged at the right end position opposite to the light source 32L with respect to the center (center) of the multi-eye interchangeable lens 20 having a basically circular plane.

[0144] Note that the light sources 32L and 32R may be arranged at any different positions of (the feeding unit 23 of) the polycular interchangeable lens 20 .

[0145] Note that the light sources 32L and 32R may be arranged so that the point light images PL and PR of the point lights emitted by the respective light sources 32L and 32R are located outside the region of the monocular image included in the captured image (before being captured by the monocular lens 311). i In this case, it is possible to suppress the point light images PL and PR from overlapping with the single-eye image to deteriorate the image quality of the single-eye image.

[0146] Figure 4 B is a view showing an example of a captured image captured by the image sensor 51 of the camera body 10 mounted with the poly-eye interchangeable lens 20 in which the single-eye lenses 310 to 314 and the light sources 32L and 32R are arranged, as shown in FIG. Figure 4 As shown in A.

[0147] The captured image captured by the image sensor 51 of the camera body 10 equipped with the multi-eye interchangeable lens 20 having single-eye lenses 310 to 314 and light sources 32L and 32R includes single-eye images E0, E1, E2, E3 and E4 corresponding to images formed by light beams respectively concentrated by the single-eye lenses 310 to 314, and point light images PL and PR of the point lights of the respective light sources 32L and 32R.

[0148] Area specifying unit 52 ( Figure 2 ) Based on the optical axis center position as the mounting error reflecting position information of each single-eye image E#i obtained in the position calculation unit 57, the image having been passed through the single-eye lens 31 is displayed on the captured image. i A rectangular area having a predetermined size centered at the center position of the optical axis is designated as an area irradiated by the light beam for each single eye lens 31. i The area of ​​the monocular image E#i, the center position of the optical axis is the installation error reflecting position information of the monocular image E#i.

[0149] Therefore, for the single-eye lens 31 i The single-eye image E#i becomes the same as the single-eye image E#i from the single-eye lens 31 i The position of the image is similar to the captured image obtained by performing imaging using an independent camera or an independent image sensor, that is, by using a single-lens lens 31 i The image is obtained by imaging the viewpoint at the position.

[0150] Therefore, parallax occurs between any two monocular images E#i and E#j among the monocular images E0 to E4 of the respective monocular lenses 310 to 314. That is, the same subject captured in the monocular images E#i and E#j appears at a position shifted by the parallax.

[0151] <Attachment Error of Multi-eye Interchangeable Lens 20>

[0152] Figure 5 is a diagram for describing an attachment error when the poly-interchangeable lens 20 is attached (mounted) to the camera body 10 .

[0153] That is to say, Figure 5 An example of a captured image captured by the camera system 1 in which the poly-interchangeable lens 20 is attached to the camera body 10 is shown.

[0154] In the case where the poly-eye interchangeable lens 20 is attached to the camera body 10, the attachment position of the poly-eye interchangeable lens 20 relative to the light receiving surface of the image sensor 51 of the camera body 10 may be shifted mainly in the horizontal direction (x direction), the vertical direction (y direction), and the rotational direction, particularly in the rotational direction. First, due to manufacturing errors, the attachment position of the poly-eye interchangeable lens 20 shifts by an amount that differs for each individual. In addition, when the poly-eye interchangeable lens 20 is attached to the camera body 10, or when an impact is applied to the camera system 1 having the poly-eye interchangeable lens 20 attached to the camera body 10 when the poly-eye interchangeable lens 20 is used, the attachment position of the poly-eye interchangeable lens 20 changes.

[0155] Now, for example, an error in an actual attachment position relative to a predetermined position such as a designed attachment position of the poly-eye interchangeable lens 20 is referred to as an attachment error. The attachment error with respect to the designed attachment position is caused by manufacturing variations and the like, and changes when the poly-eye interchangeable lens 20 is attached to the camera body 10 or when an impact is applied to the camera system 1 having the poly-eye interchangeable lens 20 attached to the camera body 10.

[0156] The attachment error is an error in the actual attachment position of the polyocular interchangeable lens 20, and appropriately includes a manufacturing error and an installation error. For example, in the case of using the designed attachment position of the polyocular interchangeable lens 20 as a reference, the attachment error includes both the manufacturing error and the installation error. In addition, for example, in the case of using a position offset by the manufacturing error from the designed attachment position of the polyocular interchangeable lens 20 as a reference, the attachment error does not include the manufacturing error, and includes the installation error.

[0157] As reference Figure 4 The single-eye image E#i is similar to the single-eye image E#i through the single-eye lens 31. i The positions of the images are images obtained by imaging as viewpoints, and therefore, the monocular images E0 to E4 are images with different viewpoints.

[0158] In the case of obtaining disparity information using, for example, monocular images E0 to E4 which are images having different viewpoints, it is necessary for the monocular lenses 310 to 314 to Figure 3 The baseline length and baseline angle described in .

[0159] Since the single-eye lenses 310 to 314 are fixed to the polyocular interchangeable lens 20 , the base line length does not change due to mounting errors and is a fixed value that can be measured in advance after the polyocular interchangeable lens 20 is manufactured.

[0160] Meanwhile, the base angle varies due to an attachment error (mounting error) in the rotational direction of the polycular interchangeable lens 20. Therefore, in order to obtain accurate parallax information using the monocular images E0 to E4, the attachment error in the rotational direction must be processed.

[0161] Here, in the case of a single-lens lens 31 i In the case where the image distortion caused by the lens aberration is small and can be ignored in some cases, there is no problem with the attachment errors in the horizontal and vertical directions. However, in the case where the image distortion caused by the lens aberration is large and distortion correction of the image distortion is required, it is necessary to accurately grasp the optical axis center position of the monocular image E#i in order to perform appropriate distortion correction. In order to accurately grasp the optical axis center position of the monocular image E#i, it is necessary to grasp the attachment errors (installation errors) in the horizontal and vertical directions.

[0162] Now, if Figure 5 As shown, in a specific xy coordinate system (two-dimensional coordinate system), the optical axis center positions of the monocular images E0 to E4 are expressed as (x0, y0), (x1, y1), (x2, y2), (x3, y3) and (x4, y4).

[0163] In the single-eye lenses 310 to 314 , the single-eye image E0 of the single-eye lens 310 located at the center is referred to as a central image E0 , and the single-eye images E1 to E4 of the single-eye lenses 311 to 314 located at the periphery are referred to as peripheral images E1 to E4 .

[0164] The relative optical axis center positions (hereinafter also referred to as relative optical axis center positions) (dx1, dy1), (dx2, dy2), (dx3, dy3) and (dx4, dy4) of the peripheral images E1 to E4 of the monocular images E0 to E4 with respect to a monocular image (i.e., for example, the central image E0) can be obtained by formula (1).

[0165] [Mathematical formula 1]

[0166] (dx1, dy1) = (x1-x0, y1-y0)

[0167] (dx2, dy2) = (x2 - x0, y2 - y0)

[0168] (dX3, dy3) = (x3 - x0, y3 - y0)

[0169] (dx4,dy4)=(x4-x0,y4-y0)...(1)

[0170] When the optical axis center position (x0, y0) of the central image E0 is set as the origin of the xy coordinate system, the relative optical axis center positions (dx1, dy1), (dx2, dy2), (dx3, dy3) and (dx4, dy4) are equal to the optical axis center positions (x1, y1), (x2, y2), (x3, y3) and (x4, y4) of the peripheral images E1 to E4.

[0171] The relative optical axis center position (dx#i, dy#i) (where i = 1, 2, 3, 4) can be regarded as a vector in the direction of the baseline connecting the optical axis center position (x0, y0) of the central image E0 and the optical axis center position (x#i, y#i) of the peripheral image E#i. According to the relative optical axis center position (dx#i, dy#i), the baseline angle (tan- 1 ((y#i-y0) / (x#i-x0))=tan- 1 (dy#i / dx#i).

[0172] Therefore, if the relative optical axis center position (dx#i, dy#i) can be obtained, the baseline angle representing the direction of the baseline L0#i at that time can be obtained, and the baseline angle can be used to obtain accurate parallax information that is not affected by attachment errors in the rotation direction.

[0173] In the present technology, the optical axis center positions (x0, y0) to (x4, y4) of each monocular image E0 to E4 on a known captured image of a predetermined object located at a known distance, which is captured by an image sensor 51, are obtained, that is, with the optical axis center position (x0, y0) of the center image E0 being set as the origin, the relative optical axis center positions (dx1, dy1) to (dx4, dy4) of the each monocular image E1 to E4 are obtained as individual differences reflecting position information. In addition, in the present technology, individual difference reflection position information ((x0, y0) to (x4, y4) or (dx1, dy1) to (dx4, dy4) and a captured image are used to obtain the optical axis center positions (x0', y0') to (x4', y4') of each monocular image E0 to E4 on the captured image when the captured image is imaged, that is, when the optical axis center position (x0', y0') of the center image E0 is set as the origin, the relative optical axis center positions (dx1', dy1') to (dx4', dy4') of each monocular image E1 to E4 are obtained as the installation error reflection position information.

[0174] If the relative optical axis center positions (dx1', dy1') to (dx4', dy4') of each monocular image E1 to E4 on the captured image are obtained as position information reflecting the installation error, the baseline angle when imaging the captured image can be obtained, and accurate parallax information that is not affected by the attachment error in the rotation direction can be obtained using this baseline angle.

[0175] Figure 2 The position calculation unit 57 uses the relative optical axis center positions (dx1, dy1) to (dx4, dy4) as individual difference reflection position information, and obtains the relative optical axis center positions (dx1', dy1') to (dx4', dy4') of each monocular image E1 to E4 on the captured image as installation error reflection position information.

[0176] <Calculation method for obtaining the relative optical axis center position (dx#i', dy#i') of the single-eye image E#i on the captured image as the installation error reflecting position information>

[0177] Figure 6 It is a diagram for describing a calculation method of obtaining relative optical axis center positions (dx1′, dy1′) to (dx4′, dy4′) as mounting error reflecting position information.

[0178] In the following, for the sake of simplicity, an xy coordinate system is used with the optical axis center position (x0, y0) of the center image E0 as the origin. In this case, as described above, the relative optical axis center positions (dx1, dy1), (dx2, dy2), (dx3, dy3), and (dx4, dy4) are equal to the optical axis center positions (x1, y1), (x2, y2), (x3, y3), and (x4, y4).

[0179] Figure 6 : A shows an example of a known captured image obtained by imaging a predetermined object in the camera system 1 in which the poly-interchangeable lens 20 is attached to the camera body 10 .

[0180] The object appearing in the known captured image is, for example, a chart image in which a predetermined chart is drawn (for example, a circle divided into four equal parts by a line segment passing through the center of the circle). The known captured image is captured by arranging the chart image at a position of a known distance on the monocular optical axis of the monocular lens 310 so that the center of the circle of the chart as the chart image appears, for example, at a predetermined point on the center image E0 (that is, for example, the optical axis center position (x0, y0) = (0, 0) of the center image E0). Therefore, the known captured image is an image obtained by imaging the chart image in which the predetermined chart is drawn at a known distance.

[0181] Since the known shot image is shot as described above, a chart image whose center is located at the optical axis center position (x0, y0) = (0, 0) as a circle of the chart appears in the center image E0 on the known shot image. In addition, similar to the center image E0, the chart image appears in the peripheral image E#i. Note that in the peripheral image E#i, the position of the circle as the chart is offset from the position of the circle as the chart appearing in the center image E0 according to the parallax with the center image E0.

[0182] Therefore, in the central image E0 on the known captured image, the center of the circle serving as the chart is located at the optical axis center position (x0, y0) = (0, 0), but in the peripheral image E#i, the center of the circle serving as the chart is offset from the optical axis center position (x#i, y#i) according to the parallax with the central image E0.

[0183] Since the chart image is placed at a known distance, when a known photographic image is taken, the known distance and the single-lens lens 31 can be viewed. i The parallax between the peripheral image E#i and the central image E0 is obtained by the base line length and base line angle between the monocular lens 310 .

[0184] Here, imaging of a known captured image may be performed, for example, before the multi-eye interchangeable lens 20 is shipped from the factory. Therefore, the baseline angle when imaging the known captured image may be measured when imaging the known captured image. Alternatively, the single-eye lens 31 may be adjusted when imaging the known captured image. i Attachment (fixing) to the lens barrel 21 is such that the base line angle becomes a predetermined value such as a design value.

[0185] Since the optical axis center position (x#i, y#i) of the peripheral image E#i is a position moved from the center of a circle of a chart appearing in the peripheral image E#i according to the parallax with the central image E0, the optical axis center position can be obtained from the center position of the circle of a chart appearing in the peripheral image E#i and the parallax with the central image E0.

[0186] In addition, since the center of the circle of the chart serving as the chart image appears at the optical axis center position (x0, y0) (= (0, 0)) of the central image E0 on the known captured image, the optical axis center position (x0, y0) of the central image E0 can be obtained by detecting the position of the center of the circle serving as the chart from the central image E0.

[0187] As described above, the optical axis center positions (x1, y1) to (x4, y4) of the peripheral images E1 to E4 and the optical axis center position (x0, y0) of the central image E0 on the known captured image can be obtained from the known captured image.

[0188] Based on the individual difference reflecting position information of the optical axis center position (x0, y0) of the central image E0 on the known captured image and the individual difference reflecting position information of the optical axis center position (x#i, y#i) of the peripheral image E#i, the relative optical axis center position (dx#i, dy#i) as the individual difference reflecting relative position information can be obtained, and the individual difference reflecting relative position information is the relative individual difference reflecting position information of the peripheral image E#i with respect to the individual difference reflecting position information (x0, y0) of the central image E0, and the relative optical axis center position (dx#i, dy#i) as the individual difference reflecting relative position information is stored as lens information. Figure 2 The storage unit 41 is stored in the storage unit 41. Here, the individual difference reflecting position information (x#i, y#i) is also referred to as individual difference reflecting absolute position information, in contrast to the individual difference reflecting relative position information (dx#i, dy#i).

[0189] Note that, as lens information, in addition to using individual differences to reflect relative position information (relative optical axis center position) (dx#i, dy#i) (i=1, 2, 3, 4), individual differences to reflect absolute position information (optical axis center position) (x#i, y#i) (i=0, 1, 2, 3, 4) can also be used. This is because individual differences to reflect relative position information (dx#i, dy#i) can be obtained from individual differences to reflect absolute position information (x#i, y#i) according to equation (1), and individual differences to reflect relative position information (dx#i, dy#i) is (substantially) equivalent information to individual differences to reflect absolute position information (x#i, y#i).

[0190] When the relative optical axis center positions (dx1′, dy1′) to (dx4′, dy4′) are obtained as the installation error reflecting position information, in addition to the individual difference reflecting relative position information (dx#i, dy#i) (or the individual difference reflecting absolute position information (x#i, y#i)), the individual difference point light position information (x#i, y#i) of the positions of the point light images PL and PR of the point lights of the respective light sources 32L and 32R on the known captured image is obtained. L , Y L ) and (X R , Y R ) is obtained in advance.

[0191] For example, the centroid position of the point light image PL on the known captured image may be used as the individual difference point light position information (X L , Y L Similarly, the centroid position of the point light image PR on the known captured image can be used as the individual difference point light position information (X R , Y R ).

[0192] For individual difference light position information (X L , Y L ) and (X R , Y R ), and obtain individual difference light position information (X L , Y L ) and (X R , Y R ) is the midpoint (X C , Y C ), and individual difference light position information (X L , Y L ) and (X R , Y R ) and the midpoint (X C , Y C ) as lens information stored in Figure 2 In the storage unit 41.

[0193] Note that individual difference light position information (X L , Y L ) and (X R , Y R ) between the midpoint (X C , Y C This is because the individual difference light position information (X L , Y L ) and (X R , Y R ) to obtain individual difference light position information (X L , Y L ) and (X R , Y R ) is the midpoint (X C , Y C ).

[0194] The position calculation unit 57 calculates the relative optical axis center position (hereinafter also referred to as the optical axis center position) (dx#i, dy#i) and the individual difference point light position information (x#i) as the individual difference reflecting the relative position information. L , Y L ) and (X R , Y R ) (hereinafter also referred to as individual difference reflecting position information) obtains the (relative) optical axis center positions (dx1′, dy1′) to (dx4′, dy4′) as the installation error reflecting position information on the unknown captured image).

[0195] Figure 6 B shows an example of an unknown captured image captured in the camera system 1 in which the poly-interchangeable lens 20 is attached to the camera body 10 .

[0196] The unknown captured image is an image captured in the camera system 1 in which the poly-interchangeable lens 20 is attached to the camera body 10 without restrictions (such as restrictions on the distance of a known subject) when a known captured image is captured.

[0197] When an unknown captured image is imaged, an attachment error (mounting error) in a rotational direction different from that when a known captured image is imaged may occur.

[0198] In the xy coordinate system with the optical axis center position (x0', y0') of the central image E0 on the unknown captured image as the origin (0, 0), the optical axis center position (x#i', y#i') (i=1, 2, 3, 4) of the peripheral image E#i on the unknown captured image is equal to the relative optical axis center position (dx#i', dy#i') of the peripheral image E#i with respect to the optical axis center position (x0', y0') of the central image E0 = (x#i', y#i') - (x0', y0').

[0199] Here, the position information about the positions of the point light images PL and PR of the point lights of the respective light sources 32L and 32R on the unknown captured image is also referred to as installation error point light position information (unknown light position). Similar to the installation error reflecting position information, the installation error point light position information is position information including the manufacturing error and the installation error of the poly-eye interchangeable lens 20. As the installation error point light position information, the positions of the point light images PL and PR of the point lights of the respective light sources 32L and 32R on the unknown captured image can be used. Note that the positions of the point light images PL and PR of the point lights of the respective light sources 32L and 32R on the unknown captured image are respectively expressed as (X L ', Y L ') and (X R ', Y R ').

[0200] Installation error point light position information (X L ', Y L ') and (X R ', Y R ') can be obtained from the point light images PL and PR on the unknown captured image, similar to the individual difference point light position information (X L , Y L ) and (X R , Y R ).

[0201] In addition, the installation error point light position information (X L ', Y L ') and (X R ', Y R The midpoint between C ', Y C ').

[0202] Now, assume that the relative rotation error is expressed as θ Error The relative rotation error is based on the attachment error in the rotation direction when imaging the known captured image. The relative rotation error θ Error The individual difference light position information (X) included in the lens information can be used according to equation (2)L , Y L ) and (X R , Y R ) and the installation error point light position information (X L ', Y L ') and (X R ', Y R ') to obtain.

[0203] [Mathematical formula 2]

[0204]

[0205] (Note X R >X L ,X R '>X L ')...(2)

[0206] According to equation (2), the relative rotation error θ Error It is used to represent the position information of the light difference between the connected individuals (X L , Y L ) and (X R , Y R ) is used as the reference for the angle of the straight line direction to connect the installation error point light position information (X L ', Y L ') and (X R ', YR') and the individual difference light position information (X L , Y L ) and (X R , Y R ) the larger the interval (installation error point light position information (X L ', Y L ') and (X R ', Y R Therefore, by arranging the light sources 32L and 32R as far apart as possible, the relative rotation error θ can be accurately obtained. Error .

[0207] Note that in the case where three or more light sources are provided in the polycular interchangeable lens 20, the rotation error θ of each pair of two light sources obtained from the three or more light sources is obtained according to equation (2): Error , and the rotation error θ obtained for each pair can be used Error The average value of the final rotation error θ Error .

[0208] Relative rotation error θ Error is the installation error point light position information (X L ', YL ')(or(X R ', Y R ')) and individual difference light position information (X L , Y L )(or(X R , Y R )). According to formula (3), the relative rotation error θ Error The relative position information of the optical axis center position (dx#i, dy#i) can be obtained by rotating the optical axis center position (dx#i, dy#i) as an individual difference, and the relative rotation error θ Error The relative optical axis center position (dx#i', dy#i') on the unknown captured image as the mounting error reflecting position information.

[0209] [Mathematical formula 3]

[0210]

[0211] In the case where the optical axis center positions (dx1′, dy1′) to (dx4′, dy4′) of the respective monocular images E1 to E4 on the unknown captured image as the mounting error reflecting position information are obtained according to equations (2) and (3), the attachment errors in the horizontal direction and the vertical direction can be obtained by obtaining the mounting error point light position information (X1′, dy1′) of the point light images PL and PR of the respective light sources 32L and 32R on the unknown captured image. L ', Y L ') and (X R ', Y R ') and the individual difference point light position information (X) of the point light images PL and PR of the respective light sources 32L and 32R on the known captured image L , Y L ) and (X R , Y R ) to obtain the translation between them.

[0212] That is, for example, the attachment error X in the horizontal direction can be obtained according to equation (4): Error and the attachment error Y in the vertical direction Error .

[0213] [Formula 4]

[0214]

[0215] Note that in equation (4), the installation error point light position information (X L ', Y L ') and (X R ', Y R ') of the midpoint (XC ', Y C ') and the individual difference point light position information (X) of the point light images PL and PR of the respective light sources 32L and 32R on the known captured image L , Y L ) and (X R , Y R ) is the midpoint (X C , Y C ), as the horizontal attachment error X Error and the attachment error Y in the vertical direction Error Or, for example, as the attachment error X in the horizontal direction Error and the attachment error Y in the vertical direction Error , the installation error point light position information (X L ′,Y L ′) and individual difference light position information (X L , Y L ), or the installation error point light position information (X R ', Y R ') and individual difference light position information (X R , Y R ) between .

[0216] In the camera system 1, first, when the optical axis center positions (dx1′, dy1′) to (dx4′, dy4′) of the respective monocular images E1 to E4 on the unknown captured image are obtained as the installation error reflecting position information, the processing of obtaining the optical axis center position (dx#i, dy#i) and the like as the individual difference reflecting relative position information required in the case of obtaining the optical axis center position (dx#i', dy#i') is performed.

[0217] The process of acquiring individual difference reflecting position information and the like may be executed by the camera body 10 , a computer described below, etc. For convenience, a device that executes the process of acquiring individual difference reflecting position information and the like is referred to as an acquisition processing device.

[0218] The acquisition processing device acquires a known captured image obtained by imaging a chart image of a predetermined object placed at a position of a known distance on the single-eye optical axis of the single-eye lens 310 by the camera system 1 in which the polycular interchangeable lens 20 is attached to the camera body 10 .

[0219] The acquisition processing device acquires a predetermined point of the chart image as a predetermined object appearing in each monocular image E#i included in the known captured image, for example, as the position of the center of a circle of the chart.

[0220] For each of the monocular images (peripheral images) E1 to E4, using the distance to the chart image as the object, and the baseline length and baseline angle of the polycular interchangeable lens 20, the processing device obtains the parallax between the center of a predetermined point of the chart image as the object appearing in the monocular image E#i and the center of a predetermined point of the chart image as the object appearing in the monocular image (central image) E0.

[0221] In addition, for each of the monocular images E1 to E4, the acquisition processing device obtains the optical axis center position (position on the known captured image) (x#i, y#i) of the monocular image E#i located at a position moved from the center position of the circle based on the parallax of the center of a predetermined point of the chart image as the object appearing in the monocular image E#i as the individual difference reflection absolute position information (x#i, y#i) of the monocular image E#i. In addition, the acquisition processing device obtains the optical axis center position (x0, y0) as the position of the center of the chart image as the object appearing in the monocular image E0 as the individual difference reflection absolute position information (x0, y0) of the monocular image E0.

[0222] Then, the processing device obtains the absolute position information (x#i, y#i) reflected by the individual difference for each of the monocular images E1 to E4, and obtains the relative position information (dx#i, dy#i) reflected by the individual difference of the monocular image E#i based on the absolute position information (x0, y0) reflected by the individual difference of the monocular image E0 according to equation (1).

[0223] In addition, the acquisition processing device obtains the centroid positions of the point light images PL and PR of the point lights of the light sources 32L and 32R on the known captured image as the individual difference point light position information (X L , Y L ) and (X R , Y R ).

[0224] The individual difference reflecting relative position information (dx#i, dy#i) and the individual difference spot light position information (x#i) obtained in the above-mentioned process of acquiring the individual difference reflecting relative position information and the like are L , Y L ) and (X R , Y R ) as part of the lens information, stored in Figure 2 In the storage unit 41.

[0225] When using the camera system 1 having the multi-eye interchangeable lens 20 mounted on the camera body, the individual difference reflecting relative position information (dx#i, dy#i) and the individual difference spot light position information (x L , Y L ) and (X R, Y R ), in the camera body 10, an installation error reflection position information calculation process is performed to obtain the relative optical axis center positions (dx1′, dy1′) to (dx4′, dy4′) of each monocular image E1 to E4 on the unknown captured image, etc. as the installation error reflection position information.

[0226] That is, in the camera body 10 ( Figure 2 ), when the poly-eye interchangeable lens 20 is mounted, the communication unit 56 communicates with the communication unit 42 of the poly-eye interchangeable lens 20, receives the lens information of the poly-eye interchangeable lens 20 transmitted from the communication unit 42, and provides the lens information to the position calculation unit 57. As described above, the position calculation unit 57 acquires the lens information provided from the communication unit 56.

[0227] The position calculation unit 57 waits for an unknown captured image to be captured, and acquires the unknown captured image, which is a captured image in which an arbitrary object appears. That is, in the camera system 1 having the poly-eye interchangeable lens 20 attached to the camera body 10, the position calculation unit 57 acquires the captured image captured by the image sensor 51 as the unknown captured image.

[0228] The position calculation unit 57 detects the point light images PL and PR of the point lights of the light sources 32L and 32R included in the unknown captured image, and further detects respective positions (detected light image positions) of the point light images PL and PR, such as the position of the center of gravity, as installation error point light position information (X L ', Y L ') and (X R ', Y R The position calculation unit 57 is a part of a processing unit that performs processing based on the detection result of the spot light images PL and PR, and calculates the position of the spot light according to the installation error spot light position information (X) as the detection result of the spot light images PL and PR. L ', Y L ') and (X R ', Y R '), specifies the imaged monocular image position as the position of the monocular image in the unknown captured image.

[0229] That is, the position calculation unit 57 calculates the position information of the light point (X) based on the installation error. L ', Y L ')(or(X R ', Y R ')) (detection light image position) and individual difference light position information (X) included in the lens information L , Y L )(or(X R , Y R))(stored light image position indicating the positions of the point light images PL and PR of the point lights of the light sources 32L and 32R emitted to the image sensor 51), specifies (calculates) the imaged monocular image position.

[0230] For example, the position calculation unit 57 calculates the position information of the light point (X) based on the installation error. L ', Y L ')(or(X R ', Y R ')) (detection light image position) and individual difference light position information (X L , Y L )(or(X R , Y R )) (storing the optical image position), correcting the relative optical axis center position (dx#i, dy#i) (representing the relative position information from the plurality of single-eye lenses 31 in the image sensor 51) as the individual difference reflected in the lens information i The storage monocular image position of each emission position of each imaging light emitted in the unknown captured image is designated as the imaging monocular image position, i.e., the (relative) optical axis center position (dx#i', dy#i'), as the installation error reflecting position information of each monocular image E1 to E4 included in the unknown captured image).

[0231] Specifically, first, the position calculation unit 57 obtains the installation error point light position information (X L ', Y L ')(or(X R ', Y R ')) and individual difference light position information (X L , Y L )(or(X R , Y R The rotation angle between them is taken as the (relative) rotation error θ Error .

[0232] For example, the position calculation unit 57 obtains the individual difference light position information (X) included in the lens information according to formula (2). L , Y L ) and (X R , Y R ) is used as the reference to indicate the position information of the connection installation error point light (X L ', Y L ') and (X R ', Y R ') is the relative angle of the direction of the line segment, as the rotation error θ Error .

[0233] The position calculation unit 57 calculates the rotation error θ obtained according to equation (2) Error The rotation reflects the relative position information as the individual difference contained in the lens information, and the (relative) optical axis center position (dx#i', dy#i') is obtained as the rotation error θ according to equation (3). Error The installation error of each of the monocular images E1 to E4 included in the unknown captured image reflects the position information.

[0234] Furthermore, the position calculation unit 57 obtains individual difference point light position information (X) included in the lens information as needed. L , Y L ) or (X R , Y R ) and the installation error point light position information (X L ', Y L ') or (X R ', Y R '), as the horizontal attachment error X Error and the attachment error Y in the vertical direction Error .

[0235] That is, for example, the position calculation unit 57 obtains the installation error point light position information (X L ', Y L ') and (X R ', Y R ') of the midpoint (X C ', Y C ') relative to the individual difference light position information (X L , Y L ) and (X R , Y R ) is the midpoint (X C , Y C ) as the horizontal attachment error X Error and the attachment error Y in the vertical direction Error .

[0236] On the camera body ( Figure 2 ), the area designation unit 52 can use the relative optical axis center positions (dx1′, dy1′) to (dx4′, dy4′) of the monocular images E1 to E4 on the unknown captured image as the installation error reflection position information obtained in the above-mentioned installation error reflection position information calculation processing to perform area designation processing for designating the area of ​​the monocular image E#i on the unknown captured image.

[0237] In the area specifying process, the area specifying unit 52 acquires the relative optical axis center position (dx#i′, dy#i′) as the mounting error reflecting position information of the monocular images E1 to E4 included in the unknown captured image supplied from the position calculating unit 57 .

[0238] Then, the area designation unit 52 designates the area of ​​the monocular images E1 to E4 on the unknown captured image based on the optical axis center position (dx#i', dy#i') as the installation error reflection position information. That is, for example, on the xy coordinate system when the individual difference reflection relative position information (dx#i, dy#i) is obtained, the area designation unit 52 designates a rectangular area of ​​a predetermined size centered on the installation error reflection position information (dx#i', dy#i') on the unknown captured image as the area of ​​the monocular image E#i (i=1, 2, 3, 4). In addition, the area designation unit 52 designates a rectangular area of ​​a predetermined size centered on the origin on the unknown captured image as the area of ​​the monocular image E0.

[0239] Thereafter, the area specifying unit 52 extracts the monocular images E0 to E4 from the captured images, and outputs the extracted images as area specifying result information.

[0240] As described above, in a camera system 1 equipped with a multi-eye interchangeable lens 20, the multi-eye interchangeable lens 20 includes a feed unit 23 and light sources 32L and 32R provided with single-eye lenses 310 to 314 arranged so as not to overlap in the optical axis direction of the lens barrel optical axis (as observed), and for a single-eye image E#i on an unknown captured image, the optical axis center position (dx#i', dy#i') is obtained as position information reflecting the installation error on the single-eye image E#i.

[0241] Therefore, regardless of the feeding state of the feeding unit 23, it is possible to easily obtain an image having multiple viewpoints, that is, an image having multiple viewpoints in the single-eye lens 31 from an unknown captured image. i Monocular image E#i of the viewpoint at the position of .

[0242] Furthermore, the coordinates representing the baseline L0#i ( Figure 5 ) direction of the baseline angle tan -1 (dy#i / dx#i)), and accurate parallax information that is not affected by the attachment error in the rotation direction of the polycular interchangeable lens 20 can be obtained using the baseline angle.

[0243] Here, for example, it is assumed that an xy coordinate system is used with the optical axis center position (x0, y0) of the monocular image E0 on the known captured image as the origin, and the attachment error X in the horizontal direction is Errorand the attachment error Y in the vertical direction Error is 0. In this case, in the unknown captured image, the optical axis center position as the mounting error reflecting position information (x0′, y0′) of the monocular image E0 is the origin, and the area of ​​the monocular image E0 is the area centered on the origin.

[0244] At the same time, the attachment error X in the horizontal direction Error Or the attachment error Y in the vertical direction Error If it is not 0, in the unknown captured image, the optical axis center position of the monocular image E0 is offset from the origin by the attachment error X in the horizontal direction. Error and the attachment error Y in the vertical direction Error .

[0245] In this case, assuming that the optical axis center position of the installation error reflecting position information (x0', y0') of the monocular image E0 is the origin, when a rectangular area having a predetermined size centered on the origin is designated as the area of ​​the monocular image E0, the optical axis center position of the actual installation error reflecting position information (x0', y0') is offset from the origin, and therefore, a rectangular area on the unknown captured image having a predetermined size centered on a position offset from the actual optical axis center position of the monocular image E0 is designated as the area of ​​the monocular image E0.

[0246] As a result, for each of the other monocular images E1 to E4, a rectangular area on the unknown captured image having a predetermined size centered at a position offset from the optical axis center position (x#i', y#i') of the monocular image E#i is designated as the area of ​​the monocular image E#i.

[0247] That is, the attachment error X in the horizontal direction Error Or the attachment error Y in the vertical direction Error When it is not 0, a rectangular area on the unknown captured image having a predetermined size centered on a position offset by the same translation amount from the optical axis center position (x#i′, y#i′) of the monocular image E#i is designated as the area of ​​the monocular image E#i of each of the monocular images E0 to E4.

[0248] Note that the attachment error X in the horizontal direction Error Or the attachment error Y in the vertical direction Error The case where it is not 0 and the attachment error X in the horizontal direction Error And the attachment error Y in the vertical direction Error In both cases where the angle is 0, the baseline angle (tan -1(dy#i / dx#i)).

[0249] Therefore, the baseline angle obtained for the monocular image E#i centered at a position offset by the same translation amount from the optical axis center position (x#i', y#i') is the same as the baseline angle obtained for the monocular image E#i centered at the optical axis center position (x#i', y#i') on the unknown captured image.

[0250] That is, even if the attachment error X in the horizontal direction Error Or the attachment error Y in the vertical direction Error When it is not 0, the attachment error X in the horizontal direction can also be obtained. Error and the attachment error Y in the vertical direction Error The same baseline angle as when 0. Then, accurate parallax information that is not affected by the attachment error of the polycular interchangeable lens 20 can be obtained from the monocular image E#i having a designated area on the unknown captured image using the baseline angle.

[0251] Note that in the present embodiment, individual difference reflecting position information and individual difference point light position information are obtained for each multi-eye interchangeable lens 20, and are included and stored in the lens information. However, for each model of the multi-eye interchangeable lens 20, a common value may be adopted as the individual difference reflecting position information and the individual difference point light position information. In the case where common individual difference reflecting position information and individual difference point light position information are adopted for each model of the multi-eye interchangeable lens 20, the individual difference reflecting relative position information (dx#i, dy#i) and the individual difference point light position information (x#i) of each model are included and stored in the lens information. L , Y L ) and (X R , Y R ) is included in equations (2) and (3), so that the camera body 10 can obtain the rotation error θ of equation (2) Error As for the final optical axis center position (dx#i′, dy#i′) as the mounting error reflecting position information of formula (3), it is sufficient as long as the model of the polycular interchangeable lens 20 can be identified.

[0252] <Configuration Example of Image Processing Unit 53>

[0253] Figure 7 It is shown Figure 2 A block diagram of a configuration example of the image processing unit 53 in FIG.

[0254] Figure 7 For example, a configuration example of a part of the image processing unit 53 that performs image processing for obtaining parallax information is shown.

[0255] exist Figure 7, the image processing unit 53 includes a calibration data generating unit 101 , a calibration data storing unit 102 , an interpolation unit 103 , and a parallax information generating unit 104 .

[0256] The image processing unit 53 generates disparity information about disparity by using the monocular image supplied from the area specifying unit 52. Here, the disparity information includes a disparity value expressing the disparity by the number of pixels, a distance in the depth direction corresponding to the disparity, and the like.

[0257] The parallax information obtained using the monocular image is controlled by the monocular lens 31 i Therefore, in order to remove the influence of the position of the single-eye lens 31 i The image processing unit 53 performs calibration to generate a signal about the single-eye lens 31. i The parameters such as the position, lens distortion, etc. are used as calibration data.

[0258] In the calibration, for example, calibration data is generated from a calibration image which is a monocular image on a captured image obtained by imaging a plane chart for calibration (hereinafter also referred to as a calibration chart) which is a known object.

[0259] That is, in calibration, the focus position of the camera system 1 is controlled to a certain predetermined distance (position), and a calibration chart installed at the focus position is photographed. Then, calibration data for the focus position of the camera system 1 in which the focus position is controlled to a certain predetermined distance is generated using a calibration image that is a monocular image obtained by imaging the calibration chart.

[0260] The calibration data generated as described above is calibration data of the focus position of the camera system 1 when the calibration chart is captured. Therefore, for example, in a case where the focus position in the case of imaging a general object is different from the focus position in the case of imaging the calibration chart, if the calibration data of the focus position of the camera system 1 when the calibration chart is captured is used to perform parallax information calculation using a monocular image appearing in a captured image obtained by imaging the general object, the accuracy of the parallax information decreases.

[0261] Therefore, the image processing unit 53 can obtain high-precision parallax information by interpolating the calibration data.

[0262] The calibration data generation unit 101 generates calibration data for each of a plurality of feed amounts from the feed amount detection unit 64 by using the monocular image supplied from the area specification unit 52 , and supplies the calibration data to the calibration data storage unit 102 .

[0263] That is, the calibration data generating unit 101 generates calibration data of multiple feed amounts corresponding to multiple focal positions from a monocular image as calibration images at multiple focal positions obtained by imaging a calibration chart at multiple focal positions corresponding to multiple feed amounts of the feed unit 23 during calibration, and provides the calibration data to the calibration data storage unit 102.

[0264] The calibration data storage unit 102 stores the calibration data of a plurality of feed amounts supplied from the calibration data generation unit 101 .

[0265] The interpolation unit 103 generates calibration data of the feed amount when imaging a general object (an object other than the calibration image) (hereinafter also referred to as general imaging) provided by the feed amount detection unit 64 by interpolating using the calibration data of multiple feed amounts stored in the calibration data storage unit 102, and provides the generated calibration data to the parallax information generation unit 104.

[0266] The parallax information generating unit 104 generates parallax information by using the calibration data from the interpolating unit 103 and a monocular image on a captured image captured by general imaging (hereinafter also referred to as a general captured image) supplied from the area specifying unit 52. The general captured image is equivalent to an unknown captured image.

[0267] According to the camera system 1 including the image processing unit 53 as described above, for example, an RGB image as a monocular image having an image quality similar to that of a general monocular camera and parallax information as depth information can be simultaneously obtained. The parallax information can be used for, for example, image processing such as refocusing to generate (reconstruct) an image focused on an arbitrary object, generating an obstacle-removed image in which obstacles for a desired object desired by a user have been removed, lens simulation for simulating a lens having arbitrary characteristics, and synthesis taking into account CG depth and actual shooting.

[0268] With the camera system 1 , for example, after the camera system 1 is manufactured, the operation mode of the camera system 1 is set to a calibration mode for performing calibration, and calibration is performed in a manufacturing plant or the like of the camera system 1 .

[0269] That is, in the camera system 1, the focus of the poly-interchangeable lens 20 is controlled to a focus position (hereinafter also referred to as a reference focus position) at which calibration data is generated, and a calibration chart is mounted at the reference focus position and photographed. Then, in the camera system 1, the calibration data generation unit 101 generates calibration data (hereinafter also referred to as a reference feed amount) of the feed unit 23 corresponding to the reference focus position based on a calibration image obtained by imaging the calibration chart, and causes the calibration data storage unit 102 to store the calibration data. Calibration is performed for a plurality of reference focus positions, thereby generating calibration data for a plurality of reference feed amounts corresponding to the plurality of reference focus positions.

[0270] When calibration is performed and the camera system 1 is shipped from the manufacturing plant, the operation mode of the camera system 1 is set to a general imaging mode in which general imaging is performed. In the general imaging mode, the feed amount of the feed unit 23 (hereinafter also referred to as imaging feed amount) corresponding to the focal position when a general photographic image is photographed (hereinafter also referred to as imaging focal position) is provided from the feed amount detection unit 64 to the interpolation unit 103.

[0271] The interpolation unit 103 generates calibration data of an imaging feed amount corresponding to an imaging focus position by interpolation using the calibration data of a plurality of reference feed amounts corresponding to a plurality of reference focus positions stored in the calibration data storage unit 102, and supplies the generated calibration data to the parallax information generation unit 104. Note that in the case where there is calibration data for all positions, interpolation is not necessary.

[0272] The parallax information generating unit 104 generates parallax information from a monocular image on a normal captured image captured in a state where the imaging feed is sent out of the feed unit 23, using the calibration data for the imaging feed unit from the interpolating unit 103. As a result, accurate parallax information with suppressed influence of lens distortion and the like is generated.

[0273] Figure 8 is a diagram for describing calibration performed by the camera system 1 .

[0274] In the calibration, a position P at a predetermined distance Z mm from the camera system 1 is set as a reference focus position P, a calibration chart is mounted at the reference focus position P, and the camera system 1 images the calibration chart.

[0275] Figure 8 The calibration chart shown is, for example, an object on which a grid-like pattern is drawn, but any object having a known positional relationship or the like may be employed as the calibration chart.

[0276] In the calibration, calibration data of the feed amount P corresponding to the reference focus position P is generated from a calibration image obtained by imaging a calibration chart at the reference focus position P.

[0277] In the calibration image obtained by imaging the calibration chart by the camera system 1, for example, due to the single-eye lens 31 i That is, there is a positional deviation between the true position where the object should appear in the calibration image (the position where the object should originally appear without lens distortion, etc.) and the position where the object actually appears in the calibration image.

[0278] Since the calibration image is a monocular image on a captured image obtained by installing and imaging a calibration chart (which is a known object) at a focal position P (which is a known position), the true position where the object should appear, that is, the true position where each part of the calibration chart (e.g., a grid point) should appear on the calibration image, can be obtained in advance by calculation.

[0279] Furthermore, the actual position of the object appearing in the calibration image can be obtained from the calibration image.

[0280] The calibration data generation unit 101 obtains the actual position where the object (e.g., the grid point of the calibration image) appears from the calibration image. Then, the calibration data generation unit 101 generates information about the positional deviation between the actual position where the object appears and the real position where the object should appear, which can be obtained in advance by calculation, as calibration data.

[0281] Note that as calibration data, so-called internal parameters and external parameters of the camera system 1 may be generated, but here, to simplify the description, information on positional deviations of objects in a calibration image obtained by imaging a calibration chart by the camera system 1 is generated as calibration data.

[0282] Fig. 9 It is a diagram for describing generation of calibration data of a plurality of feed amounts corresponding to a plurality of reference focus positions.

[0283] When generating calibration data of multiple feed amounts corresponding to multiple reference focus positions, first, position P1 at a predetermined distance Z1 mm from the camera system 1 is set as the reference focus position P1, a calibration chart is installed at the reference focus position P1, and the camera system 1 images the calibration chart.

[0284] Then, a position P2 at a distance Z2 mm different from the distance Z1 mm from the camera system 1 is set as a reference focus position P2, a calibration chart is mounted at the reference focus position P2, and the camera system 1 images the calibration chart.

[0285] Furthermore, a position P3 at a distance Z3 mm different from the distances Z1 and Z2 mm from the camera system 1 is set as a reference focus position P3 , a calibration chart is installed at the reference focus position P3 , and the camera system 1 images the calibration chart.

[0286] Note that in Fig. 9 the distances Z1 mm, Z2 mm, and Z3 mm have a relationship represented by the expression Z1 mm < Z2 mm < Z3 mm.

[0287] The camera system 1 controls the focus position to position P1 and images the calibration chart mounted at position P1. Through this imaging, a calibration image at the reference focus position P1, that is, a calibration image having position P1 as the reference focus position, is obtained.

[0288] Similarly, the camera system 1 controls the focus position to positions P2 and P3 and images the calibration charts mounted at positions P2 and P3. Through the imaging, calibration images at the reference focus positions P2 and P3, that is, calibration images having positions P2 and P3 as the reference focus positions, are obtained.

[0289] Note that there is no particular limitation on the imaging order of the calibration charts mounted at the reference focus positions P1, P2, and P3.

[0290] Furthermore, here, three different positions P1, P2, and P3 are adopted as the reference focus positions, but in addition to the three different positions, two different positions or four or more different positions can also be adopted as the reference focus positions.

[0291] As described above, in the camera system 1, calibration data of a plurality of feed amounts corresponding to a plurality of reference focus positions (reference focus positions P1, P2, P3) are generated from the calibration images obtained by imaging the calibration chart by the camera system 1 that controls the focus position to a plurality of positions (reference focus positions P1, P2, and P3).

[0292] Fig.10 is a diagram for describing general imaging performed by the camera system 1.

[0293] In general imaging, the camera system 1 sets the position P4 at an arbitrary distance Z4 mm from the camera system 1 as the imaging focus position P4 and images a general object existing at the imaging focus position P4.

[0294] Note that in Fig.10 the position P4 does not match any of the positions P1, P2, and P3 and is a position farther than position P1 and closer than position P2.

[0295] In the camera system 1, image processing is performed on a general captured image obtained by imaging an object in the camera system 1 that controls the focus position to the imaging focus position P4.

[0296] Here, in the camera system 1, in a case where the imaging focus position when performing general imaging does not match any of the multiple reference focus positions during calibration, if image processing is performed on the general captured image using calibration data such as any of the multiple reference feed amounts corresponding to the multiple reference focus positions, inappropriate image processing may be performed.

[0297] exist Fig.10 Since the imaging focus position P4 does not match any of the reference focus positions P1, P2 and P3, if image processing is performed on a general captured image captured by the camera system 1 in which the focus position is controlled to the imaging focus position P4, using calibration data such as any of the reference feed amounts P1, P2 and P3 corresponding to the reference focus positions P1, P2 and P3, that is, the imaging feed amount is set to the feed amount corresponding to the imaging focus position P4, then inappropriate image processing may be performed.

[0298] Therefore, in the camera system 1, in order to perform appropriate image processing on a general captured image, the interpolation unit 153 ( Figure 7 ) uses the calibration data of the reference feed amounts P1, P2, P2 corresponding to the reference focus positions P1, P2 and P3, and generates the calibration data of the imaging feed amount P4 corresponding to the imaging focus position P4 by interpolation.

[0299] That is, when the focal position (i.e., the feed amount of the feed unit 23) is different, the lens condition (lens state) of the camera system 1 is different. Therefore, even in the same pixel of a general captured image, the position deviation (amount) is different between the case where the feed amount of the feed unit 23 is the feed amount corresponding to a certain focal position and the case where the feed amount is the feed amount corresponding to another focal position.

[0300] Therefore, if the parallax information is generated for a general captured image using calibration data of a reference feed amount that does not match the imaging feed amount, the error of the parallax information caused by the above-mentioned positional deviation is not properly corrected, and accurate parallax information may not be obtained.

[0301] In order to perform appropriate image processing on a general captured image, that is, to obtain accurate parallax information, for example, the camera system 1 images a calibration chart at a plurality of (different) reference focus positions, and generates calibration data of a plurality of reference feed amounts corresponding to the plurality of reference focus positions from the calibration images at the plurality of reference focus positions obtained by imaging.

[0302] Furthermore, the camera system 1 generates calibration data for an imaging feed amount corresponding to an imaging focus position by interpolating calibration data using a plurality of reference feed amounts corresponding to a plurality of reference focus positions, and performs image processing such as generating parallax information using the calibration data for the imaging feed amount.

[0303] Fig.11 3 is a diagram for describing generation by interpolating calibration data of the imaging feed amount corresponding to the imaging focal point position P4.

[0304] exist Fig.11 , the vertical direction (vertical axis) represents calibration data, and the horizontal direction (horizontal axis) represents the feeding amount of the feeding unit 23.

[0305] exist Fig.11 , calibration data for each of the reference feed amounts P1, P2, and P3 corresponding to the reference focus positions P1, P2, and P3 is represented by a circle.

[0306] The interpolation unit 103 generates calibration data of the imaging feed amount P4 corresponding to the imaging focus position P4 (the portion indicated by a triangle in the figure) by performing linear interpolation or another interpolation using calibration data of at least two or more of the reference feed amounts P1, P2 and P3.

[0307] As described above, the camera system 1 generates calibration data for the imaging feed amount P4 by interpolation using the calibration data for the plurality of feed amounts P1 , P2 , and P3 .

[0308] Therefore, in a case where the feed amount of the feed unit 23 is the imaging feed amount P4, appropriate image processing can be performed on a general captured image at the imaging focus position P4 using the calibration data of the imaging feed amount P4.

[0309] For example, in the case where nonlinear distortion exists in a monocular image extracted from a captured image (in which an object appears), image processing using calibration data in the image processing unit 53 is useful. According to the image processing using calibration data in the image processing unit 53, image processing in which the distortion of the monocular image is corrected can be performed. Note that in calibration, in addition to each pixel of the monocular image, information on positional deviations from the actual position where an object as calibration data should appear can be generated for some pixels, and this information can be generated by interpolation in the spatial direction for other pixels at the time of general imaging.

[0310] <Configuration Example of Light Sources 32L and 32R>

[0311] Fig.12 : is a cross-sectional view showing a configuration example of the light sources 32L and 32R.

[0312] Here, hereinafter, the light sources 32L and 32R will be referred to as the light source 32 unless there is a need to distinguish them.

[0313] exist Fig.12 In the embodiment, the light source 32 includes a housing 121 , an LED 122 and a lens 123 .

[0314] The housing 121 is, for example, an elongated cylindrical housing, and accommodates the LED 122 and the lens 123 therein.

[0315] The LED 122 emits light as point light.

[0316] The lens 123 collects the point light emitted from the LED 122 .

[0317] In the light source 32 configured as described above, point lights emitted from the LED 122 are collected by the lens 123 .

[0318] Therefore, from Fig.12 The point light emitted by the light source 32 in FIG. 1 is (ideally) gathered at one point and then diffused, and is therefore non-parallel light (light that is not parallel light).

[0319] Here, as from Fig.12 The point where the non-parallel light emitted by the light source 32 in the image is focused is also called a focusing point.

[0320] Hereinafter, assuming that the light source 32 emits non-parallel light as point light, a method of detecting the feeding amount of the feeding unit 23 will be described.

[0321] <Configuration Example of Multi-eye Interchangeable Lens 20>

[0322] Fig.13 2 is a cross-sectional view showing a configuration example of the polycular interchangeable lens 20 .

[0323] As reference Figure 1 The feeding unit 23 is provided with a single-eye lens 31 i The feeding unit 23 is configured to be movable in the cylindrical lens barrel 21 along the optical axis direction of the lens barrel optical axis (in the figure, the up-down direction), and can be fed from the front side (from the image sensor 51 side) to the depth side. The feeding of the feeding unit 23 causes the single-eye lens 31 to i Move and adjust focus.

[0324] <First method for detecting feed amount>

[0325] Fig.14 2 is a diagram for describing a first detection method of detecting a feeding amount of the feeding unit 23 .

[0326] Fig.14A shows a cross section of the polycular interchangeable lens 20 in a minimum feeding state (wide end) in which the feeding unit 23 is fed out to a minimum extent (on the frontmost side). Fig.14 B shows a cross section of the polycular interchangeable lens 20 in a maximum feed state (distal end) in which the feed unit is fed out to the maximum extent (on the deepest side).

[0327] When adopting the first detection method, the camera system 1 is configured so that the image sensor 51 is located between the gathering point where the non-parallel light as point light is gathered when the feed unit 23 is in the minimum feed state and the gathering point where the non-parallel light as point light is gathered when the feed unit 23 is in the maximum feed state.

[0328] Furthermore, the light source 32 emits non-parallel light as point light in the same direction as the barrel optical axis.

[0329] As described above, when the image sensor 51 is located between the gathering point of the point light when the feeding unit 23 is in the minimum feeding state and the gathering point of the point light when the feeding unit 23 is in the maximum feeding state, the difference between the size of the point light image in the minimum feeding state (hereinafter also referred to as the point size) (for example, the radius of the point light image, etc.) and the point size in the maximum feeding state is minimized.

[0330] In the case where the light source 32 emits non-parallel light as spot light, the spot size varies according to the feeding amount of the feeding unit 23 , and therefore, the feeding amount of the feeding unit 23 can be detected from the spot size.

[0331] Fig.15 : is a (cross-sectional) diagram showing an example of a change in the spot size of non-parallel light as spot light.

[0332] like Fig.15 As shown, the spot size at the focusing point is minimized, and the spot size increases as the distance from the focusing point increases in the direction of the optical axis of the lens barrel.

[0333] In a case where the image sensor 51 is located between the collecting point where the spot light is collected when the feeding unit 23 is in the minimum feeding state and the collecting point where the spot light is collected when the feeding unit 23 is in the maximum feeding state, there may be two feeding amounts where the dot size becomes the same.

[0334] That is, for example, Fig.15 As shown, two positions located at the same distance from the focusing point in the direction of the optical axis of the lens barrel have different feed amounts but the same spot size.

[0335] As mentioned above, the existence of two feeds for a particular dot size is also known as binary uncertainty.

[0336] In the case where there are two feed amounts for a certain dot size, it is necessary to specify the true feed amount, that is, the feed amount in a state where a spot light image of the dot size is obtained from the two feed amounts, and resolve the binary uncertainty.

[0337] Examples of methods of specifying a true feed amount from two feed amounts when there are two feed amounts for a specific dot size include a method of performing template matching of a point light image, and a method of changing the feed amount to be large or small and detecting a direction of change in which the dot size increases or decreases based on the direction of change in the feed amount.

[0338] Fig.16 is a flowchart for describing an example of a process of detecting a feed amount by a first detection method.

[0339] In accordance with Fig.16 In the process of the flowchart, binary uncertainty is resolved by performing template matching of the point light image.

[0340] In step S111 , the point light image detection unit 62 acquires lens information from the polycular interchangeable lens 20 via the communication unit 56 , and the process proceeds to step S112 .

[0341] In step S112, the control unit 61 waits for the user to perform an operation to adjust the focus, and transmits information for specifying the focus to the multi-eye interchangeable lens 20 via the communication unit 56 according to the operation. In the multi-eye interchangeable lens 20, the control unit 43 moves (feeds) the feed unit 23 according to the information for specifying the focus transmitted via the communication unit 56, and the process proceeds to step S113. Note that the control unit 61 can automatically adjust the focus by an autofocus function or the like without a user operation.

[0342] In step S113 , the image sensor 51 images a captured image and supplies the captured image to the area specifying unit 52 , the position calculating unit 57 , and the point light image detecting unit 62 , and the process proceeds to step S114 .

[0343] In step S114, the point light image detection unit 62 detects a point light image from the captured image from the image sensor 51 (near the individual difference point light position information) based on the individual difference point light position information included in the lens information acquired from the poly-eye interchangeable lens 20, and provides the image as a point light image to the feed amount detection unit 64, and the processing proceeds to step S115.

[0344] Here, as reference Figure 6As described above, a value common to each model of the poly-eye interchangeable lens 20 may be adopted as the individual difference light position information. In the case where the common individual difference light position information is adopted for each model of the poly-eye interchangeable lens 20, the individual difference light position information of each model is pre-stored in the camera body 10, so that the camera body 10 does not need to acquire the individual difference light position information (lens information including the individual difference light position information) from the poly-eye interchangeable lens 20. This similarly applies to the embodiments described below.

[0345] In step S115 , the feed amount detection unit 64 acquires point light image information that has not been matched with a point light image from the point light image information associated with the feed amount in the feed amount information stored in the feed amount information storage unit 63 , and the process proceeds to step S116 .

[0346] Here, in the case of adopting the first detection method, the feed amount information storage unit 63 stores feed amount information, wherein each of multiple feed amounts is associated with point light image information when the feed unit 23 feeds out the feed amount, wherein the image as a point light image is used as the point light image information.

[0347] In step S116, the feed amount detection unit 64 performs (template) matching between the point light image detected from the captured image in step S114 (hereinafter also referred to as the detected point light image) and the point light image information acquired in the immediately preceding step S115, and the processing proceeds to step S117.

[0348] In step S117 , the feed amount detection unit 64 determines whether matching with the detected point light images has been performed on all the point light image information associated with the feed amounts in the feed amount information stored in the feed amount information storage unit 63 .

[0349] In step S117 , in a case where it is determined that all the point light image information associated with the feed amount in the feed amount information stored in the feed amount information storage unit 63 has not been completed, the process returns to step S115 .

[0350] Then, in step S115 , point light image information that has not been matched with a point light image is acquired from the point light image information associated with the feed amount in the feed amount information stored in the feed amount information storage unit 63 , and similar processing is repeated thereafter.

[0351] Furthermore, in step S117 , in the event that it is determined that all the point light image information associated with the feed amounts in the feed amount information stored in the feed amount information storage unit 63 has been completed, the process proceeds to step S118 .

[0352] In step S118, the point light image information that best matches the detected point light image is selected from the point light image information associated with the feed amount in the feed amount information stored in the feed amount information storage unit 63, and the feed amount associated with the point light image information is detected as the feed amount of the feed unit 23, and is provided to the image processing unit 53, and the processing is terminated. Here, for example, in the case where the current focus position, which is the focus position after the focus adjustment in step S112, is not used as the focus position when generating the feed amount information, the current focus position and the focus position when generating the feed amount information with the feed amount detected in step S118 do not completely match each other, and the accuracy of the feed amount detected in step S118 may deteriorate. Therefore, as the feed amount information, in addition to the feed amount and the point light image information, information in which the focus position when the feed unit 23 is fed out with the feed amount is associated can also be used. In this case, in step S112, when the current focus position does not match any of the focus positions included in the feed amount information, the current focus position may be mapped (readjusted) to the focus position closest to the current focus position among the focus positions included in the feed amount information. As a result, in step S118, the accurate feed amount may be detected.

[0353] Note that Fig.16 In the method, the feed amount is detected by using an image as a point light image as point light image information and performing image matching. Alternatively, for example, the feed amount can be detected by using a one-dimensional intensity distribution or a two-dimensional intensity distribution of the point light image as point light image information and performing matching of the one-dimensional intensity distribution or the two-dimensional intensity distribution.

[0354] Furthermore, in the case where one or more light sources 32 are provided, the first detection method may be adopted.

[0355] <Second Detection Method of Feed Amount>

[0356] Fig.17 2 is a diagram for describing a second detection method of detecting the feeding amount of the feeding unit 23 .

[0357] When the second detection method is adopted, similar to the first detection method, the camera system 1 is configured so that the image sensor 51 is located between the gathering point where the non-parallel light as point light is gathered when the feed unit 23 is in the minimum feed state and the gathering point where the non-parallel light as point light is gathered when the feed unit 23 is in the maximum feed state.

[0358] Furthermore, similarly to the first detection method, the light sources 32L and 32R emit non-parallel light as point light in the same direction as the barrel optical axis.

[0359] Note that when the second detection method is adopted, the camera system 1 is configured so that the focusing point of the point light emitted by the light source 32L and the focusing point of the point light emitted by the light source 32R having respective feed amounts are located at points having different distances from the image sensor 51 .

[0360] exist Fig.17 In the embodiment, the distance from the image sensor 51 to the focusing point of the point light emitted from the light source 32L having a certain feed amount is different from the distance from the image sensor 51 to the focusing point of the point light emitted from the light source 32R. Fig.17 , the image sensor 51 exists on the lower side in the figure, and the focusing point of the point lights emitted by the light source 32L is closer to the image sensor 51 than the focusing point of the point lights emitted by the light source 32R.

[0361] In this case, the binary uncertainty cannot be resolved by only the spot size of the spot light emitted by one of the light sources 32L and 32R, but the binary uncertainty can be resolved by a combination of the spot sizes of the spot lights emitted by the light sources 32L and 32R.

[0362] Fig.18 is a flowchart for describing an example of a process of detecting a feed amount by the second detection method.

[0363] In accordance with Fig.18 In the process of the flowchart of FIG. 32 , binary ambiguity is resolved by using a combination of the spot sizes of the spot lights emitted by the light sources 32L and 32R.

[0364] In steps S121 to S123, the Fig.16 The processing is similar to the processing in steps S111 to S113 in .

[0365] Then, in step S124, the point light image detection unit 62 detects (as an image) a point light image PL of the point light emitted from the light source 32L and a point light image PR of the point light emitted from the light source 32R from the captured image from the image sensor 51 based on the individual difference point light position information contained in the lens information acquired from the multi-eye interchangeable lens 20, and the processing proceeds to step S125.

[0366] In step S125 , the point light image detection unit 62 detects the dot sizes of the point light images PL and PR and supplies the dot sizes to the feed amount detection unit 64 , and the process proceeds to step S126 .

[0367] In step S126, the feed amount detection unit 64 selects point light image information that matches the combination of dot sizes of the point light images PL and PR detected in step S125 (hereinafter also referred to as a combination of detected dot sizes) from the point light image information associated with the feed amount in the feed amount information stored in the feed amount information storage unit 63. In addition, in step S126, the feed amount detection unit 64 detects the feed amount associated with the point light image information that matches the combination of detected dot sizes as the feed amount of the feed unit 23, provides the feed amount to the image processing unit 53, and terminates the processing.

[0368] Here, in the case of adopting the second detection method, the feed amount information storage unit 63 stores feed amount information, wherein each of multiple feed amounts is associated with point light image information when the feed unit 23 is fed out with the feed amount, wherein a combination of the dot sizes of the point light images PL and PR is used as the point light image information.

[0369] In the second detection method, among the feed amount information in which each of the multiple feed amounts is associated with a combination of the dot sizes of the point light images PL and PR which is the point light image information when the feed unit 23 is fed out with the feed amount, the feed amount associated with the combination of the dot sizes of the point light images PL and PR that matches the combination of the detected point sizes is detected as the feed amount of the feed unit 23.

[0370] According to the second detection method, the binary uncertainty is resolved, and the feeding amount of the feeding unit 23 can be detected based on (a combination of) the detection point sizes.

[0371] Note that in the case where two or more light sources 32 are provided, the second detection method may be employed.

[0372] <Third Detection Method of Feed Amount>

[0373] Fig.19 2 is a diagram for describing a third detection method of detecting the feeding amount of the feeding unit 23 .

[0374] Fig.19 A shows a cross section of the polyocular interchangeable lens 20 in a minimum feed state in which the feed unit 23 is fed to the minimum extent (at the front). Fig.19 B shows a cross section of the polyocular interchangeable lens 20 in a maximum feed state in which the feed unit is fed to the maximum extent (on the deepest side).

[0375] If the third detection method is adopted, the camera system 1 is configured so that when the feeding unit 23 moves from the minimum feeding state to the maximum feeding state, the focusing point where the non-parallel light as point light is focused is located on one of the depth side and the front side including the image sensor 51 .

[0376] Furthermore, the light source 32 emits non-parallel light as point light in the same direction as the barrel optical axis.

[0377] As described above, in the case where the feeding unit 23 moves from the minimum feeding state to the maximum feeding state, when the focusing point where the non-parallel light as the point light is focused is located at one of the depth side and the front side including the image sensor 51, the difference between the dot size of the point light image in the minimum feeding state and the dot size in the maximum feeding state is maximized. In addition, in the case where the feeding unit 23 moves from the minimum feeding state to the maximum feeding state, the dot size decreases monotonically or increases monotonically. Therefore, binary uncertainty does not occur.

[0378] Fig. 20 1 is a diagram for describing a state in which a collecting point where non-parallel light as point light is collected is located at one of the rear side and the front side including the image sensor 51 when the feeding unit 23 moves from the minimum feeding state to the maximum feeding state.

[0379] Fig. 20 A shows that when the feeding unit 23 moves from the minimum feeding state to the maximum feeding state, the collecting point where the non-parallel light as the point light is collected is located on the front side including the image sensor 51 (with the single-eye lens 31 disposed thereon). i the state of the side opposite to the side on which the

[0380] exist Fig. 20 In A, when the feeding unit 23 moves from the minimum feeding state to the maximum feeding state, the dot size decreases monotonically.

[0381] Fig. 20 B shows a state in which, when the feeding unit 23 moves from the minimum feeding state to the maximum feeding state, the focusing point where the non-parallel light as the point light is focused is located on the depth side including the image sensor 51 (the side where the single-eye lens 31i is provided).

[0382] exist Fig. 20 In B, when the feeding unit 23 moves from the minimum feeding state to the maximum feeding state, the dot size increases monotonically.

[0383] When the third detection method is adopted, the feed amount information storage unit 63 stores feed amount information, wherein each of multiple feed amounts is associated with point light image information when the feed unit 23 is fed out with the feed amount, wherein the dot size of the point light image is used as the point light image information.

[0384] Then, in the third detection method, similarly to the second detection method, the point light image detection unit 62 detects the dot size of the point light image.

[0385] In addition, in the feed amount detection unit 64, among the feed amount information in which each of the multiple feed amounts is associated with the dot size of the point light image which is the point light image information when the feed unit 23 is fed out with the feed amount, the feed amount associated with the dot size which is the point light image information matching the dot size detected by the point light image detection unit 62 is detected as the feed amount of the feed unit 23.

[0386] According to the third detection method, binary uncertainty does not occur, and the feeding amount of the feeding unit 23 can be detected according to the dot size.

[0387] Note that in the case where one or more light sources 32 are provided, a third detection method may be employed.

[0388] <Fourth Detection Method of Feed Amount>

[0389] Fig.21 2 is a diagram for explaining a fourth detection method for detecting the feeding amount of the feeding unit 23. Note that the fourth detection method can be applied not only to the case where the point light is non-parallel light but also to the case where the point light is parallel light as described below.

[0390] Right now, Fig.21 A cross section of a polyocular interchangeable lens 20 is shown.

[0391] In the case of adopting the fourth detection method, the camera system 1 is configured so that the light source 32 emits point light in an oblique direction tilted from the barrel optical axis direction.

[0392] exist Fig.21 , the light source 32 emits point light in a direction from the peripheral portion toward the central portion of the image sensor 51.

[0393] Fig. 22 2 is a diagram showing the irradiation position of the spot light in the case where the feeding unit 23 is in the minimum feeding state and the irradiation position of the spot light in the case where the feeding unit 23 is in the maximum feeding state.

[0394] Fig. 22 A shows the irradiation position of the spot light when the feeding unit 23 is in the minimum feeding state, Fig. 22 B shows the irradiation position of the spot light when the feeding unit 23 is in the maximum feeding state.

[0395] exist Fig. 22 In the figure, the irradiation position of the point light (i.e., the position of the point light images PL' and PR' of the point light emitted by the light sources 32L and 32R) when the feeding unit 23 is in the minimum feeding state is the position closest to the outer side of the image sensor 51 (the captured image) within the movable range of the point light images PL and PR of the point light.

[0396] When the feeding unit 23 moves from the minimum feeding state to the maximum feeding state, the point light images PL and PR move toward the center of the image sensor 51 .

[0397] Then, the irradiation position of the point light (i.e., the position of the point light images PL" and PR" of the point light emitted by the light sources 32L and 32R) when the feeding unit 23 is in the maximum feeding state is the position closest to the center side of the image sensor 51 within the movable range of the point light images PL and PR of the point light.

[0398] Fig.23 is a diagram showing an example of a captured image in which the point light images PL′ and PR′ appear when the feed unit 23 is in the minimum feed state, and a captured image in which the point light images PL″ and PR″ appear when the feed unit 23 is in the maximum feed state.

[0399] exist Fig.23 , the point light images PL′ and PR′ in the case where the feeding unit 23 is in the minimum feeding state are located on the outermost peripheral side of the captured image.

[0400] When the feeding unit 23 moves from the minimum feeding state to the maximum feeding state, the point light images PL and PR move toward the center of the captured image.

[0401] Then, the point light images PL" and PR" in the case where the feeding unit 23 is in the maximum feeding state are located on the most center side of the captured image.

[0402] As described above, in the case where the light sources 32L and 32R emit point lights in the oblique directions, the positions of the point light images PL and PR change depending on the feeding amount of the feeding unit 23 .

[0403] Furthermore, in the case where the light sources 32L and 32R emit point lights in a direction as an oblique direction, for example, from the peripheral portion toward the central portion of the image sensor 51, in addition to the positions of the point light images PL and PR, the distance between (the positions of) the point light images PL and PR also changes depending on the feed amount of the feed unit 23. Fig.23 , the distance between the point light images PL' and PR' when the feeding unit 23 is in the minimum feeding state is the maximum value of the distance between the point light images PL and PR. In addition, the distance between the point light images PL" and PR" when the feeding unit 23 is in the maximum feeding state is the minimum value of the distance between the point light images PL and PR.

[0404] In the fourth detection method, the feeding amount of the feeding unit 23 is detected based on the position of (one or both of) the point light images PL and PR or the distance between the point light images PL and PR obtained from these positions.

[0405] Fig.24 is a flowchart for describing an example of a process of detecting a feed amount by a fourth detection method.

[0406] In steps S131 to S134, the Fig.18 The processing is similar to the processing in steps S121 to S124 in .

[0407] Then, in step S135, the point light image detection unit 62 detects the positions of the point light images PL and PR (detects the light image positions), and detects the distance between the light images, which is the distance between the positions. The point light image detection unit 62 provides the distance between the light images to the feed amount detection unit 64, and the process proceeds from step S135 to step S136.

[0408] In step S136, the feed amount detection unit 64 selects point light image information that matches the distance between the light images detected in step S135 (hereinafter also referred to as the detected distance between the light images) from the point light image information associated with the feed amount in the feed amount information stored in the feed amount information storage unit 63. In addition, in step S136, the feed amount detection unit 64 detects the feed amount associated with the point light image information that matches the distance between the detected light images as the feed amount of the feed unit 23, provides the feed amount to the image processing unit 53, and terminates the processing.

[0409] Here, when adopting the fourth detection method, the feed amount information storage unit 63 stores feed amount information, wherein each of multiple feed amounts is associated with point light image information when the feed unit 23 is fed out with the feed amount, wherein the distance between light images is used as the point light image information.

[0410] In the fourth detection method, in feed amount information in which each of a plurality of feed amounts is associated with the distance between light images which is point light image information when the feed unit 23 is fed out with the feed amount, the feed amount associated with the distance between light images that matches the distance between the detected light images is detected as the feed amount of the feed unit 23.

[0411] According to the fourth detection method, the feeding amount of the feeding unit 23 can be detected based on the distance between the detection light images without causing binary uncertainty.

[0412] Note that in the fourth detection method, the position of the point light image (detection light image position) may be adopted as the point light image information instead of the distance between the light images. In the case where the position of the point light image is adopted as the point light image information, the feed amount information storage unit 63 stores feed amount information in which each of a plurality of feed amounts is associated with the position of the point light image as the point light image information when the feed unit 23 is fed out with the feed amount. Furthermore, in this case, the point light image detection unit 62 detects the position of the point light image.

[0413] Then, the feed amount detection unit 64 detects the feed amount associated with the position of the point light image matching the position of the point light image detected in the point light image detection unit 62 as the feed amount of the feed unit 23 in the feed amount information stored in the feed amount information storage unit 63.

[0414] In addition, in the fourth detection method, similarly to the first detection method, an image as a point light image can be adopted as the point light image information, and similarly to the second and third detection methods, a dot size can be adopted as the point light image information.

[0415] Here, in the fourth detection method, when the distance between light images is used as the point light image information, it is necessary to set two or more light sources 32. However, not all of the two or more light sources 32 need to emit point light in an oblique direction, but at least one light source 32 needs to emit point light in an oblique direction.

[0416] Furthermore, in the fourth detection method, when the position of the point light image is adopted as the point light image information, one or more light sources 32 need to be provided.

[0417] <Another Configuration Example of the Multi-eye Interchangeable Lens 20>

[0418] Fig.25 is a diagram showing another configuration example of the polycular interchangeable lens 20 .

[0419] Note that in the accompanying drawings, Figure 4 , 13 The corresponding parts in the above description are given the same figure marks, and their description will be appropriately omitted hereinafter.

[0420] Fig.25 The configuration of the multi-eye interchangeable lens 20 is similar to Figure 4 and 13 The same applies to the case where light sources 32U and 32D configured similarly to the light sources 32L and 32R are newly provided.

[0421] Fig.25The polycular interchangeable lens 20 has a configuration in which two light sources 32U and 32D as a plurality of light sources are arranged on a straight line (eg, an orthogonal straight line in a plan view) which is not parallel to a straight line connecting the light sources 32L and 32R.

[0422] In the case where the polycular interchangeable lens 20 having two light sources 32U and 32D arranged on a straight line orthogonal to the straight line connecting the light sources 32L and 32R described above is mounted on the camera body 10, when there is a lens tilt about the x-axis or a lens tilt about the y-axis of the polycular interchangeable lens 20, the spot sizes and positions of the spot light images of the spot lights emitted by the light sources 32L, 32R, 32U, and 32D are as follows, relative to the case where there is no lens tilt. Fig.25 Change as shown.

[0423] Therefore, the lens tilt about the x-axis and the lens tilt about the y-axis of the polycular interchangeable lens 20 can be detected from (the spot size and position of) the spot light image.

[0424] In this case, when the lens tilt exceeding the allowable amount is detected, the user can be urged to reattach the polycular interchangeable lens 20. In addition, the lens tilt amount is detected, and parallax information can be obtained or the area of ​​the monocular image can be specified to eliminate the influence of the lens tilt amount.

[0425] <Another Configuration Example of Light Sources 32L and 32R>

[0426] Fig.26 32L and 32R is a cross-sectional view showing another configuration example of the light sources 32L and 32R.

[0427] Note that in the accompanying drawings, Fig.12 The corresponding parts to those in are given the same figure marks, and their descriptions will be appropriately omitted hereinafter.

[0428] exist Fig.26 In the embodiment, the light source 32 includes a housing 121 , an LED 122 , and lenses 123 and 124 .

[0429] therefore, Fig.26 The light source 32 is similar to the light source 32 in terms of the housing 121 to the lens 123. Fig.12 and is the same as that of Fig.12 The situation is different.

[0430] The lens 124 is provided on the image sensor 51 side of the lens 123 , and converts the point light collected by the lens 123 into parallel light and emits the parallel light.

[0431] Therefore, from Fig.26The point light emitted by the light source 32 in FIG. 3 is parallel light. The light source 32 that emits parallel light as point light is also referred to as parallel light source 32 hereinafter.

[0432] The parallel light source 32 may be provided in the feed unit 23 of the poly-eye interchangeable lens 20. In the case where the parallel light source 32 is provided in the feed unit 23, the spot size is constant regardless of the outward feed of the feed unit 23. Therefore, by reducing the spot size, the calculation error and the calculation amount when obtaining the center of gravity of the spot light image as the position of the spot light image are reduced compared to the case where non-parallel light whose spot size changes is adopted as the spot light. Therefore, the attachment error and the feed amount can be obtained with higher accuracy, and the calculation load when obtaining the attachment error and the feed amount can be reduced.

[0433] The parallel light source 32 may be provided in the feeding unit 23 so that the spot light becomes parallel to the barrel optical axis. Note that in this case, the attachment error can be obtained using the spot light (spot light image), but the feeding amount cannot be detected.

[0434] A parallel light source 32 is provided in the feeding unit 23 so as to emit point light in an oblique direction inclined from the optical axis direction of the lens barrel, thereby Figures 21 to 24 The fourth detection method is to detect the feed amount.

[0435] Furthermore, even in the case of adopting the parallel light source 32, two parallel light sources 32U and 32D as a plurality of parallel light sources may be arranged on a straight line that is not parallel to the straight line connecting the parallel light sources 32L and 32R, for example, on an orthogonal straight line in a plan view, for example, as shown in FIG. Fig.25 situation.

[0436] In this case, when the poly-eye interchangeable lens 20 is mounted on the camera body 10, the positions of the point light images of the point lights emitted by the parallel light sources 32L, 32R, 32U and 32D change according to the lens tilt of the poly-eye interchangeable lens 20 about the x-axis or the lens tilt about the y-axis.

[0437] Fig. 27 3 is a diagram showing a state in which the position of a point light image of parallel light emitted by the parallel light source 32 changes according to the inclination of the lens.

[0438] When a polyeye interchangeable lens 20 having two light sources 32U and 32D arranged on a straight line orthogonal to a straight line connecting parallel light sources 32L and 32R is mounted on a camera body 10, the position of a point light image of parallel light emitted by the parallel light source 32 changes depending on a lens tilt about the x-axis or a lens tilt about the y-axis of the polyeye interchangeable lens 20.

[0439] Therefore, the lens tilt about the x-axis and the lens tilt about the y-axis of the polycular interchangeable lens 20 can be detected from the position of the spot light image.

[0440] In this case, when the lens tilt exceeding the allowable amount is detected, the user can be urged to reattach the polycular interchangeable lens 20. In addition, the lens tilt amount is detected, and parallax information can be obtained or the area of ​​the monocular image can be specified to eliminate the influence of the lens tilt amount.

[0441] <Another Electrical Configuration Example of Camera System 1>

[0442] Fig.28 It is shown Figure 1 1 is a block diagram of another electrical configuration example of the camera system 1.

[0443] Note that in the accompanying drawings, Figure 2 The corresponding parts to those in are given the same figure marks, and their descriptions will be appropriately omitted hereinafter.

[0444] Here, the camera system 1 (or lens-integrated camera system) to which the present technology is applied holds single-eye image position information for specifying the position of a single-eye image on the entire image (captured image) captured by the image sensor 51, that is, indicating the position of the single-eye image captured from the plurality of single-eye lenses 31 in the image sensor 51. i Monocular image position information of emission position of emitted imaging light. Furthermore, the camera system 1 holds point light image position information for specifying the position of a point light image of the point light of the light source 32.

[0445] Here, the entire image refers to the entire captured image captured by the image sensor 51 , or an image obtained by deleting a part or all of the outside of all monocular images included in the captured image from the entire captured image.

[0446] Note that the monocular image position information and the spot light image position information may be information calculated for each camera system 1 , or may be information calculated for each model.

[0447] Furthermore, the monocular image position information may be information on the absolute position of the monocular image, or may be information on the absolute position of the monocular image using a predetermined monocular lens 31. i The information of the absolute position of the monocular image as the reference lens relative to the reference lens, and the information of the relative positions of other monocular images based on the position of the monocular image relative to the reference lens.

[0448] For example, the monocular image position information and the point light image position information held by the camera system 1 may be values ​​corresponding to the individual difference reflected position information (known reference position) and the individual difference point light position information (known light position), respectively, but are not limited thereto.

[0449] The camera system 1 corrects the monocular image position information by using the position of a point light image (detected point light image position information) detected from the entire image at the time of actual imaging (at the time of imaging a general captured image (unknown captured image)).

[0450] In the multi-eye interchangeable lens 20 of the camera system 1, the single-eye lens 31 i The single-eye lens 31 and the light source 32 are integrally inserted and ejected. Therefore, even if the single-eye lens 31 is inserted and ejected with the adjusted focal length (or zoom), i , can also be used with a single-lens 31 i The detected point light image position information of the given light source 32 is integrally input to accurately modify (correct) the single eye image position information.

[0451] That is, when the single-eye lens 31 i When the single-lens lens 31 is fed in and out, different position deviation amounts are generated depending on the feeding amount due to "various reasons". i The positional relationship between the monocular image and the light source 32 (i.e., the positional relationship between the monocular image and the point light image) does not change. Therefore, by detecting the position of the point light image on the entire image (the detected point light image position information) during actual imaging and grasping the deviation from the point light image position information held by the camera system 1, the monocular image position information of the monocular image can be accurately modified.

[0452] "Various reasons" include, for example, a group of single-eye lenses 31 that move integrally when entering and exiting i (Single-eye lens unit) tilt, single-eye lens unit rotation at the time of feeding, due to attachment error, the polycular interchangeable lens 20 has a rotation error at the time of attachment, a tilt error, etc. even in the case of the same feeding amount.

[0453] Since the camera system 1 is capable of accurately modifying the monocular image position information of the monocular image, the camera system 1 is capable of using the modified monocular image position information (modified monocular image position information) to accurately extract (cut out) a predetermined range centered on the optical axis center position from the entire image as the monocular image, and is capable of performing processing (generation of parallax information, etc.) for suppressing the influence of lens distortion, etc.

[0454] Note that the detected point light image position information detected from the entire image at the time of actual imaging may be, but is not limited to, for example, a value corresponding to the installation error point light position information (unknown light position). In addition, the modified monocular image position information obtained by modifying the monocular image position information using the detected point light image position information may be, but is not limited to, for example, a value corresponding to the installation error reflection position information (unknown reference position).

[0455] exist Fig.28 , the area specifying unit 52 includes a monocular image position information modifying unit 211 and a monocular image extracting unit 212.

[0456] The point light image position information and the monocular image position information stored as (a part of) the lens information of the storage unit 41 of the multi-eye interchangeable lens 20 are supplied from the communication unit 56 to the monocular image position information modification unit 211. In addition, the detected point light image position information detected from the entire image (captured image) captured by the image sensor 51 is supplied from the point light image detection unit 62 to the monocular image position information modification unit 211.

[0457] The monocular image position information modification unit 211 modifies the monocular image position information from the communication unit 56 using the point light image position information from the communication unit 56 and the detected point light image position information from the point light image detection unit 62, and provides the obtained modified monocular image position information to the monocular image extraction unit 212 and the association unit 221. In the monocular image position information modification unit 211, the monocular image position information obtained in the position calculation unit 57 ( Figure 2 ) is similar to the process of obtaining the installation error reflecting the position information, and the monocular image position information is modified.

[0458] As described above, the modified monocular image position information is supplied from the monocular image position information modification unit 211 , and the entire image (captured image) is supplied from the image sensor 51 to the monocular image extraction unit 212 .

[0459] The monocular image extraction unit 212 obtains region information indicating the region of the monocular image of the monocular lenses 310 to 314 on the entire image from the image sensor 51, using the modified monocular image position information from the monocular image position information modification unit 211. For example, the monocular image extraction unit 212 obtains information indicating a rectangular region centered on the modified monocular image position information as the region information.

[0460] Then, the monocular image extraction unit 212 extracts the area indicated by the area information from the entire image from the image sensor 51 as a monocular image, and supplies the extracted monocular image to the display unit 54 and the association unit 221 as necessary.

[0461] Furthermore, the monocular image extraction unit 212 supplies the entire image from the image sensor 51 to the display unit 54 and the association unit 221 as necessary.

[0462] The display unit 54 displays the entire image and the monocular image from the monocular image extraction unit 212 .

[0463] The associating unit 221 associates the modified monocular image position information from the monocular image position information modifying unit 211 with the monocular image or the entire image from the monocular image extracting unit 212 .

[0464] The associating unit 221 associates, for example, the monocular image extracted from the same entire image from the monocular image extracting unit 212. Furthermore, the associating unit 221 associates, for example, the monocular image extracted from the same entire image and the modified monocular image position information used to extract the monocular image from the monocular image extracting unit 212. Furthermore, the associating unit 221 associates, for example, the entire image (captured image) from the monocular image extracting unit 212 with the modified monocular image position information (imaged monocular image position) obtained by modifying the monocular image position information using the detected point light image position information detected from the entire image from the monocular image position information modifying unit 211.

[0465] For example, the association may be performed by recording the associated objects in the same recording medium or by assigning the same identification (ID) to the associated objects. Alternatively, the association may be performed using, for example, metadata of the associated objects (metadata of the monocular image or the entire image associated with the modified monocular image position information).

[0466] The associating unit 221 may collectively record or transmit information associated by association (associated information).

[0467] Note that the associating unit 221 may, for example, associate the point light image position information (included in the lens information) with the detected point light image position information detected from the entire image.

[0468] In addition, the association information obtained in the association unit 221 can be a subject for post-processing in the camera system 1 or an external device. In the post-processing, for example, the modified monocular image position information included in the association information can be used to extract a monocular image from the entire image included in the association information in which the entire image is associated with the modified monocular image position information.

[0469] In addition, the associating unit 221 may associate the entire image (captured image), the detected point light image position information (detected light image position) detected from the entire image, the point light image position information (stored light image position), and the monocular image position information (stored monocular image position). In this case, in post-processing of the associated information in which the entire image, the detected point light image position information, the point light image position information, and the monocular image position information are associated with each other, the monocular image position information is modified using the detected point light image position information and the point light image position information, and the monocular image can be extracted from the entire image using the obtained modified monocular image position information.

[0470] In addition, the association unit 221 can associate, for example, point light image position information (stored light image position), the difference between the point light image position information and the detected point light image position information (the difference between the stored light image position and the detected light image position), monocular image position information (stored monocular image position) and the entire image (captured image).

[0471] Furthermore, the associating unit 221 may adopt any association capable of specifying the position of the monocular image on the captured image.

[0472] As the target image to be associated, in addition to the entire image and the monocular image extracted from the entire image, one synthetic image in which the monocular image extracted from the entire image is arranged may be adopted.

[0473] Furthermore, as the target information to be associated with the target image, in addition to the modified monocular image position information, arbitrary information capable of specifying the position of the monocular image on the captured image may be adopted.

[0474] As the target information, for example, a set of monocular image position information, point light image position information, and detected point light image position information may be employed.

[0475] In the association of the target image and the target information, the target image and the target information may be associated with each other and stored in a storage medium, transmitted via a transmission medium, or made into one file.

[0476] Here, "association" means, for example, making one data available (linkable) when processing another data. The form of the target image and the target information as data (file) is arbitrary. For example, the target image and the target information can be collected as one data (file), or they can be set as data (files) separately. For example, the target information associated with the target image can be transmitted on a transmission path different from the transmission path of the target image. In addition, for example, the target information associated with the target image can be recorded in a recording medium different from the target image, or can be recorded in different recording areas in the same recording medium. The target image and the target information can be combined into one stream data or one file.

[0477] The target image may be a still image or a moving image. In the case of a moving image, the target information and the target image of each frame may be associated with each other.

[0478] "Association" can be performed on partial data (file) rather than the entire data of the target image. For example, in the case where the target image is a moving image including multiple frames, the target information can be associated with an arbitrary unit of the target image (e.g., multiple frames, one frame, or a portion of a frame).

[0479] Note that, in the case where the target image and the target information are separate data (files), the target image and the target information can be associated with each other by assigning the same ID (identification number) to both the target image and the target information. In addition, for example, in the case where the target image and the target information are placed together in one file, the target information can be added to the header of the file, etc.

[0480] <Post-processing equipment>

[0481] Fig.29 is a block diagram showing a configuration example of a post-processing device that performs post-processing on association information.

[0482] exist Fig.29 In the embodiment, the post-processing device 230 includes a region specifying unit 231, an image processing unit 232, a display unit 233, a recording unit 234, and a sending unit 235. The region specifying unit 231 includes a monocular image position information modifying unit 241 and a monocular image extracting unit 242.

[0483] Association information in which the entire image, the detected point light image position information, the point light image position information, and the monocular image position information are associated is supplied to the post-processing device 230 from a recording medium or a transmission medium (not shown).

[0484] Note that there are many ways to associate which types of information and how to modify this information. For example, if point light image position deviation information indicating the position deviation of the point light image is pre-calculated from the detected point light image position information and the point light image position information, and the entire image or the monocular image is associated with the point light image position deviation information, it is not necessary to associate the detected point light image position information and the point light image position information with the entire image, etc. In addition, the monocular image position information can be modified in advance, and the modified monocular image position information can be associated with the entire image or the monocular image. In this case, there is no need to modify the monocular image position information in post-processing. In addition, as the detected point light image position information, for example, in addition to the position of the point light image itself on the entire image, information of the image portion of the area where the point light image appears in the entire image can also be used. In the case of using information of the image portion of the area where the point light image appears in the entire image as the detected point light image position information, for example, the position of the point light image on the entire image can be obtained from the information.

[0485] The monocular image position information included in the association information, the detected point light image position information, and the point light image position information are supplied to the monocular image position information modification unit 241. The entire image included in the association information is supplied to the monocular image extraction unit 242.

[0486] The monocular image position information modification unit 241 modifies the monocular image position information included in the association information by using the point light image position information included in the association information and the detected point light image position information, and supplies the obtained modified monocular image position information to the monocular image extraction unit 242. The monocular image position information modification unit 241 is similar to the monocular image position information modification unit 211 ( Fig.28 ) to modify the monocular image position information.

[0487] The monocular image extraction unit 242 obtains region information indicating the region of the monocular image of the monocular lenses 310 to 314 on the entire image included in the association information, using the modified monocular image position information from the monocular image position information modification unit 241 .

[0488] Then, the monocular image extraction unit 242 extracts the area indicated by the area information from the entire image as a monocular image, and provides the extracted monocular image to the image processing unit 232, the display unit 233, the recording unit 234, and the sending unit 235 as needed.

[0489] Note that in addition to the monocular image, the monocular image extraction unit 242 may also provide the entire image and modified monocular image position information to the image processing unit 232 , the display unit 233 , the recording unit 234 , and the sending unit 235 as necessary.

[0490] The image processing unit 232 performs image processing on the monocular image from the monocular image extraction unit 242, and provides the result of the image processing as needed to the display unit 233, the recording unit 234, and the transmission unit 235. For example, the image processing unit 232 can generate disparity information using the monocular image from the monocular image extraction unit 242 and the modified monocular image position information, and can perform refocusing using the disparity information and the monocular image.

[0491] The display unit 233 displays the entire image, the monocular image, the modified monocular image position information from the monocular image extraction unit 242, and the result of the image processing by the image processing unit 232 as needed. The recording unit 234 records the entire image, the monocular image, the modified monocular image position information from the monocular image extraction unit 242, and the result of the image processing by the image processing unit 232 in a recording medium (not shown) as needed. The transmission unit 235 transmits the entire image, the monocular image, the modified monocular image position information from the monocular image extraction unit 242, and the result of the image processing by the image processing unit 232 via a transmission medium (not shown) as needed.

[0492] As post-processing, the post-processing device 230 can perform processing of modifying the monocular image position information using the detected point light image position information and the point light image position information, and processing of extracting the monocular image from the entire image using the modified monocular image position information obtained by modifying the monocular image position information.

[0493] The post-processing device 230 as described above may be provided in a device that reproduces and displays a monocular image and performs image processing on the monocular image.

[0494] Fig.30 is a block diagram showing another configuration example of a post-processing device that performs post-processing on association information.

[0495] Note that in the accompanying drawings, Fig.29 The corresponding components of the post-processing device 230 in FIG. 1 are given the same figure marks, and their description will be appropriately omitted hereinafter.

[0496] exist Fig.30 In the embodiment, the post-processing device 250 includes a region specifying unit 231 to a sending unit 235. The region specifying unit 231 includes a monocular image position information modifying unit 241.

[0497] Therefore, in terms of including the area specifying unit 231 to the sending unit 235, the post-processing device 250 is Fig.29 However, the post-processing device 250 is different from the post-processing device 230 in that the area specifying unit 231 does not include the monocular image extracting unit 242.

[0498] Association information in which multiple monocular images, detected point light image position information, point light image position information, and multiple monocular image position information respectively corresponding to the multiple monocular images are associated is provided from a recording medium or a transmission medium (not shown) to the post-processing device 250.

[0499] The monocular image position information, detected spotlight image position information, and spotlight image position information included in the association information are provided to the monocular image position information modification unit 241. The monocular image included in the association information is provided to the image processing unit 232, the display unit 233, the recording unit 234, and the transmission unit 235 as needed.

[0500] As reference Fig.29As described, the monocular image position information modification unit 241 modifies the monocular image position information included in the association information by using the point light image position information included in the association information and the detected point light image position information. The monocular image position information modification unit 241 provides the modified monocular image position information obtained by modifying the monocular image position information to the image processing unit 232, the display unit 233, the recording unit 234, and the sending unit 235 as needed.

[0501] The image processing unit 232 performs image processing on the monocular image and provides the result of the image processing to the display unit 233, the recording unit 234, and the transmission unit 235 as needed. Fig.29 Similar to the case of , the image processing unit 232 may generate disparity information using the monocular image and the modified monocular image position information, and may perform refocusing using the disparity information and the monocular image.

[0502] The display unit 233 displays the entire image, the monocular image, the modified monocular image position information from the monocular image extraction unit 242, the result of the image processing by the image processing unit 232, and the modified monocular image position information obtained by the monocular image position information modification unit 241 as needed. The recording unit 234 records the monocular image, the modified monocular image position information, the result of the image processing by the image processing unit 232, and the modified monocular image position information obtained by the monocular image position information modification unit 241 as needed. The transmission unit 235 transmits the monocular image, the modified monocular image position information, the result of the image processing by the image processing unit 232, and the modified monocular image position information obtained by the monocular image position information modification unit 241 as needed.

[0503] The post-processing device 250 may use the detected spot light image position information and the spot light image position information to perform a process of modifying the monocular image position information as post-processing. Fig.30 In the example, the entire image may be included in the associated information. The entire image may be provided to the image processing unit 232 to the sending unit 235 and may be processed.

[0504] The post-processing device 250 as described above may be provided in a device that reproduces and displays a monocular image and performs image processing for the monocular image. Note that, as described above, the post-processing device 230 and the post-processing device 250 may be provided in the camera system 1 as a post-processing function.

[0505] <Other embodiments of the camera system to which the present technology is applied>

[0506] <First Alternative Embodiment of Camera System>

[0507] Fig.31 is a block diagram showing an electrical configuration example of a first another embodiment of a camera system to which the present technology is applied.

[0508] Note that in the accompanying drawings, Figure 2 The corresponding parts to those in are given the same figure marks, and their descriptions will be appropriately omitted hereinafter.

[0509] exist Fig.31 , the camera system 300 is a lens-integrated camera system. The camera system 300 includes a lens unit 320, an image sensor 351, a RAW signal processing unit 352, an area extraction unit 353, a camera signal processing unit 354, a through-image generation unit 355, an area designation unit 356, an image reconfiguration processing unit 357, a bus 360, a display unit 361, a storage unit 362, a communication unit 364, an archiving unit 365, a control unit 381, a storage unit 382, ​​and an optical system control unit 384.

[0510] The lens unit 320 includes a feeding unit 23. Figure 1 and 2 As described, the feeding unit 23 includes the single-eye lenses 310 to 314 and the light sources 32L and 32R.

[0511] The feeding unit 23 is Fig.31 The lens barrel 21 (not shown) Figure 1 ) moves in the optical axis direction along the barrel optical axis. As the feeding unit 23 moves, the single-eye lenses 310 to 314 and the light sources 32L and 32R included in the feeding unit 23 also move integrally.

[0512] Single lens 31 i The optical paths of the light passing through the lens are configured to be independent of each other. That is, on the light receiving surface (eg, effective pixel area) of the image sensor 351, the light passing through the single-eye lens 31 is emitted at different positions from each other. i light without entering the other single-lens 31 j On the light receiving surface of the image sensor 351, the single-eye lens 31 i At least the optical axes of the light beams are located at different positions from each other, and the light that has passed through the single-eye lens 31 is emitted at different positions from each other on the light receiving surface of the image sensor 351. i at least some of the light in the image.

[0513] Therefore, in the captured image generated by the image sensor 351 (the entire image output by the image sensor 351), the image captured via the single-eye lens 31 i The images of the object formed are formed at different positions from each other. In other words, the image having the image formed in the single-eye lens 31 is obtained from the captured image.i An image (monocular image) of a viewpoint at the position of .

[0514] The image sensor 351 is a complementary metal oxide semiconductor (CMOS) image sensor, for example, similar to the image sensor 51, and images a subject to generate a captured image. The light receiving surface of the image sensor 351 is irradiated with the light beams collected by the single-eye lenses 310 to 314. The image sensor 351 receives the light beams (irradiation light) from the single-eye lenses 310 to 314 and performs photoelectric conversion, thereby generating a light beam including light emitted by the single-eye lenses 310 to 314. i A captured image of a monocular image of a viewpoint at .

[0515] The image sensor 351 may be a monochrome (so-called monochromatic) image sensor similar to the image sensor 51, or may be a color image sensor in which, for example, color filters of a Bayer array are arranged in pixel groups. That is, the captured image output by the image sensor 351 may be a monochrome image or a color image. Hereinafter, a description will be given based on the assumption that the image sensor 351 is a color image sensor and generates and outputs a captured image in a RAW format.

[0516] Note that in this embodiment, the RAW format refers to an image in a state in which the positional relationship of the arrangement of the color filters of the image sensor 351 is maintained, and may include an image obtained by performing signal processing such as image size conversion processing, noise reduction processing, and defect correction processing of the image sensor 351, and compression encoding on the image output from the image sensor 351.

[0517] The image sensor 351 can output the captured image (whole image) in RAW format, which is generated by photoelectric conversion of irradiated light. For example, the image sensor 351 can provide the captured image (whole image) in RAW format to at least one of the bus 360, the RAW signal processing unit 352, the area extraction unit 353, or the area designation unit 356.

[0518] For example, the image sensor 351 may provide a captured image (entire image) in RAW format to the storage unit 362 via the bus 360 and store the captured image in the storage medium 363. In addition, the image sensor 351 may provide a captured image (entire image) in RAW format to the communication unit 364 via the bus 360 to transmit the captured image to the outside of the camera system 300. In addition, the image sensor 351 may provide a captured image (entire image) in RAW format to the archiving unit 365 via the bus 360 to convert the captured image into a file. In addition, the image sensor 351 may provide a captured image (entire image) in RAW format to the image reconfiguration processing unit 357 via the bus 360 to perform image reconfiguration processing.

[0519] Note that the image sensor 351 may be a single-board image sensor, or may be a group of image sensors (also referred to as a multi-board image sensor) including a plurality of image sensors (eg, a three-board image sensor).

[0520] Note that in the case of a multi-plate image sensor, the image sensor is not limited to an image sensor for RGB. All image sensors may be monochrome, or may include color filters such as a Bayer array. Note that in the case where all color filters are Bayer arrays, noise reduction can be performed by setting all arrays to be the same and adjusting the positional relationship of the pixels relative to each other, and image quality can also be improved by using the effect of so-called spatial pixel shifting by changing the positional relationship of the RGB image sensors.

[0521] Even in the case of such a multi-panel imaging device, a plurality of monocular images or a plurality of viewpoint images are included in a captured image output from each image sensor (ie, from one image sensor).

[0522] The RAW signal processing unit 352 performs processing related to signal processing on the image in RAW format. For example, the RAW signal processing unit 352 can acquire a captured image (entire image) in RAW format provided from the image sensor 351. In addition, the RAW signal processing unit 352 can perform predetermined signal processing on the acquired captured image. The content of such signal processing is arbitrary. For example, the signal processing may be defect correction, noise reduction, compression (encoding), etc., or other signal processing. Of course, the RAW signal processing unit 352 can also perform various types of signal processing on the captured image. Note that the signal processing that can be performed on the image in RAW format is limited to signal processing in which the image after the signal processing is in a state in which the positional relationship of the arrangement of the color filters of the image sensor 351 is maintained as described above (in the case of a multi-board imaging device, the image maintains the state of the R image, the G image, and the B image).

[0523] The RAW signal processing unit 352 may provide the captured image (RAW') in the RAW format to which the signal processing is applied or the compressed (encoded) captured image (compressed RAW) to the storage unit 362 via the bus 360, and store the captured image in the storage medium 363. In addition, the RAW signal processing unit 352 may provide the captured image (RAW') in the RAW format to which the signal processing is applied or the compressed (encoded) captured image (compressed RAW) to the communication unit 364 via the bus 360 to transmit the captured image. In addition, the RAW signal processing unit 352 may provide the captured image (RAW') in the RAW format to which the signal processing is applied or the compressed (encoded) captured image (compressed RAW) to the archiving unit 365 via the bus 360 to convert the captured image into a file. In addition, the RAW signal processing unit 352 may provide the captured image (RAW') in the RAW format to which the signal processing is applied or the compressed (encoded) captured image (compressed RAW) to the image reconfiguration processing unit 357 via the bus 360 to perform image reconfiguration processing. Note that in the case where there is no need to distinguish and describe RAW, RAW′, and compressed RAW, they are referred to as RAW images.

[0524] The region extraction unit 353 performs processing related to extracting a partial region from a captured image in a RAW format (cutting out a partial image). For example, the region extraction unit 353 can obtain a captured image in a RAW format (the entire image) from the image sensor 351. In addition, the region extraction unit 353 can acquire information (also referred to as extraction region information) indicating a region extracted from the captured image provided from the region designation unit 356. Then, the region extraction unit 353 can extract a partial region from the captured image (cut out a partial image) based on the extraction region information.

[0525] For example, the region extraction unit 353 can cut out the region having the area between each of the single-lens lenses 31 from the captured image (the entire image). i . In addition, in the captured image, the area from which the monocular image is cut out (the area corresponding to the monocular image) is also referred to as the monocular image area. For example, the area extraction unit 353 can obtain viewpoint related information, which is provided from the area designation unit 356 and is used to designate the monocular image area as the information for extracting the area information, and can extract each monocular image area indicated in the viewpoint related information from the captured image, that is, cut out each monocular image. Then, the area extraction unit 353 can provide the cut-out monocular image (RAW format) to the camera signal processing unit 354.

[0526] The viewpoint-related information may be, for example, a value corresponding to the position information reflecting individual differences or the position information reflecting installation errors, but is not limited thereto, and does not need to be set in order to distinguish or correct such individual differences, installation errors, etc. In consideration of various errors including the above-mentioned errors, even the information indicating only the relationship between the area of ​​the monocular image on the captured image and the position of the point light may be modified.

[0527] The area extraction unit 353 can combine the monocular images cut out from the captured image (whole image) to generate a composite image. The composite image is obtained by combining the monocular images into one data or one image. For example, the area extraction unit 353 can generate an image in which the monocular images are arranged in a plane as a composite image. The area extraction unit 353 can provide the generated composite image (RAW format) to the camera signal processing unit 354.

[0528] Furthermore, for example, the area extraction unit 353 may supply the entire image to the camera signal processing unit 354. For example, the area extraction unit 353 may extract a partial area including all monocular images from the acquired captured image (i.e., cut out a partial image including all monocular images), and supply the cut-out partial image (i.e., an image obtained by deleting a part or all of an area other than all monocular images included in the captured image) as the entire image to the camera signal processing unit 354 in a RAW format. In this case, the position (range) of the area to be extracted may be predetermined in the area extraction unit 353, or may be specified by viewpoint-related information supplied from the area specifying unit 356.

[0529] Furthermore, the area extraction unit 353 may also supply the acquired captured image (ie, not the cut-out partial image including all the monocular images but the entire captured image) as the entire image to the camera signal processing unit 354 in the RAW format.

[0530] Note that the area extraction unit 353 can provide the partial image (whole image, monocular image or composite image) in RAW format cut out from the captured image as described above to the storage unit 362, the communication unit 364, the archiving unit 365, the image reconfiguration processing unit 357, etc. via the bus 360, similar to the case of the image sensor 351.

[0531] In addition, the area extraction unit 353 may provide a partial image in RAW format (whole image, single-eye image, or composite image) to the RAW signal processing unit 352 to perform predetermined signal processing or compression (encoding). Even in this case, the RAW signal processing unit 352 may provide a captured image in RAW format (RAW′) or a compressed (encoded) captured image (compressed RAW) to which signal processing is applied to the storage unit 362, the communication unit 364, the archiving unit 365, the image reconfiguration processing unit 357, and the like via the bus 360.

[0532] That is, at least one of the entire image, the monocular image, or the synthesized image may be a RAW image.

[0533] The camera signal processing unit 354 performs processing related to camera signal processing on the image. For example, the camera signal processing unit 354 can acquire the image (whole image, monocular image, or composite image) provided from the area extraction unit 353. In addition, the camera signal processing unit 354 can apply camera signal processing (camera processing) to the acquired image. For example, the camera signal processing unit 354 can perform color separation processing of separating each color of RGB to generate an R image, a G image, and a B image, each image having the same number of pixels as the image to be processed (in the case of using a mosaic color filter such as a Bayer array, a demosaicing process), a YC conversion process of converting the color space of the image after color separation from RGB into YC (brightness / color difference), and the like. In addition, the camera signal processing unit 354 can perform processing such as defect correction, noise reduction, automatic white balance (AWB), or gamma correction on the image to be processed. In addition, the camera signal processing unit 354 can also compress (encode) the image to be processed. Of course, the camera signal processing unit 354 can perform a plurality of camera signal processes on the image to be processed, and can also perform camera signal processes other than the above examples.

[0534] Note that in the following description, it is assumed that the camera signal processing unit 354 acquires an image in RAW format, performs color separation processing or YC conversion on the image, and outputs an image in YC format (YC). The image may be an entire image, a monocular image, or a composite image. In addition, the image in YC format (YC) may be encoded or not. That is, the data output from the camera signal processing unit 354 may be encoded data or unencoded image data.

[0535] That is, at least one of the entire image, the monocular image, or the synthesized image may be an image in a YC format (also referred to as a YC image).

[0536] In addition, the image output by the camera signal processing unit 354 may be an image to which the full development process has not been applied, and may be an image to which some or all of the processes related to irreversible image quality adjustment (color adjustment) (such as gamma correction and color matrix) have not been applied, as an image in YC format (YC). In this case, the image in YC format (YC) can be substantially returned to an image in RAW format without being degraded in subsequent stages, reproduction, etc.

[0537] The camera signal processing unit 354 may provide the image (YC) in the YC format to which the camera signal processing has been applied to the storage unit 362 via the bus 360, and store the image (YC) in the storage medium 363. In addition, the camera signal processing unit 354 may provide the image (YC) in the YC format to which the camera signal processing has been applied to the communication unit 364 via the bus 360 to transmit the image to the outside. In addition, the camera signal processing unit 354 may provide the image (YC) in the YC format to which the camera signal processing has been applied to the archiving unit 365 via the bus 360 to convert the image into a file. In addition, the camera signal processing unit 354 may provide the image (YC) in the YC format to which the camera signal processing has been applied to the image reconfiguration processing unit 357 via the bus 360 to perform image reconfiguration processing.

[0538] Furthermore, for example, the camera signal processing unit 354 may supply the image (YC) in the YC format to the through image generating unit 355 .

[0539] Note that, in the case where an image in RAW format (whole image, single-eye image, or composite image) is stored in the storage medium 363, the camera signal processing unit 354 can read the image in RAW format from the storage medium 363 and perform signal processing. Even in this case, the camera signal processing unit 354 can provide the image in YC format (YC) to which the camera signal processing has been applied to the display unit 361, the storage unit 362, the communication unit 364, the archive unit 365, the image reconfiguration processing unit 357, and the like via the bus 360.

[0540] Furthermore, the camera signal processing unit 354 may perform camera signal processing on a captured image (entire image) in a RAW format output from the image sensor 351 , and the region extraction unit 353 may extract a partial region from the captured image (entire image) after the camera signal processing.

[0541] The through image generation unit 355 performs processing related to the generation of the through image. The through image is an image displayed to the user for checking the image to be shot during shooting or during shooting preparation (non-recording). The through image is also called a live view image or an electronic-to-electronic (EE) image. Note that the through image is an image before shooting when shooting a still image. However, when shooting a moving image, a through image corresponding not only to the image during preparation for shooting but also to the image during shooting (recording) is displayed.

[0542] For example, the through-image generation unit 355 may acquire an image (whole image, monocular image, or composite image) provided from the camera signal processing unit 354. In addition, for example, the through-image generation unit 355 may generate a through-image, which is an image for display, by performing image size (resolution) conversion of the acquired image into an image size corresponding to the resolution of the display unit 361. The through-image generation unit 355 may provide the generated through-image to the display unit 361 via the bus 360 to display the through-image.

[0543] The region specifying unit 356 performs processing related to the specification (setting) of the region to be extracted from the captured image by the region extracting unit 353. For example, the region specifying unit 356 acquires viewpoint-related information VI for specifying the region to be extracted from the captured image, and supplies the viewpoint-related information VI to the region extracting unit 353 as extraction region information.

[0544] The viewpoint-related information VI may be, for example, a value corresponding to the individual difference-reflecting position information, but is not limited thereto. The viewpoint-related information VI indicates, for example, the designed position of a single-eye image in a captured image, the position when imaging a known captured image, and the like.

[0545] The viewpoint-related information VI includes, for example, monocular area information indicating a monocular image area in a captured image. The monocular area information may represent a monocular image area in any manner. For example, the monocular image area may be represented by coordinates representing a position corresponding to the optical axis of the monocular lens 31 in the captured image (optical axis center position) (also referred to as the center coordinates of the monocular image area) and a resolution (number of pixels) of the monocular image (monocular image area). That is, the monocular area information may include the center coordinates of the monocular image area in the captured image and the resolution of the monocular image area. In this case, the position of the monocular image area in the entire image may be specified based on the center coordinates of the monocular image area and the resolution (number of pixels) of the monocular image area.

[0546] Note that the monocular image area information is set for each monocular image area. That is, in the case where the captured image includes multiple monocular images, the viewpoint related information VI may include viewpoint identification information (e.g., ID) and monocular area information for identifying the monocular image (area) for each monocular image (each monocular image area).

[0547] In addition, the viewpoint related information VI may include any other information. For example, the viewpoint related information VI may include viewpoint time information indicating the shooting time of the shot image from which the monocular image is extracted.

[0548] The area designation unit 356 may provide the viewpoint related information VI to the bus 360. For example, the area designation unit 356 may provide the viewpoint related information VI to the storage unit 362 via the bus 360, and store the viewpoint related information VI in the storage medium 363. In addition, the area designation unit 356 may provide the viewpoint related information VI to the communication unit 364 via the bus 360 to transmit the viewpoint related information VI. In addition, the area designation unit 356 may provide the viewpoint related information VI to the archiving unit 365 via the bus 360 to convert the viewpoint related information VI into a file. In addition, the area designation unit 356 may provide the viewpoint related information VI to the image reconfiguration processing unit 357 via the bus 360 for image reconfiguration processing.

[0549] For example, the region specifying unit 356 may acquire the viewpoint related information VI from the control unit 381, and provide the viewpoint related information VI to the region extracting unit 353 and the bus 360. In this case, the control unit 381 reads the viewpoint related information VI stored in the storage medium 383 via the storage unit 382, ​​and provides the viewpoint related information VI to the region specifying unit 356. The region specifying unit 356 provides the viewpoint related information VI to the region extracting unit 353 and the bus 360.

[0550] The viewpoint-related information VI provided to the storage unit 362, the communication unit 364, or the archiving unit 365 via the bus 360 in this way is associated with the image (whole image, monocular image, or synthetic image) therein. For example, the storage unit 362 may store the provided viewpoint-related information VI and the image (whole image, monocular image, or synthetic image) in the storage medium 363 in association with each other. In addition, the communication unit 364 may send the provided viewpoint-related information VI to the outside in association with the image (whole image, monocular image, or synthetic image). In addition, the archiving unit 365 may generate a file including the provided viewpoint-related information VI and the image (whole image, monocular image, or synthetic image) in association with each other.

[0551] In addition, the area designation unit 356 may acquire a captured image in a RAW format provided from the image sensor 351, generate viewpoint related information VI′ based on the captured image, and provide the generated viewpoint related information VI′ to the area extraction unit 353 and the bus 360. In this case, the area designation unit 356 specifies each monocular image area from the captured image, and generates viewpoint related information VI′ indicating the monocular image area (for example, indicating the monocular image area by the center coordinates of the monocular image area in the captured image and the resolution of the monocular image area). Then, the area designation unit 356 provides the generated viewpoint related information VI′ to the area extraction unit 353 and the bus 360. Note that the point light information SI′ generated by the area designation unit 356 based on the captured image may be provided together with the viewpoint related information VI′.

[0552] The spot light information is information about a spot light image, and may be, but is not limited to, for example, a value corresponding to the above-mentioned individual difference spot light position information or installation error spot light position information.

[0553] The area designation unit 356 may obtain the viewpoint related information VI from the control unit 381, obtain the captured image in the RAW format provided from the image sensor 351, generate the point light information SI' based on the captured image, add the point light information SI' to the viewpoint related information VI, and provide the viewpoint related information VI to the area extraction unit 353 and the bus 360. In this case, the control unit 381 reads the viewpoint related information VI stored in the storage medium 383 via the storage unit 382, ​​and provides the viewpoint related information VI to the area designation unit 356. The area designation unit 356 corrects the viewpoint related information VI using the point light information SI', and generates the corrected viewpoint related information VI'. The area designation unit 356 provides the viewpoint related information VI' to the area extraction unit 353 and the bus 360.

[0554] In addition, the area designation unit 356 may obtain the viewpoint related information VI from the control unit 381, obtain the captured image in the RAW format provided from the image sensor 351, generate the point light information SI′ based on the captured image, correct the viewpoint related information VI using the point light information SI′, and provide the corrected viewpoint related information VI′ to the area extraction unit 353 and the bus 360. In this case, the control unit 381 reads the viewpoint related information VI stored in the storage medium 383 via the storage unit 382, ​​and provides the viewpoint related information VI to the area designation unit 356. The area designation unit 356 corrects the viewpoint related information VI using the point light information SI′ to generate the viewpoint related information VI′. The area designation unit 356 provides the viewpoint related information VI′ to the area extraction unit 353 and the bus 360.

[0555] For example, the spotlight information SI' may be, but is not limited to, a value corresponding to the above-mentioned installation error spotlight position information or spotlight image information. The spotlight information SI' indicates, for example, the position and / or spot size of a spotlight image appearing in a captured image.

[0556] Here, due to various offsets associated with the movement of the feeding unit 23 (for example, an offset between a direction perpendicular to the light receiving surface of the image sensor 351 and the moving direction of the feeding unit 23, and a shift of the single-eye lens 31 associated with the movement of the feeding unit 23), i The position of the monocular image in the captured image may be shifted as the feeding unit 23 moves.

[0557] In a case where the position of the monocular image in the captured image shifts with the movement of the feeding unit 23, when the image is cut out (extracted) from the position of the captured image indicated by the viewpoint-related information VI, the image of the area shifted from the monocular image area of ​​the (original) monocular image which is the monocular image.

[0558] Therefore, the area specifying unit 356 can detect the positional deviation (amount) of the monocular image in the captured image using the position and / or the spot size of the spot light image indicated by the spot light information SI′ generated from the captured image.

[0559] Then, the region specifying unit 356 may obtain information for modifying the position at which the monocular image is cut out from the captured image based on the positional deviation of the monocular image, and provide the information to the region extracting unit 353 .

[0560] That is, the area specifying unit 356 corrects the viewpoint-related information VI according to the positional deviation of the monocular image so as to represent the position of the monocular image after the positional deviation, and provides the viewpoint-related information VI′ obtained by the correction to the area extracting unit 353 .

[0561] Here, the storage medium 383 stores, for example, viewpoint-related information VI and point light information SI. The point light information SI may be, for example, but is not limited to, a value corresponding to the individual difference point light position information described above. The point light information SI indicates, for example, the designed position and / or point size of the point light image in the captured image, the position and / or point size during imaging of a known captured image, and the like.

[0562] The viewpoint-related information VI and the point light information SI are information obtained at the same timing. For example, in the case where the viewpoint-related information VI is information indicating the position (viewpoint) of the designed monocular image, etc., the point light information SI is also information indicating the position of the designed point light image, etc. In addition, for example, in the case where the viewpoint-related information VI is information indicating the position of the monocular image detected when imaging a known captured image, etc., the point light information SI is also information indicating the position of the point light image detected when imaging a known captured image, etc.

[0563] For example, the area designation unit 356 may detect the difference between the point light information SI generated from the captured image and the point light information SI' as the positional deviation of the monocular image in the captured image. Then, the area designation unit 356 may use the positional deviation of the monocular image in the captured image, that is, the difference between the point light information SI generated from the captured image and the point light information SI', to correct the viewpoint-related information VI, and generate the viewpoint-related information VI' in which the positional deviation of the monocular image in the captured image has been corrected (modified).

[0564] In addition, the area specifying unit 356 can detect the feeding amount of the feeding unit 23 using the spot light information SI′ generated from the captured image.

[0565] The image reconfiguration processing unit 357 performs processing related to image reconfiguration. For example, the image reconfiguration processing unit 357 can obtain an image (whole image, monocular image, or composite image) in YC format from the camera signal processing unit 354 or the storage unit 362 via the bus 360. In addition, the image reconfiguration processing unit 357 can obtain viewpoint related information from the area specifying unit 356 or the storage unit 362 via the bus 360.

[0566] In addition, the image reconfiguration processing unit 357 can perform image processing, such as generation of depth information and refocusing for generating (reconfiguring) an image focused on an arbitrary object, using, for example, the acquired image and viewpoint-related information associated with the acquired image. For example, in the case where a monocular image is to be processed, the image reconfiguration processing unit 357 performs processing such as generation of depth information and refocusing using each monocular image. In addition, in the case where a captured image or a synthesized image is to be processed, the image reconfiguration processing unit 357 extracts each monocular image from the captured image or the synthesized image, and performs processing such as generation of depth information and refocusing using the extracted monocular image.

[0567] The image reconfiguration processing unit 357 may provide the generated depth information and the refocused image as a processing result to the storage unit 362 via the bus 360, and store the processing result in the storage medium 363. In addition, the image reconfiguration processing unit 357 may provide the generated depth information and the refocused image as a processing result to the communication unit 364 via the bus 360 to transmit the processing result to the outside. In addition, the image reconfiguration processing unit 357 may provide the generated depth information and the refocused image as a processing result to the archiving unit 365 via the bus 360 to convert the processing result into a file.

[0568] The image sensor 351, the RAW signal processing unit 352, the area extraction unit 353, the camera signal processing unit 354, the through image generation unit 355, the area designation unit 356, the image reconfiguration processing unit 357, the display unit 361, the storage unit 362, the communication unit 364, and the archiving unit 365 are connected to the bus 360. The bus 360 serves as a transmission medium (transmission path) for various data exchanged between these blocks. Note that the bus 360 can be implemented by wire or wirelessly.

[0569] The display unit 361 includes, for example, a liquid crystal panel, an organic electroluminescent (EL) panel, or the like, and is provided integrally with or separately from the housing of the camera system 300. For example, the display unit 361 may be provided on the back side of the housing of the camera system 300 (the surface opposite to the surface on which the lens unit 320 is provided).

[0570] The display unit 361 performs processing related to image display. For example, the display unit 361 can acquire the through image in the YC format provided from the through image generation unit 355, convert the through image into the RGB format, and display the image in the RGB format. In addition, for example, the display unit 361 can also display information such as menus and settings of the camera system 300.

[0571] In addition, the display unit 361 can acquire and display an image in the YC format (the entire image, a monocular image, or a synthesized image) provided from the storage unit 362. In addition, the display unit 361 can acquire and display a thumbnail in the YC format provided from the storage unit 362. In addition, the display unit 361 can acquire and display an image in the YC format (the entire image, a monocular image, or a synthesized image) provided from the camera signal processing unit 354.

[0572] The storage unit 362 controls storage of a storage medium 363 including, for example, a semiconductor memory, etc. The storage medium 363 may be a removable storage medium or a storage medium built in the camera system 300. For example, the storage unit 362 may store an image (entire image, a monocular image, or a synthesized image) provided via the bus 360 in the storage medium 363 according to an operation of the control unit 381 or a user, etc.

[0573] For example, the storage unit 362 may acquire an image in RAW format (whole image, single eye image, or synthesized image) supplied from the image sensor 351 or the area extraction unit 353, and store the image in the storage medium 363. Furthermore, the storage unit 362 may acquire an image in RAW format (whole image, single eye image, or synthesized image) to which signal processing is applied or a compressed (encoded) image in RAW format (whole image, single eye image, or synthesized image) supplied from the RAW signal processing unit 352, and store the image in the storage medium 363. Furthermore, the storage unit 362 may acquire an image in YC format (whole image, single eye image, or synthesized image) supplied from the camera signal processing unit 354, and store the image in the storage medium 363.

[0574] At this time, the storage unit 362 may acquire the viewpoint related information provided from the area specifying unit 356, and associate the viewpoint related information with the above-mentioned image (whole image, monocular image, or synthesized image). That is, the storage unit 362 may store the image (whole image, monocular image, or synthesized image) and the viewpoint related information in association with each other in the storage medium 363. That is, the storage unit 362 functions as an associating unit that associates at least one of the whole image, monocular image, or synthesized image with the viewpoint related information.

[0575] In addition, for example, the storage unit 362 may acquire the depth information and the refocused image provided from the image reconfiguration processing unit 357, and store the information in the storage medium 363. In addition, the storage unit 362 may acquire a file provided from the archiving unit 365 and store the file in the storage medium 363. The file includes, for example, an image (entire image, a monocular image, or a synthesized image) and viewpoint related information. That is, in the file, the image (entire image, a monocular image, or a synthesized image) and viewpoint related information are associated with each other.

[0576] In addition, for example, the storage unit 362 can read data, files, etc. stored in the storage medium 363 according to the control unit 381, user operations, etc., and provide the read data, files, etc. to the camera signal processing unit 354, the display unit 361, the communication unit 364, the archiving unit 365, the image reconfiguration processing unit 357, etc. via the bus 360. For example, the storage unit 362 is capable of reading an image (whole image, single eye image, or composite image) in the YC format from the storage medium 363, providing the image to the display unit 361, and displaying the image. In addition, the storage unit 362 can read an image (whole image, single eye image, or composite image) in the RAW format from the storage medium 363, providing the image to the camera signal processing unit 354, and applying camera signal processing to the image.

[0577] In addition, the storage unit 362 can read the data or file of the image (whole image, monocular image or synthetic image) and the viewpoint related information stored in the storage medium 363 in association with each other, and provide the data or file to other processing units. For example, the storage unit 362 can read the image (whole image, monocular image or synthetic image) and the viewpoint related information associated with each other from the storage medium 363, and provide the image and the viewpoint related information to the image reconfiguration processing unit 357 to perform processing such as generation of depth information and refocusing. In addition, the storage unit 362 can read the image (whole image, monocular image or synthetic image) and the viewpoint related information associated with each other from the storage medium 363, and provide the image and the viewpoint related information to the communication unit 364 to send the image and the viewpoint related information. In addition, the storage unit 362 can read the image (whole image, monocular image or synthetic image) and the viewpoint related information associated with each other from the storage medium 363, and provide the image and the viewpoint related information to the archiving unit 365 to convert the image and the viewpoint related information into a file.

[0578] Note that the storage medium 363 may be a read-only memory (ROM) or a rewritable memory such as a random access memory (RAM) or a flash memory. In the case of a rewritable memory, the storage medium 363 may store any information.

[0579] The communication unit 364 communicates with a server on the Internet, a PC on a wired or wireless LAN, other external devices, etc. through an arbitrary communication method. For example, the communication unit 364 can send data or files of an image (entire image, a monocular image, or a synthesized image) and viewpoint-related information to a communication partner (external device) through communication by a streaming method, an uploading method, etc., according to the control unit 381, the user's operation, etc.

[0580] For example, the communication unit 364 may acquire and transmit an image in a RAW format (the entire image, a monocular image, or a synthesized image) supplied from the image sensor 351 or the area extraction unit 353. Furthermore, the communication unit 364 may acquire and transmit an image in a RAW format (the entire image, a monocular image, or a synthesized image) to which signal processing is applied or a compressed (encoded) image (the entire image, a monocular image, or a synthesized image) supplied from the RAW signal processing unit 352. Furthermore, the communication unit 364 may acquire and transmit an image in a YC format (the entire image, a monocular image, or a synthesized image) supplied from the camera signal processing unit 354.

[0581] At this time, the communication unit 364 can obtain the viewpoint related information provided from the area designation unit 356, and associate the viewpoint related information with the above-mentioned image (whole image, monocular image, or synthetic image). That is, the communication unit 364 can send the image (whole image, monocular image, or synthetic image) and the viewpoint related information in association with each other. For example, in the case of sending an image by a streaming method, the communication unit 364 repeats the process of obtaining the image to be sent (whole image, monocular image, or synthetic image) from the processing unit that provides the image and sending the viewpoint related information and the image provided from the area designation unit 356 in association with each other. That is, the communication unit 364 serves as an association unit that associates at least one of the whole image, monocular image, or synthetic image with the viewpoint related information.

[0582] In addition, for example, the communication unit 364 can acquire and transmit the depth information and the refocused image provided from the image reconfiguration processing unit 357. In addition, the communication unit 364 can acquire and transmit the file provided from the archiving unit 365. The file includes, for example, an image (entire image, a monocular image, or a synthesized image) and viewpoint related information. That is, in the file, the image (entire image, a monocular image, or a synthesized image) and viewpoint related information are associated with each other.

[0583] The archiving unit 365 performs processing related to file generation. For example, the archiving unit 365 can obtain an image in RAW format (whole image, monocular image, or synthesized image) provided from the image sensor 351 or the area extraction unit 353. In addition, the archiving unit 365 can obtain an image in RAW format (whole image, monocular image, or synthesized image) to which signal processing has been applied, or a compressed (encoded) image in RAW format (whole image, monocular image, or synthesized image) provided from the RAW signal processing unit 352. In addition, the archiving unit 365 can obtain an image in YC format (whole image, monocular image, or synthesized image) provided from the camera signal processing unit 354. In addition, for example, the archiving unit 365 can obtain viewpoint related information provided from the area designation unit 356.

[0584] The archiving unit 365 can associate multiple acquired data with each other by converting the multiple data into files and generating a file including the multiple data. For example, the archiving unit 365 can associate the above-mentioned image (whole image, monocular image, or composite image) and viewpoint related information with each other by generating a file from the image and viewpoint related information. That is, the archiving unit 365 serves as an associating unit that associates at least one of the whole image, monocular image, or composite image with viewpoint related information.

[0585] In addition, for example, the archiving unit 365 can obtain the depth information and the refocused image provided from the image reconfiguration processing unit 357 and convert the information into a file. In addition, the archiving unit 365 can generate a file based on the interrelated images (whole image, monocular image, or synthesized image) and viewpoint related information provided from the storage unit 362.

[0586] Note that the archiving unit 365 can generate a thumbnail image of the image to be archived (e.g., a monocular image) and include the thumbnail image in the generated file. That is, the archiving unit 365 can associate the thumbnail image with the image (whole image, monocular image, or composite image) and viewpoint related information by generating a file.

[0587] The archiving unit 365 may provide the generated file (the associated images and the viewpoint related information) to the storage unit 362 via the bus 360, for example, and store the file in the storage medium 363. In addition, for example, the archiving unit 365 may provide the generated file (the associated images and the viewpoint related information) to the communication unit 364 via the bus 360 to transmit the file.

[0588] The storage unit 362, the communication unit 364, and the archiving unit 365 are also referred to as the association unit 70. The association unit 70 associates an image (whole image, monocular image, or synthetic image) with viewpoint related information. For example, the storage unit 362 may cause the storage medium 363 to store at least one of the whole image, monocular image, or synthetic image in association with viewpoint related information. In addition, the communication unit 364 may send at least one of the whole image, monocular image, or synthetic image in association with viewpoint related information. In addition, the archiving unit 365 may associate the whole image, monocular image, or synthetic image with viewpoint related information by generating a file from at least one of the whole image, monocular image, or synthetic image and viewpoint related information.

[0589] The associating unit 70 may associate point light information and viewpoint-related information in addition to an image (entire image, a monocular image, or a synthesized image).

[0590] The control unit 381 performs control processing on the camera system 300. That is, the control unit 381 can cause each unit of the camera system 300 to perform processing. For example, the control unit 381 can cause the lens unit 320 (single-eye lens 310) to perform processing via the optical system control unit 384. i Each of the optical systems (in the image sensor 351) sets the optical system for imaging, such as the aperture and the focus position. In addition, the control unit 381 can control the image sensor 351 so that the image sensor 351 performs imaging (photoelectric conversion) and generates a captured image.

[0591] In addition, the control unit 381 can provide the viewpoint related information VI and the point light information SI to the area specifying unit 356 to specify the area to be extracted from the captured image. The control unit 381 can read the viewpoint related information VI and the point light information SI stored in the storage medium 383 via the storage unit 382, ​​and provide the read information to the area specifying unit 356.

[0592] In addition, the control unit 381 may acquire an image via the bus 360, and control the aperture based on the brightness of the image via the optical system control unit 384. In addition, the control unit 381 may control the focus based on the sharpness of the image via the optical system control unit 384. In addition, the control unit 381 may control the camera signal processing unit 354 based on the RGB ratio of the image to control the white balance gain.

[0593] The storage unit 382 controls the storage of a storage medium 383 including, for example, a semiconductor memory or the like. The storage medium 383 may be a removable storage medium or a built-in memory. The storage medium 383 stores, for example, viewpoint related information VI. The viewpoint related information VI is information corresponding to the lens unit 320 (each monocular lens 31) and the image sensor 351. That is, the viewpoint related information VI is information about a monocular image having a viewpoint at the position of each monocular lens 31 of the lens unit 320, and is information for specifying a monocular image area. The storage medium 383 may further store point light information SI.

[0594] For example, the storage unit 382 may read the viewpoint-related information VI and the point light information SI stored in the storage medium 383 according to the operation of the control unit 381 or the user, and provide the read information to the control unit 381 .

[0595] Note that the storage medium 383 may be a ROM or a rewritable memory such as a RAM or a flash memory. In the case of a rewritable memory, the storage medium 383 may store arbitrary information.

[0596] Furthermore, the storage unit 382 and the storage medium 383 may be replaced by the storage unit 362 and the storage medium 363. That is, the information to be stored in the storage medium 383 (viewpoint related information VI, etc.) may be stored in the storage medium 363. In this case, the storage unit 382 and the storage medium 383 may be omitted.

[0597] The optical system control unit 384 controls the lens unit 320 (the feed unit 23, the single-eye lens 31) under the control of the control unit 381. i For example, the optical system control unit 384 may control each single-eye lens 31 i and aperture to control each monocular lens 31 i Note that in the case where the camera system 300 has an electric focus adjustment function, the optical system control unit 384 may control each of the single-eye lenses 31 of the lens unit 320 to adjust the focus of the camera unit 300. i ) of the focus (focal length). In addition, the optical system control unit 384 can control each single-eye lens 31 i The aperture (F value).

[0598] Note that the camera system 300 may include a mechanism (physical configuration) for controlling the focal length by manually operating a focus ring provided in the lens barrel, rather than having such an electric focus adjustment function. In this case, the optical system control unit 384 may be omitted.

[0599] <Relationship of viewpoint-related information, etc.>

[0600] As described above, in the camera system 300, the lens unit 320 (a plurality of single-eye lenses 31 i ) captures the subject in the image sensor 351, and generates a signal including the signal received by the single-lens lens 31 i The camera system 300 extracts some or all of the monocular images from the captured images, thereby generating a monocular image having a monocular lens 31. i Since the multiple monocular images extracted from one captured image are images of viewpoints different from each other, it is possible to use these monocular images, for example, to perform processing such as depth estimation by multi-eye matching and correction for suppressing attachment errors of multi-eye lenses. However, in order to perform the processing, information such as the relative positions between the monocular images is necessary.

[0601] Therefore, the camera system 300 combines viewpoint-related information with the image through a plurality of single-eye lenses 31 having optical paths independent of each other. i A captured image generated by imaging a subject using an image sensor 351 as an imaging element, and a plurality of single-lens lenses 31 extracted from the captured image.i The viewpoint-related information is associated with a plurality of monocular images of a viewpoint at a position of a captured image, or a synthesized image obtained by combining a plurality of monocular images, the viewpoint-related information being information for specifying a region of the plurality of monocular images in the captured image.

[0602] For example, the associating unit 70 obtains viewpoint related information corresponding to an image (whole image, monocular image, or synthetic image) from the area specifying unit 356, and associates the image with the viewpoint related information. For example, the storage unit 362 causes the storage medium 363 to store at least one of the whole image, monocular image, or synthetic image in association with the viewpoint related information. In addition, the communication unit 364 sends at least one of the whole image, monocular image, or synthetic image in association with the viewpoint related information. In addition, the archiving unit 365 associates the whole image, monocular image, or synthetic image with the viewpoint related information by generating a file from at least one of the whole image, monocular image, or synthetic image and the viewpoint related information.

[0603] Through the above association, not only the camera system 300 but also a system other than the camera system 300 can perform high-precision image processing on a monocular image or the like using viewpoint-related information.

[0604] In the association, the monocular image and the like may be associated with the viewpoint related information VI. Furthermore, in the association, the monocular image and the like may be associated with the corrected viewpoint related information VI′ obtained by correcting the viewpoint related information VI. Furthermore, in the association, the monocular image and the like may be associated with the viewpoint related information VI, the point light information SI, and the point light information SI′. Furthermore, in the association, the monocular image and the like may be associated with the viewpoint related information VI and the difference between the point light information SI and the point light information SI′.

[0605] In the case of associating a monocular image, etc. with viewpoint-related information VI′, in the case of associating a monocular image, etc. with viewpoint-related information VI, point light information SI and point light information SI′, and in the case of associating a monocular image, etc. with viewpoint-related information VI and the difference between point light information SI and point light information SI′, even if a position deviation of the monocular image in the captured image occurs with the movement of the feeding unit 23, the position of each monocular image in the captured image (the position of the viewpoint) can be accurately identified.

[0606] <Handling of Deviation of Attachment Position of Lens Unit 320>

[0607] In the lens-integrated camera system 300, the attachment position of the lens unit 320 may deviate due to a manufacturing error. In addition, the attachment position of the lens unit 320 may shift with the movement of the feed unit 23. When the attachment position of the lens unit 320 shifts and an attachment error occurs at the attachment position, the accuracy of the process of cutting out a monocular image from a captured image and calculating parallax information using the monocular image is reduced.

[0608] Therefore, the area specifying unit 356 can detect an attachment error which is a deviation (amount) of the attachment position of the lens unit 320 using a point light image appearing in a captured image.

[0609] For example, the area designation unit 356 is capable of detecting the incident range of the point light emitted from the light sources 32L and 32R to the image sensor 351 from the captured image, that is, the point light image appearing in the captured image, and is capable of generating (detecting) point light information SI′ related to the point light image.

[0610] In addition, the area designation unit 356 can detect the difference between the point light information SI' and the point light information SI, for example, the difference between the position of the point light image represented by the point light information SI' and the position of the point light image represented by the point light information SI, as an attachment error.

[0611] Then, the area designation unit 356 can correct the viewpoint related information VI using the attachment error, and generate the viewpoint related information VI'. For example, the area designation unit 356 can correct the position of the monocular image represented by the viewpoint related information VI by the difference between the position of the point light image represented by the point light information SI' and the position of the point light image represented by the point light information SI', based on the point light information SI and the point light information SI', and generate the viewpoint related information VI' as information for specifying the position of the monocular image deviated with the attachment error.

[0612] The region extraction unit 353 can accurately cut out a monocular image by cutting out a monocular image from a captured image using the viewpoint related information VI′. In addition, the region extraction unit 353 uses the viewpoint related information VI′ to accurately specify the position of the viewpoint of the monocular image, and can accurately obtain disparity information using the monocular image and the position of the viewpoint.

[0613] Here, the area specifying unit 356 that generates the point light information SI′ from the captured image can be said to be a detection unit that detects the incident range of the point light emitted from the light sources 32L and 32R to the image sensor 351.

[0614] In addition, the area designation unit 356 that generates viewpoint related information VI' by correcting the viewpoint related information VI using the difference between the point light information SI' and the point light information SI as an attachment error can also be said to be a processing unit that performs processing to correct the viewpoint related information VI based on the detection result of the detection unit (i.e., the optical image on the captured image).

[0615] <Second other embodiment of camera system>

[0616] Fig.31 is a block diagram showing an electrical configuration example of a second other embodiment of a camera system to which the present technology is applied.

[0617] Note that in the accompanying drawings, Figure 2 and 31 The corresponding parts in are given the same figure marks, and their descriptions will be appropriately omitted hereinafter.

[0618] exist Fig.32 , the camera system 400 is a lens interchangeable camera system. The camera system 400 includes a camera body 410 and a poly-eye interchangeable lens 420 (lens unit). In a state where the poly-eye interchangeable lens 420 is mounted on the camera body 410, the camera system 400 has a configuration substantially similar to that of the camera system 300, and substantially performs similar processing. That is, the camera system 400 functions as an imaging device that images an object and generates image data of a captured image similar to that of the camera system 300.

[0619] The camera body 410 has a configuration in which the multi-eye interchangeable lens 420 and other general interchangeable lenses can be attached and detached, similar to the camera body 10 having a configuration in which the multi-eye interchangeable lens 20 and the like can be attached or detached.

[0620] The multi-eye interchangeable lens 420 includes a feeding unit 23. Figure 1 and 2 As depicted, the feeding unit 23 includes the single-eye lenses 310 to 314 and the light sources 32L and 32R.

[0621] The feeding unit 23 is Fig.32 The lens barrel 21 (not shown) Figure 1 ) moves in the optical axis direction along the barrel optical axis. As the feeding unit 23 moves, the single-eye lenses 310 to 314 and the light sources 32L and 32R included in the feeding unit 23 also move integrally.

[0622] exist Fig.32 In the case of the camera system 300, similarly, the single-lens lens 31 i The optical paths of the light passing through the lens are independent of each other. iThe light of the image sensor 351 is emitted at different positions on the light receiving surface (eg, effective pixel area) of the image sensor 351 without entering the other single-eye lenses 31. i . Single lens 31 i At least the optical axes of the image sensor 351 are located at different positions on the light receiving surface of the image sensor 351, and the light having passed through the single-eye lens 31 is emitted at different positions on the light receiving surface of the image sensor 351. i At least some light.

[0623] Therefore, in the camera system 400, similarly to the case of the camera system 300, in the captured image generated by the image sensor 351 (the entire image output by the image sensor 351), the image is captured via the single-eye lens 31. i The images of the object formed are formed at positions different from each other. In other words, the image obtained from the captured image has the image formed at the single-eye lens 31. i That is, by mounting the polycular interchangeable lens 420 on the camera body 410 and imaging the subject, a plurality of monocular images can be obtained.

[0624] In the camera system 400, the camera body 410 includes an image sensor 351, a RAW signal processing unit 352, an area extraction unit 353, a camera signal processing unit 354, a through-image generation unit 355, an area specifying unit 356, an image reconfiguration processing unit 357, a bus 360, a display unit 361, a storage unit 362, a communication unit 364, an archiving unit 365, a control unit 381, and a storage unit 382. That is, the camera body 410 has a configuration other than the lens unit 320 and the optical system control unit 384 of the camera system 300.

[0625] Note that, in addition to the above configuration, the camera body 410 also includes a communication unit 441. The communication unit 441 communicates with (the communication unit 451 of) the poly-eye interchangeable lens 420 properly mounted on the camera body 410, and exchanges information, etc. The communication unit 441 can communicate with the poly-eye interchangeable lens 420 by any communication method. The communication may be wired communication or wireless communication.

[0626] For example, the communication unit 441 is controlled by the control unit 381, communicates with (the communication unit 451 of) the polyocular interchangeable lens 420, and acquires information provided from the polyocular interchangeable lens 420. In addition, for example, the communication unit 441 provides the information provided from the control unit 381 to the polyocular interchangeable lens 420 by communicating with (the communication unit 451 of) the polyocular interchangeable lens 420. The information exchanged between the communication unit 441 and the polyocular interchangeable lens 420 is arbitrary. For example, the information may be data or control information such as commands and control parameters.

[0627] In the camera system 400, the poly-eye interchangeable lens 420 further includes an optical system control unit 384, a communication unit 451, and a storage unit 452. The communication unit 451 communicates with the communication unit 441 in the poly-eye interchangeable lens 420 properly mounted on the camera body 410. This communication enables information exchange between the camera body 410 and the poly-eye interchangeable lens 420. The communication method of the communication unit 451 is arbitrary, and may be wired communication or wireless communication. In addition, the information exchanged by the communication may be data or control information such as commands and control parameters.

[0628] For example, the communication unit 451 acquires control information and other various types of information transmitted from the camera body 410 via the communication unit 441. The communication unit 451 supplies the information acquired in this way to the optical system control unit 384 as needed, and can use the information to control the feed unit 23, the single-eye lens 31 i wait.

[0629] Furthermore, the communication unit 451 may provide the acquired information to the storage unit 452 and store the information in the storage medium 453. Furthermore, the communication unit 451 may read the information stored in the storage medium 453 via the storage unit 452 and transmit the read information to the camera body 410 (communication unit 441).

[0630] In the camera system 400, the storage location of the viewpoint related information VI and the point light information SI corresponding to the poly-eye interchangeable lens 420 is arbitrary. For example, the viewpoint related information VI and the point light information SI may be stored in the storage medium 453 of the poly-eye interchangeable lens 420. Then, for example, the control unit 381 of the camera body 410 may access the storage unit 452 via the communication unit 451 and the communication unit 441 to read the viewpoint related information VI and the point light information SI from the storage medium 453. Then, after acquiring the viewpoint related information VI and the point light information SI, the control unit 381 may provide and set the viewpoint related information VI and the point light information SI to the area specifying unit 356.

[0631] For example, such processing may be performed at any timing or triggering time prior to imaging, such as when the poly-interchangeable lens 420 is properly mounted on the camera body 410, when the camera system 400 is powered on, or when the drive mode of the camera system 400 is switched to an imaging mode in which imaging of an object can be performed.

[0632] In this way, the camera body 410 can perform image processing on the captured image and the monocular image using the viewpoint-related information VI and the point light information SI corresponding to the polycular interchangeable lens 420.

[0633] In addition, the control unit 381 may provide the viewpoint related information VI and the point light information SI of the multi-eye interchangeable lens 420 acquired from the multi-eye interchangeable lens 420 together with the ID of the multi-eye interchangeable lens 420 to the storage unit 382 to store the information. In this case, the storage unit 382 stores the provided ID, viewpoint related information VI, and point light information SI in association with each other in the storage medium 383. That is, the camera body 410 may manage the viewpoint related information VI, the point light information SI, and the ID of the multi-eye interchangeable lens 420. The camera body 410 may manage the viewpoint related information VI and the point light information SI of a plurality of multi-eye interchangeable lenses 420.

[0634] In this way, the control unit 381 can obtain the ID of the poly-eye interchangeable lens 420 next time, thereby reading the viewpoint related information VI and the point light information SI corresponding to the ID from the storage unit 382 (storage medium 383). That is, the control unit 381 can easily obtain the viewpoint related information VI and the point light information SI corresponding to the poly-eye interchangeable lens 420.

[0635] In addition, the storage medium 383 may store the viewpoint related information VI and the point light information SI of each of the plurality of polycular interchangeable lenses 420 in advance in association with the ID of the polycular interchangeable lens 420. That is, in this case, the camera body 410 manages the viewpoint related information VI and the point light information SI of the plurality of polycular interchangeable lenses 420 in advance.

[0636] In this way, by using the ID of the poly-interchangeable lens 420 correctly mounted on the camera body 410, the control unit 381 can easily read the viewpoint-related information VI and the point light information SI corresponding to the ID from the storage unit 382 (storage medium 383).

[0637] <Handling of Attachment Position Deviation of Multi-eye Interchangeable Lens 420>

[0638] In the lens interchangeable camera system 400, similarly to the lens-integrated camera system 300, the attachment position of the poly-eye interchangeable lens 420 may deviate due to a manufacturing error or movement of the feed unit 23. In addition, in the lens interchangeable camera system 400, the attachment position of the poly-eye interchangeable lens 420 may deviate due to a mounting error. When the attachment position of the poly-eye interchangeable lens 420 deviates and an attachment error occurs at the attachment position, the accuracy of the process of cutting out a monocular image from a captured image and calculating parallax information using the monocular image is reduced.

[0639] Therefore, the area specifying unit 356 is able to detect the attachment error as a deviation (amount) of the attachment position of the polycular interchangeable lens 420 using the spot light image appearing in the captured image.

[0640] For example, the area specifying unit 356 may generate spot light information SI′ about a spot light image appearing in the captured image from the captured image, and detect a difference between the spot light information SI′ and the spot light information SI as an attachment error.

[0641] Furthermore, the region specifying unit 356 can correct the viewpoint-related information VI using the attachment error, and generate the viewpoint-related information VI′ as information for specifying the position of the monocular image deviated with the attachment error.

[0642] Then, the region extraction unit 353 can accurately cut out a monocular image from the captured image using the viewpoint related information VI′. In addition, the region extraction unit 353 uses the viewpoint related information VI′ to accurately specify the position of the viewpoint of the monocular image, and can accurately obtain disparity information using the monocular image and the position of the viewpoint.

[0643] As described above, in the camera system 300 or 400, since the monocular lenses 310 to 314 and the light sources 32L and 32R are integrally fed and fed, appropriate processing can be performed. That is, when a combination of attachment errors, installation errors, and various other errors is referred to as a monocular image position error, the monocular lenses 310 to 314 and the light sources 32L and 32R are integrally moved with the feeding unit 23, thereby being able to detect a positional deviation between viewpoint-related information and the position of the detected point light, and specifying the accurate position of the monocular image in the entire image without worrying about a monocular image position error caused by any reason at any feeding position.

[0644] <Description of Computer to Which the Present Technology is Applied>

[0645] Next, a series of processing of the area specifying unit 52, the image processing unit 53, the position calculating unit 57, the point light image detecting unit 62, the feed amount detecting unit 64, the area extracting unit 353, the area specifying unit 356, etc. may be performed by hardware or software. In the case where a series of processing is performed by software, a program configuring the software is installed in a general-purpose computer or the like.

[0646] Fig.33 : is a block diagram showing a configuration example of an embodiment of a computer in which a program for executing the above-described series of processes is installed.

[0647] The program can be recorded in advance in the hard disk 905 or the ROM 903 as a recording medium built in the computer.

[0648] Alternatively, the program may be stored (recorded) in a removable recording medium 911 driven by the drive 909. Such a removable recording medium 911 may be provided as a so-called software package. Here, examples of the removable recording medium 911 include a floppy disk, a compact disk read-only memory (CD-ROM), a magneto-optical (MO) disk, a digital versatile disk (DVD), a magnetic disk, a semiconductor memory, and the like.

[0649] Note that in addition to installing the program from the removable recording medium 911 to the computer as described above, the program can also be downloaded to the computer via a communication network or a broadcasting network and installed in the built-in hard disk 905. In other words, for example, the program can be transmitted to the computer in a wireless manner from a download site via an artificial satellite used for digital satellite broadcasting, or can be transmitted to the computer in a wired manner via a network such as a local area network (LAN) or the Internet.

[0650] The computer includes a central processing unit (CPU) 902 , and an input / output interface 910 is connected to the CPU 902 via a bus 901 .

[0651] When a user operating the input unit 907 or the like inputs a command through the input / output interface 910, the CPU 902 executes a program stored in a read only memory (ROM) 903 according to the command. Alternatively, the CPU 902 loads a program stored in a hard disk 905 into a random access memory (RAM) 904 and executes the program.

[0652] As a result, the CPU 902 executes the above-described processing according to the flowchart or the above-described processing of the block diagram. Then, the CPU 902 causes the output unit 906 to output the processing result, the communication unit 908 to transmit the processing result, and the hard disk 905 to record the processing result, for example, via the input / output interface 910 as needed.

[0653] Note that the input unit 907 is configured by a keyboard, a mouse, a microphone, etc. Furthermore, the output unit 906 is configured by a liquid crystal display (LCD), a speaker, and the like.

[0654] Here, in this specification, the processing performed by the computer according to the program does not necessarily have to be performed in chronological order in the order described as a flowchart. In other words, the processing performed by the computer according to the program also includes processing performed in parallel or individually (for example, parallel processing or processing performed by an object).

[0655] In addition, the program may be processed by one computer (processor), or may be processed in a distributed manner by a plurality of computers. In addition, the program may be transmitted to a remote computer and executed.

[0656] In addition, in this specification, the term "system" refers to a group of multiple configuration elements (devices, modules (components), etc.), and whether all the configuration elements are in the same housing is irrelevant. Therefore, multiple devices housed in separate housings and connected via a network and one device housing multiple modules in one housing are both systems.

[0657] Note that embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present technology.

[0658] For example, in the present technology, a configuration of cloud computing may be adopted in which one function is shared and processed collaboratively by a plurality of devices through a network.

[0659] Furthermore, the steps described in the above flowcharts may be performed by one device, or may be shared and performed by a plurality of devices.

[0660] Furthermore, in the case where a plurality of processes are included in one step, the plurality of processes included in one step may be executed by one device, or may be shared and executed by a plurality of devices.

[0661] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may be exhibited.

[0662] Note that the present technology can adopt the following configurations.

[0663] <1> An interchangeable lens comprising:

[0664] Lens tube;

[0665] A movable unit configured to be movable along the optical axis relative to the lens barrel;

[0666] a plurality of single-eye lenses configured to be movable integrally with the movable unit and arranged so that emission positions of imaging lights emitted through the respective single-eye lenses do not overlap with each other; and

[0667] One or more light sources are configured to be movable along the optical axis integrally with the movable unit and the plurality of single-eye lenses, and are arranged so that an emission position of parallel light emitted to an image sensor provided in a camera body does not overlap with an emission position of imaging light of each of the plurality of single-eye lenses.

[0668] <2> according to <1> Interchangeable lenses, including

[0669] The light source emits parallel light in an oblique direction tilted from the optical axis.

[0670] <3> according to <2> Interchangeable lenses, including

[0671] The light source is arranged at a position different from the center of the optical axis of the movable unit, and emits parallel light in an oblique direction inclined toward the center of the optical axis.

[0672] <4> according to <1> to <3> Interchangeable lenses according to any one of the preceding claims, comprising:

[0673] A plurality of said light sources.

[0674] <5> according to <1> to <4> The interchangeable lens as described in any one of the preceding claims, further comprising:

[0675] A storage unit is configured to store point light position information indicating a position of a light source irradiating an image sensor and single-eye image position information indicating an emission position in the image sensor, the emission position being an emission position of imaging light emitted from the plurality of single-eye lenses.

[0676] <6> An information processing device, comprising:

[0677] a detection unit configured to detect a light image on a captured image captured by the image sensor, the light image being a light image of parallel light emitted from a light source of a lens unit, the lens unit comprising:

[0678] Lens tube;

[0679] A movable unit configured to be movable along the optical axis relative to the lens barrel;

[0680] a plurality of single-eye lenses configured to be movable integrally with the movable unit and arranged so that emission positions of imaging lights emitted through the respective single-eye lenses do not overlap with each other; and

[0681] one or more light sources configured to be movable along the optical axis integrally with the movable unit and the plurality of single-eye lenses, and arranged so that an emission position of parallel light emitted to an image sensor provided in a camera body does not overlap an emission position of imaging light by each of the plurality of single-eye lenses; and

[0682] The processing unit is configured to perform processing according to the detection result of the detection unit.

[0683] <7> according to <6> An information processing device, wherein

[0684] The detection unit detects a detected light image position which is a position of the light image in the captured image.

[0685] <8> according to <7> An information processing device, wherein

[0686] The processing unit specifies an imaged monocular image position, which is a position of a monocular image having a position of a monocular lens as a viewpoint in a captured image, based on the detected light image position.

[0687] <9> according to <8> The information processing device further comprises:

[0688] a storage unit configured to store a stored light image position indicating a position of a light source irradiating an image sensor, and a stored monocular image position indicating an emission position of imaging light emitted from the plurality of monocular lenses in the image sensor, wherein

[0689] The processing unit specifies the imaging monocular image position based on the relationship between the storage light image position and the detection light image position.

[0690] <10> according to <9> An information processing device, wherein

[0691] The processing unit specifies the imaging monocular image position by correcting the stored monocular image position based on the relationship between the stored light image position and the detected light image position.

[0692] <11> according to <8> to <10> The information processing device described in any one of the above further includes

[0693] The associating unit is configured to associate the captured image with the imaging monocular image position.

[0694] <12> according to <9> or <10> The information processing device further comprises:

[0695] The associating unit is configured to associate the stored light image position, the detected light image position, and the stored monocular image position with the captured image.

[0696] <13> according to <9> or <10> The information processing device further comprises:

[0697] The associating unit is configured to associate the stored light image position, the difference between the stored light image position and the detected light image position, and the stored monocular image position with the captured image.

[0698] <14> according to <8> to <13> The information processing device described in any one of the above aspects further comprises:

[0699] The extracting unit is configured to extract the monocular image from the captured image according to the position of the imaged monocular image.

[0700] <15> according to <7> to <14> The information processing device as described in any one of the preceding claims, wherein

[0701] The processing unit detects the feeding amount of the movable unit based on the detection light image position.

[0702] <16> according to <6> arrive <15> The information processing device as described in any one of the preceding claims, wherein

[0703] The detection unit detects a plurality of light images respectively corresponding to a plurality of parallel lights respectively emitted from the plurality of light sources.

[0704] <17> according to <6> arrive <16> The information processing apparatus of any one of the items is a lens-integrated imaging device including a lens unit.

[0705] <18> according to <6> arrive <16> The information processing apparatus of any one of the foregoing is an imaging device to which the lens unit can be mounted as an interchangeable lens.

[0706] <19> An information processing method, comprising:

[0707] a detection step of detecting a light image on a captured image captured by an image sensor, the light image being a light image of parallel light emitted from a light source of a lens unit, the lens unit comprising:

[0708] Lens tube;

[0709] A movable unit configured to be movable along the optical axis relative to the lens barrel;

[0710] a plurality of single-eye lenses configured to be movable integrally with the movable unit and arranged so that emission positions of imaging lights emitted through the respective single-eye lenses do not overlap with each other; and

[0711] one or more light sources configured to be movable along the optical axis integrally with the movable unit and the plurality of single-eye lenses, and arranged so that an emission position of parallel light emitted to an image sensor provided in a camera body does not overlap an emission position of imaging light by each of the plurality of single-eye lenses; and

[0712] A processing step is to perform processing according to the detection result of the detection step.

[0713] <20> A program that causes a computer to:

[0714] a detection unit configured to detect a light image on a captured image captured by the image sensor, the light image being a light image of parallel light emitted from a light source of a lens unit, the lens unit comprising:

[0715] Lens tube;

[0716] A movable unit configured to be movable along the optical axis relative to the lens barrel;

[0717] a plurality of single-eye lenses configured to be movable integrally with the movable unit and arranged so that emission positions of imaging lights emitted through the respective single-eye lenses do not overlap with each other; and

[0718] one or more light sources configured to be movable along the optical axis integrally with the movable unit and the plurality of single-eye lenses, and arranged so that an emission position of parallel light emitted to an image sensor provided in a camera body does not overlap an emission position of imaging light by each of the plurality of single-eye lenses; and

[0719] The processing unit is configured to perform processing according to the detection result of the detection unit.

[0720] Symbol List

[0721] 10 Camera body

[0722] 11 Camera Mount

[0723] 20 Multiple interchangeable lenses

[0724] 21. Lens tube

[0725] 22 Lens Mount

[0726] 23 Feed unit

[0727] 310 to 314 Single-lens

[0728] 32L, 32R, 32U, 32D light sources

[0729] 41 Storage Units

[0730] 42 Communication Unit

[0731] 43 Control Unit

[0732] 51 Image Sensor

[0733] 52 Area Designation Unit

[0734] 53 Image Processing Unit

[0735] 54 Display Unit

[0736] 55 Storage Units

[0737] 56 Communication Unit

[0738] 57 Position calculation unit

[0739] 61 Control unit

[0740] 62 Point light image detection unit

[0741] 63 Feed rate information storage unit

[0742] 64 Feed rate detection unit

[0743] 101 Calibration data generation unit

[0744] 102 Calibration data storage unit

[0745] 103 Interpolation Unit

[0746] 104 Disparity information generating unit

[0747] 121 Housing

[0748] 122 LED

[0749] 123, 124 Lens

[0750] 211 Monocular image position information modification unit

[0751] 212 Monocular image extraction unit

[0752] 221 Associated Units

[0753] 230 Post-processing equipment

[0754] 231 Area Designation Unit

[0755] 232 Image Processing Unit

[0756] 233 Display unit

[0757] 234 Recording Unit

[0758] 235 Sending Unit

[0759] 241 Monocular image position information modification unit

[0760] 242 Monocular image extraction unit

[0761] 250 Post-processing equipment

[0762] 300 Camera System

[0763] 320 lens unit

[0764] 351 Image Sensor

[0765] 352 RAW signal processing unit

[0766] 353 Region Extraction Unit

[0767] 354 Camera signal processing unit

[0768] 355 Direct image generation unit

[0769] 356 Area Designation Unit

[0770] 357 Image Reconfiguration Processing Unit

[0771] 360 Bus

[0772] 361 Display Unit

[0773] 362 storage units

[0774] 363 Storage Media

[0775] 364 Communication Unit

[0776] 365 Archive Unit

[0777] 370 associated units

[0778] 381 Control Unit

[0779] 382 Optical system control unit

[0780] 400 Camera System

[0781] 410 Camera Body

[0782] 420 Multi-eye Interchangeable Lenses

[0783] 441, 451 communication unit

[0784] 452 storage units

[0785] 453 Storage Media

[0786] 901 Bus

[0787] 902 CPU

[0788] 903 Read-only memory

[0789] 904 RAM

[0790] 905 Hard Drive

[0791] 906 Output Unit

[0792] 907 Input Unit

[0793] 908 Communication Unit

[0794] 909 Drive

[0795] 910 Input / Output Interface

[0796] 911 Removable recording media

Claims

1. An interchangeable lens comprising: Lens tube; A movable unit configured to be movable along the optical axis relative to the lens barrel; a plurality of single-eye lenses configured to be movable integrally with the movable unit and arranged so that emission positions of imaging light emitted through the respective single-eye lenses do not overlap with each other; and One or more light sources are configured to be movable along the optical axis integrally with the movable unit and the plurality of single-eye lenses, and are arranged so that an emission position of parallel light emitted to an image sensor provided in a camera body does not overlap with an emission position of imaging light of each of the plurality of single-eye lenses.

2. The interchangeable lens according to claim 1, wherein The light source emits parallel light in an oblique direction tilted from the optical axis.

3. The interchangeable lens according to claim 2, wherein The light source is arranged at a position different from the center of the optical axis of the movable unit, and emits parallel light in an oblique direction inclined toward the center of the optical axis.

4. The interchangeable lens according to claim 1, comprising: A plurality of said light sources.

5. The interchangeable lens according to claim 1, further comprising: A storage unit is configured to store point light position information indicating a position of a light source irradiating an image sensor and single-eye image position information indicating an emission position in the image sensor, the emission position being an emission position of imaging light emitted from the plurality of single-eye lenses.

6. An information processing device, comprising: a detection unit configured to detect a light image on a captured image captured by the image sensor, the light image being a light image of parallel light emitted from a light source of a lens unit, the lens unit comprising: Lens tube; A movable unit configured to be movable along the optical axis relative to the lens barrel; a plurality of single-eye lenses configured to be movable integrally with the movable unit and arranged so that emission positions of imaging lights emitted through the respective single-eye lenses do not overlap with each other; and one or more light sources configured to be movable along the optical axis integrally with the movable unit and the plurality of single-eye lenses, and arranged so that an emission position of parallel light emitted to an image sensor provided in a camera body does not overlap an emission position of imaging light by each of the plurality of single-eye lenses; and The processing unit is configured to perform processing according to the detection result of the detection unit.

7. The information processing device according to claim 6, wherein The detection unit detects a detected light image position which is a position of the light image in the captured image.

8. The information processing device according to claim 7, wherein The processing unit specifies an imaged monocular image position, which is a position of a monocular image having a position of a monocular lens as a viewpoint in a captured image, based on the detected light image position.

9. The information processing device according to claim 8, further comprising: a storage unit configured to store a stored light image position indicating a position of a light source irradiating an image sensor, and a stored monocular image position indicating an emission position of imaging light emitted from the plurality of monocular lenses in the image sensor, wherein The processing unit specifies the imaging monocular image position based on the relationship between the storage light image position and the detection light image position.

10. The information processing device according to claim 9, wherein The processing unit specifies the imaging monocular image position by correcting the stored monocular image position based on the relationship between the stored light image position and the detected light image position.

11. The information processing device according to claim 8, further comprising: The associating unit is configured to associate the captured image with the imaging monocular image position.

12. The information processing device according to claim 9, further comprising: The associating unit is configured to associate the stored light image position, the detected light image position, and the stored monocular image position with the captured image.

13. The information processing device according to claim 9, further comprising: The associating unit is configured to associate the stored light image position, the difference between the stored light image position and the detected light image position, and the stored monocular image position with the captured image.

14. The information processing device according to claim 8, further comprising: The extracting unit is configured to extract the monocular image from the captured image according to the position of the imaged monocular image.

15. The information processing apparatus according to claim 7, wherein The processing unit detects the feeding amount of the movable unit based on the detection light image position.

16. The information processing device according to claim 6, wherein The detection unit detects a plurality of light images respectively corresponding to a plurality of parallel lights respectively emitted from the plurality of light sources. 17 . The information processing apparatus according to claim 6 , wherein the information processing apparatus is a lens-integrated imaging device including a lens unit.

18. The information processing apparatus according to claim 6, wherein the information processing apparatus is an imaging device, and the lens unit is attachable to the imaging device as an interchangeable lens.

19. An information processing method, comprising: a detection step of detecting a light image on a captured image captured by an image sensor, the light image being a light image of parallel light emitted from a light source of a lens unit, the lens unit comprising: Lens tube; A movable unit configured to be movable along the optical axis relative to the lens barrel; a plurality of single-eye lenses configured to be movable integrally with the movable unit and arranged so that emission positions of imaging lights emitted through the respective single-eye lenses do not overlap with each other; and one or more light sources configured to be movable along the optical axis integrally with the movable unit and the plurality of single-eye lenses, and arranged so that an emission position of parallel light emitted to an image sensor provided in a camera body does not overlap an emission position of imaging light by each of the plurality of single-eye lenses; and A processing step is to perform processing according to the detection result of the detection step.

20. A computer program product, comprising a computer program, wherein the computer program causes a computer to: a detection unit configured to detect a light image on a captured image captured by the image sensor, the light image being a light image of parallel light emitted from a light source of a lens unit, the lens unit comprising: Lens tube; A movable unit configured to be movable along the optical axis relative to the lens barrel; a plurality of single-eye lenses configured to be movable integrally with the movable unit and arranged so that emission positions of imaging light emitted through the respective single-eye lenses do not overlap with each other; and one or more light sources configured to be movable along the optical axis integrally with the movable unit and the plurality of single-eye lenses, and arranged so that an emission position of parallel light emitted to an image sensor provided in a camera body does not overlap with an emission position of imaging light of each of the plurality of single-eye lenses; as well as The processing unit is configured to perform processing according to the detection result of the detection unit.

Images