Sensor device, calibration system and method for operating a sensor device

Abstract

A sensor device (100) for capturing images of a scene comprises a sensor substrate (110) having a light receiving surface (112) on which a plurality of light receiving elements is arranged, sensor optics (120) that is configured to guide light from the scene onto the light receiving substrate (112), and a control unit (130) that is configured to control operation of the sensor device (100). Here, when the scene comprises a calibration chart (200) that shows a texture (T) with known geometrical features arranged with a predetermined pitch (P) that lead to a moiré effect when imaged by the plurality of light receiving elements, the control unit (130) is configured to convert the image (S) showing the moiré effect such that it shows the texture (T) of the calibration chart (200) without distortion, and to determine imaging properties of the sensor optics (120) based on the converted image (M).

Description

SENSOR DEVICE, CALIBRATION SYSTEM AND METHOD FOR OPERATING A SENSORDEVICEFIELD OF THE INVENTIONThe present technology relates to a sensor device, a calibration system and a method for operating a sensor device, in particular, to a sensor device, a calibration system and a method for operating a sensor device that allow sensor calibration based on the moire effect.BACKGROUNDSensor devices such as imaging sensors, digital cameras, and the like suffer from imaging artifacts caused by the limited pixel size and imaging properties of the used optics. In order to obtain images of high quality that resemble the captured scene as accurately as possible, it is necessary to measure these imaging artifacts. Then, the imaging artifacts can be corrected in the captured images. Preferably, the imaging artifacts are measured once during calibration of a sensor device and the correction is carried out immediately once an image is captured. However, known calibration methods come with drawbacks.It is therefore desirable to improve sensor devices such as to allow a highly precise calibration that allows in turn a precise correction of imaging artifacts in order to arrive at an accurate representation of the captured scene.SUMMARY OF INVENTIONTo this end, a sensor device for capturing images of a scene is provided. The sensor device comprises a sensor substrate having a light receiving surface on which a plurality of light receiving elements is arranged, sensor optics that is configured to guide light from the scene onto the light receiving substrate, and a control unit that is configured to control operation of the sensor device. Here, when the scene comprises a calibration chart that shows a texture with known geometrical features arranged with a predetermined pitch that lead to a moire effect when imaged by the plurality of light receiving elements, the control unit is configured to convert the image showing the moire effect such that it shows the texture of the calibration chart without distortion, and to determine imaging properties of the sensor optics based on the converted image.Further, a method for operating a sensor device as described above is provided, the method comprising: capturing an image of the scene comprising a calibration chart that shows a texture with known geometrical features arranged with a predetermined pitch that lead to a moire effect when imaged by the plurality of light receiving elements; converting, by the control unit, the image showing the moire effect such that it shows the texture of the calibration chart without distortion; and determining, by the control unit, imaging properties of the sensor optics based on the converted image.Thus, when a calibration chart is used that deliberately causes a moire effect when it is captured by the sensor device, the sensor device can use the moire effect to determine imaging properties of the sensor optics. In particular, it is known that the pattern shown in the moire effect is basically an enlarged and distorted copy of the patterns of the imaged object. Thus, the moire patterns obtained when capturing the calibration chart produce an enlarged and distorted image of the texture of the calibration chart. When converting this image such that the distortion is removed, an enlarged image of the texture of the calibration chart as seen by the light receiving elements, i.e. the pixels / subpixels of the sensor device, is generated. Differences between the texture of the calibration chart and the converted image are then due to the sensor optics (understood as including lens effects as well as effects caused by pixel size limitations). Thus, by using the moire effect, properties of the sensor optics can be obtained. Due to enlargement caused by the moire effect, these properties can be measured with high resolution.BRIEF DESCRIPTION OF DRAWINGFig. 1 is a schematic block diagram of a calibration system including a sensor device and a calibration chart.Fig. 2 shows schematic diagrams for explaining a coordinate transformation.Fig. 3. is another schematic diagram for explaining a coordinate transformation.Fig. 4 is a schematic diagram for explaining a derivation of imaging properties of a sensor device.Fig. 5 shows a correspondence of points on images related by a coordinate transformation.Figs. 6A and 6B are schematic illustrations of a display of imaging properties of a sensor device.Fig. 7 is a schematic diagram of a calibration chart.Fig. 8 is a schematic diagram for explaining an iterative coordinate transformation.Fig. 9 is a schematic diagram of another calibration chart.Figs. lOAto 10C are schematic diagrams of further calibration charts.Fig. 11 is a schematic diagram for explaining a coordinate transformation.Figs. 12A and 12B are schematic diagrams of a calibration system that has a display for displaying a calibration chart.Fig. 13 is a schematic diagram of a calibration chart that can be displayed with different colors.Fig. 14 shows schematically a process flow of a method for operating a sensor device.Fig. 15 is a block diagram illustrating a configuration of an imaging device.Fig. 16 is a functional block diagram illustrating an example of an electronic apparatus (camera) including an imaging device illustrated in Fig. 15.Fig. 17 is a block diagram depicting an example of a schematic configuration of an in-vivo information acquisition system.Fig. 18 is a view depicting an example of a schematic configuration of an endoscopic surgery system.Fig. 19 is a block diagram depicting an example of a functional configuration of a camera head and a camera control unit (CCU).Fig. 20 is a block diagram depicting an example of schematic configuration of a vehicle control system.Fig. 21 is a diagram of assistance in explaining an example of installation positions of an outside-vehicle information detecting section and an imaging section.DETAILED DESCRIPTIONFig. 1 shows a schematic diagram of a calibration system 10 that includes a sensor device 100 for capturing images of a scene, and a calibration chart 200 located in the scene.The sensor device 100 comprises a sensor substrate 110 having a light receiving surface 112 on which a plurality of light receiving elements 114 such as pixels or subpixels is arranged, sensor optics 120 that is configured to guide light from the scene onto the light receiving substrate 112, and a control unit 130 that is configured to control operation of the sensor device 100. Thus, the sensor device 100 comes with the typical components of a conventional digital camera as e.g. also described below with respect to Figs. 15 and 16.Here, the sensor optics 120 usually produces imaging artifacts such as blurring or chromatic aberration. In order to allow generation of high-quality images it is necessary to calibrate the sensor device 100. In the calibration process properties of the sensor optics 120 (including e.g. a lens system of the sensor device 100, shutters, but also limitations due to pixel size) are determined, in order to allow compensation of imaging errors caused by the sensor optics 120.Determination of these properties is caried out by the control unit 130. Preferably the control unit 130 is the processing unit of the sensor device 100 that is also used to control operation of the sensor device 100. However, the control unit 130 may also be a dedicated processor within the sensor device 100. Also, the control unit 130 may be (partially or fully) located outside the sensor device 100. In this case, the sensor device 100 provides imaging data to the control unit 130 during calibration and receives compensation instructions from the outside that can be stored in the sensor device 100. In general, the control unit 130 may be implemented in hardware and / or software. It may be a general -purpose processor such as a CPU or a GPU or any other circuitry. It may also be implemented by a general-purpose computer.As schematically illustrated in Fig. 1, the calibration chart 200 shows a texture T with known geometrical features that are arranged with a predetermined pitch P. The geometric features are in particular a periodic arrangement of bright and dark regions with high contrasting edges. For example, as shown in Fig. 1, the calibration chart 200 may include bright spots (e.g. squares, circles, crosses or the like) on dark ground. Of course, just the same dark spots on bright ground can be used. When the image of the calibration chart 200 passes through the sensor optics 120 it will be modified due to the imaging properties of the sensor optics 120. In particular, the point spread function, PSF, of the sensor optics as well as the modulation transfer functions MTF, of the sensor optics 120 will lead to a blurred image B that impinges on the sensor substrate 110. This is schematically illustrated on the right of Fig. 1, where the high contrast peaks of the texture T are changed to the blurred imaged B.In principle, it is possible to determine imaging properties of the sensor optics 120 directly from the blurred image B. However, in practice this is often not possible since effects of neighboring pixels blend into each other , which makes an efficient treatment difficult.In order to deal with this problem a calibration chart 200 with a texture T is used that leads to a moire effect when imaged by the plurality of light receiving elements 114. In particular, this means that the pitch P at which the geometrical features of the texture T are arranged deviates slightly from the pitch at which the light receiving elements 114 are arranged. For example, the pitches might differ by 10%, 5%, 2% or 110 / / o.The generation of the moire effect is schematically explained at the bottom right of Fig. 1. Only few maxima of the blurred image B will be centrally aligned with the pixels on the light receiving surface 112. Starting with an aligned peak, the next peak will be slightly shifted out of the central region of the pixel. Thus, the measured intensity will decrease. This continues until the peaks of the blurred image B start to shift towards pixel centers again. In this manner an intensity distribution is measured that encodes the intensity distribution of the blurred image B in an enlarged manner. Thus, in principle, it is possible to deduce imaging properties of the sensor optics 120 from the image S that shows the moire effect.However, compared to the original texture T or the blurred image B, the image S captured by the sensor device 100 will also contain a distortion caused for example by minute tilts of the plane of the calibration chart 200 and the light receiving surface 112. Thus, using just the image S it is rather difficult to deduce the desired imaging properties.Therefore, the control unit 130 is configured to convert the image S showing the moire effect such that it shows the texture T of the calibration chart 200 without distortion. From this converted image it is then possible to determine imaging properties of the sensor optics 120. In particular, by removing the distortion on the image S an enlarged image of the blurred image B is generated. In this enlarged image it is possible to deduce imaging properties on a pixel-by-pixel basis, i.e. for light rays that hit only single pixels. Thus, using the converted image allows a detailed analysis of the imaging properties.As discussed with respect to Figs. 2 and 3, the control unit 130 may in particular be configured to convert the image S showing the moire effect by a coordinate transformation.To this end, a normalized coordinate system with normalized texture coordinates (|)Tis introduced for the texture T on the calibration chart 200 such that the size of the pitch P amounts to 1. This is illustrated at the left of Fig. 2 for the x-coordinate of (f>TAs can be seen there, the distance between the geometrical features of the texture T, e.g. rectangular spots, amounts to 1.In addition, also for the image S captured by the sensor device 100, which image S shows the moire effect, a normalized coordinate system with normalized sensor coordinates (|)sis introduced. In <j>scoordinates the pitch D between two pixels amounts to 1. Here, it should be noted that the pitch D refers to one cycle of the pixel pattern. As illustrated in the middle of Fig. 2, if all pixels are the same, the pitch D is the distance between neighboring pixels. If different pixels are used, as e.g. RGB pixels in a Bayer pattern, the pitch D amounts to the distance after which the pixel pattern repeats. On the right of Fig. 2 a Bayer pattern arrangement is shown, where 2 x 2 pixels form a unit cell of the pattern. Accordingly the pitch D amounts to the distance between two neighboring 2 x 2 pixel cells, i.e. to two (sub)pixels.Moreover, the pitch D may also be selected according to the texture pattern interval. For example, when the projected texture pattern has almost a 4 pixel cycle in the sensor plane, the sensor sampling pitch D should be chosen to be 4 pixels. This process can be used to sub-sample an image by 4 pixels. In case of a Bayer pattern, the pitch D should be chosen from the scale of 2 pixels, this means D should amount to e.g., 2, 4, 6, 8, ...pixels.The coordinates of the converted image F are then obtained by the equationF = F - F.Here, the calculation starts with a coordinate on the sensor image S. By using the known positional relation between the calibration chart 200 and the sensor device 100 (extrinsic parameters) as well as the known ideal imaging properties of the sensor device 100 (intrinsic properties) it can be deduced which position on the calibration chart was projected onto the position with normalized sensor coordinate F The normalized texture coordinate <F <F) corresponding to this position is used in the above equation. This is schematically illustrated in Fig. 3.A subtraction of the normalized sensor coordinate (|)sfrom its corresponding normalized texture coordinate F will then lead to the coordinates F of the converted image M, also called moire coordinates. As schematically illustrated in Fig. 3, this will lead to a representation of the contents of the image S, i.e. to a representation of the moire effect without distortion. That is, the converted image M merely shows the enlargement caused by the moire effect, but not the distortion. The converted image M is an enlargement of the blurred image B that contains the errors caused by the imaging properties of the sensor optics 120. By comparing this to the (appropriately scaled) original texture, said imaging properties can be deduced.Particularly useful is here if the texture T of the calibration chart 200 is a two-dimensional array of dots of known size and a known pitch P that differs from the pitch D of the light receiving elements 114 such that a moire effect is generated during imaging, preferably such that the projection of the pitch P onto the light receiving surface 112 differs from the pitch of the light receiving elements 114 by 10%, 5%, 2% or 1%. When the projected pattern pitch P is very different from the pitch of the light receiving elements 114, it is also possible to select a different sampling pitch by subsampling the image, such that both texture and sampling pitch on the sensor plane will become similar and thus generate a moire effect. As explained above, the converted image M shows the dots then with the blur caused by the sensor optics 120 and the control unit 130 is configured to determine the blur caused by the sensor optics from the converted image M.In particular, using dot shaped geometrical features on the texture T allows determining with the control unit 130 for each dot in the converted image M the PSF of the light receiving element(s) 114 at the position of the sensor substrate 110 corresponding to the respective dot in the converted image M, and / or at least one MTF along a line through the respective dot.This is illustrated schematically in Fig. 4. The texture T constituted by regularly aligned dots / circles (here for example dark circles on bright ground) is captured by the sensor device 100. Due to the moire effect the sensor device 100 does not directly capture an image of the texture T, but an image S containing a distorted and enlarged version of the texture T. Using the image S and the knowledge about the texture T that is present during the calibration procedure (size and pitch of dots, distance and orientation between calibration chart 200 and sensor device 100, ideal imaging properties of sensor device 100) the control unit 130 derives the converted image M, an enlarged and blurred version of the original texture T.Each dot on the converted image M can be used to deduce a corresponding PSF. To this end, the “dot spread function”, which is obtained by comparing the blurred image of the dot in the converted image M with the image of the corresponding dot in the texture T, is deconvoluted. It can visualize the 2d kernel of the PSF with high resolution. This is symbolized in Fig. 4 by arrow Al.Further, for each dot MTFs can be measured. In Fig. 4 this is exemplarily shown for MTFs along a horizontal line H and a vertical line V. The MTF is obtained, as in principle known, by determining intensity values across a dark-bright border (arrows A2 and A3) and by calculating the derivative of the so obtained curve followed by a Fast Fourier Transformation of the derivative (arrow A4).Here, it should be noted that although a determination of the MTFs along horizontal and vertical lines is shown in Fig. 4, it is in principle possible to determine the MTF along any line crossing the circular dot of the converted image M. Thus, using a dot texture causing a moire effect allows to measure simultaneously various MTFs, which is not possible with the known methods.As explained above, the converted image M is expected to show the geometrical features of the texture without distortion. Using this knowledge it is also possible to determine lens distortions, i.e. distortions caused by deviations of the imaging properties of the sensor optics 120 from its ideal imaging properties. In fact, any distortion that remains in the converted image M must be caused by such lens distortions. Thus, by comparing the original texture T and the converted image M, it is possible to deduce lens distortions.The usage of a texture T that causes a moire pattern when captured by the sensor device 100, and in particular the usage of circular dots, provides therefore the advantage that several imaging properties such as the PSF, the MTFs, and lens distortion can be measured simultaneously. Moreover, these quantities can be determined on a dot-per-dot basis across the sensor image S, i.e. approximately with pixel resolution. Thus, calibration using the converted moire coordinates is less time consuming and more accurate.In this regard the control unit may also generate data that allow visualizing, when rendered on a display 300, a correspondence of points on the image S showing the moire effect and the converted image M. Further, data may be generated that indicate for each dot in the image S showing the moire effect and / or each dot in the converted image M the blur caused by the sensor optics 120 for the respective dot.For example, as schematically shown in Fig. 5 it is possible to display on a screen 300 the unconverted image S as seen by the sensor device 100 together with the converted image M. Here, positions that are mapped to each other by the coordinate transformation can be visualized e.g. by displaying arrows pointing from the converted image M to the unconverted image S or vice versa. Which positions to correlate can be chosen via a user interface, e.g. by using a pointing device such as a mouse cursor. PSF, MTF and / or lens distortion may be displayed next to the according arrow.Related methods of visualizing the corresponding image positions together with their imaging properties are shown in Figs. 6A and 6B. As shown in Fig. 6A, instead of displaying the entire converted image M, only the corresponding dot on the converted image M can be displayed once a dot in the unconverted image S is selected. Together with the dot on the converted image M imaging properties at this position can be displayed, such as the corresponding dot spread function, PSF or MTFs. As shown in Fig. 6B several dots may be selected and the corresponding converted dot images are assigned by arrows to their respective locations on the unconverted image S. In this manner, the imaging properties of the sensor device 100 can be analyzed in detail.In the above description the properties of the calibration chart 200 are considered to be known by the control unit 130. This could e.g. be achieved by manually inputting the respective properties like the exact shape of the geometrical features (e.g. circular, rectangle, star, cross, irregular), the size of the geometrical features, the pitch between the features into the control unit 130. However, it is also possible to allow automatic read out of these parameters from information embedded into the calibration chart 200.For example, the calibration chart 200 may comprise a two-dimensional marker 210 that indicates information including at least one of a position of the marker 210 on the calibration chart 200, the geometrical features of the texture T (e.g. shape and size of spots), and the pitch P. The control unit 130 is then capable to deduce said parameters from the captured image of the marker. This eases the input of calibration chart parameters into the control unit 130.As illustrated in Fig. 7 the marker 210 may be a binary square fiducial marker that consists of a specific pattern of brigth squares on n x n squares. Such markers are usually non-rotationally symmetric and allow therefore a detection of the relative orientation of the sensor device 100 and the calibration chart. Examples of such markers are e.g. ArUco markers.By providing information on the position of the marker 210 in the texture, like e.g. the four comers of the marker, the mapping between the normalized sensor coordinates (|)sand the normalized texture coordinates <|> ' can be made more accurate. Since marker features, like e.g. the four comers of the marker, can be easily recognized in the sensor image S, a precise mapping between the coordinates on the sensor image S and the texture T can be established at least for the marker features. This could then e.g. be used together with the information on the geometrical features of the texture (size, shape, pitch) to determine distance and orientation between calibration chart 200 and sensor device 100. On the other hand, if all extrinsic parameters are known in principle, the marker position can be used to check and refine these parameters.This is exemplary illustrated in Fig. 8. Here, the marker information is first used to determine initial extrinsic parameters and information on the texture. This is then used to initially deduce coordinates on the calibration chart 200 and the mapping from the calibration chart 200 to the light receiving surface 112. Then, an initial converted image Ml is generated.Using the knowledge about the geometric properties of the calibration chart 200, in particular on the arrangement of geometrical features, it is possible to iteratively optimize the extrinsic parameters defining the mapping between (|)sand c|)Tuntil intensity peaks in the converted image M are aligned to the same grid as the geometrical features of the texture T. This results in an optimally aligned converted image M2. Here, it should be noted that due to information given in / by the marker 210, alignment is first achieved in areas C close to the marker 210. However, by including more intensity peaks in the converted image Ml in the alignment, the alignment will become more accurate.In the above description the calibration chart 200 was assumed to contain only a single texture T. This provides the problem that this calibration chart 200 can be used only for a given sensor device 100 in a specific distance and / or orientation. For example, for a sensor device 100 with a large PSF or a large distance between calibration chart 200 and sensor device 100 larger pitches P should be used.To mitigate this problem, calibration charts 200 as schematically illustrated in Fig. 9 may be used that show different textures Tl, T2 in different sections 220. Then, the same calibration chart 200 can be used for different sensor devices 100 or different distances for which at least one of the textures Tl, T2 on the calibration chart 200 leads to a moire effect when imaged by the sensor device 100. Of course, as shown in Fig. 9 a marker 210 may be added in one (or more sections). This marker 210 could also be used to indicate section borders as further easily discernable, and hence easily mappable portions of the calibration chart 200 and the sensor image S.Further, during calibration it is often necessary to tilt the calibration chart with respect to the sensor device 100 as shown in Fig. 10A. However, such a tilt changes the apparent distance between the geometric features on the calibration chart. Thus, a moire effect will possibly only be achieved at specific positions, but not for the entire texture T as shown in Fig. 10B. This deteriorates the measurement accuracy, and hence the reliability of the method.To mitigate this problem, the texture T comprises a two-dimensional array of dots that resemble the appearance of a regular two-dimensional array R of dots that is viewed under a slanting angle, e.g. an angle of 15°, 25°, 35°, 45°, 55°, 65° or 75°. This means, as illustrated at the top of Fig. 10C, the dots are arranged on a virtual grid whose lines converge at least in one grid direction. In the example of Fig. 10C the dots are arranged on parallel vertical lines and on converging “horizontal” lines. They resemble thereby the appearance of a regularly arranged dot pattern that is viewed under a slanting angle. While in Fig. 10C vertical lines are parallel, it is to be understood that also converging “vertical” and “horizontal” lines are possible.If one captures an accordingly textured calibration chart with a tilt that corresponds to the slanting angle, then the texture will appear to the senor device 100 regularly arranged across the entire calibration chart 200. Thus, the entire calibration chart 200 will give rise to a moire pattern, as illustrated at the bottom of Fig. 10C. Accordingly, the measurement accuracy can be kept as high as for the measurements without tilt.It has to be noted that for calculating the moire coordinates (|)Ithe normalized sensor coordinates c|)sneed to be compared with the regular arrangement that is actually seen by the sensor device 100. Thus, as illustrated schematically in Fig. 11, it is necessary to include the tilting operation into the mapping between calibration chart 200 and the sensor image S. In particular, the normalized texture coordinates c|)Tare obtained by applying the function g to the coordinates of the calibration chart that converts the regular dot array to a tilted dot array. Here g may lead to an affine conversion or a perspective conversion. The moire coordinates are then still given by the equation (f>M= (|)T- ( , but with (|)T=g(rT). with rTthe coordinates on the calibration chart 200. This allows to determine imaging properties of the sensor device 100 also fortified calibration charts 200.In the above, the calibration charts 200 were considered to be real physical objects like e.g. printed boards or photoetched plates. However, this has of course the drawback that one calibration chart 200 can only be used for a limited number of different sensor devices 100, even if it contains different sections as described above with respect to Fig. 9.To mitigate this problem, it is conceivable to provide a display 300 that is configured to display the calibration chart 200 on it. Then, it is in principle possible to display textures T that can be adapted to the presently calibrated sensor device 100. This apparently eases the calibration process.However, since the moire effect is extremely sensitive to smallest changes in the relative position and orientation of calibration chart 200 and sensor device 100, a problem might arise if pixel shape and pixel locations of the display 300 are not precisely known. In addition, the pixel substrate of the display 300 may be non-flat but slightly curved. Also such a deviation from flatness must be precisely known. Further, the properties of a cover glass of the display pixels need to be precisely known such as a refractive index and a thickness of the cover glass, since otherwise the precise optical length between display pixels and imaging pixels cannot be determined.Accordingly, at least some of these display parameters are measured for the used display 300 and provided to the control unit 130. The control unit 130 uses these display parameters then in the conversion of the sensor image S.A possible implementation of such a system is schematically illustrated in Fig. 12A, while different shapes and arrangements of display pixels 310 are illustrated in Fig. 12B. As illustrated in Fig. 12A, a pattern controller 320 may be provided that is configured to change the texture T displayed on the display 300 such as to match the sensor device 100. The pattern controller 320 may be part of the sensor device 100 or may be an external unit. Further, the display parameters are provided from a storage 330 to thecontrol unit 130. The storage 330 may be an external database, a memory of the computer to which the display 300 belongs, or a memory of the sensor device 100. While the pattern controller 320 provides the control unit 130 the parameters of the texture T, the storage 330 provides the display parameters to the control unit 130. In this manner, the control unit 130 is enabled to calculate a precise mapping also when an electronic display 300 is used as calibration chart 200.Here, as schematically illustrated in Fig. 13 the display 300 may comprise subpixels 312 of different colors. As shown in Fig. 13 RGB subpixels may constitute one display pixel 310. The display 300 displays the same calibration chart 200 / texture T then with different colors at different times. This is illustrated in Fig. 13 by three times tl, t2, and t3 at which red subpixels 312, green subpixels 312 and blue subpixels 312 of the upper and lower row are illuminated, respectively, while the display pixels 310 of the middle row remain dark at all the times. Thus, at different times the geometrical features of the texture T are emitted from slightly shifted positions.This is taken account of by information the control unit 130 about the different subpixel positions. The control unit 130 considers these positions then, when converting the sensor image S for the different colors into the converted image M. In this manner imaging properties of the sensor device 100 can be easily obtained for different color channels. This allows measurements on the chromatic aberration of the sensor optics 120 simultaneously with measurements of the PSF, the MTFs, and the lens distortion. Accordingly, a calibration process is further eased.Fig. 14 shows a schematic flow chart of a method for operating a sensor device 100 / a calibration system 10 as described above.At SI 10 an image S of the scene comprising a calibration chart 200 is captured that shows a texture T with known geometrical features arranged with a predetermined pitch P that lead to a moire effect when imaged by the plurality of light receiving elements 114.At SI 20 the control unit 130 converts the image S showing the moire effect such that it shows the texture T of the calibration chart 200 without distortion.At S130 the control unit 130 determines imaging properties of the sensor optics 120 based on the converted image M.In this manner a sensor device 100 can be easily, accurately, and reliably calibrated by using the moire effect.Fig. 15 is a block diagram illustrating, for example, an overall configuration of a sensor device 100 as described above. This sensor device 100 is a CMOS image sensor. The sensor device 100 includes a pixel section (light receiving surface 112) as an imaging area on the sensor substrate 110. The sensor device 100 includes, for example, the peripheral circuit sections 1300 in the peripheral region 1000 of the sensor substrate 110. The peripheral circuit sections 1300 include the row scanner 1310, the horizontal selector1330, the column scanner 1340, and the system controller 1320.The light receiving surface 112 includes, for example, the plurality of unit pixels, i.e. light receiving elements 114 (corresponding to the sensor device 100) two-dimensionally disposed in a matrix. These unit pixels are provided with pixel drive lines Lread (specifically, row selection line and reset control line) in each of pixel rows, for example, and provided with vertical signal lines Lsig in each of pixel columns. The pixel drive lines Lread are each for transmitting drive signals for reading out signals from pixels. One end of each of the pixel drive lines Lread is coupled to the output end of the row scanner 1310 corresponding to each row.The row scanner 1310 includes a shift register, an address decoder, and the like. The row scanner 1310 is a pixel driving section that drives the respective unit pixels in the light receiving surface 112, for example, row by row. Signals outputted from the respective unit pixels in the pixel rows selectively scanned by the row scanner 1310 are supplied to the horizontal selector 1330 through the respective vertical signal lines Lsig. The horizontal selector 1330 includes an amplifier, a horizontal selection switch, and the like provided for each of the vertical signal lines Lsig.The column scanner 1340 includes a shift register, an address decoder, and the like. The column scanner 1340 drives each of the horizontal selection switches of the horizontal selector 1330 in order while scanning the horizontal selection switches. As a result of this selective scanning by the column scanner 1340, signals of the respective pixels to be transmitted through the respective vertical signal lines Lsig are outputted to horizontal signal lines 1350 in order and are transmitted to the outside of the sensor substrate 110 through the horizontal signal lines 1350.The circuit portions including the row scanner 1310, the horizontal selector 1330, the column scanner 1340, and the horizontal signal lines 1350 may be formed directly on the sensor substrate 110 or may be provided to external control IC. In addition, those circuit portions may be formed on another substrate coupled by a cable or the like.The system controller 1320 receives a clock, data for an instruction about an operation mode, and the like that are supplied from the outside of the sensor substrate 110. In addition, the system controller 1320 outputs data such as internal information of the sensor device 100. The system controller 1320 further includes a timing generator that generates a variety of timing signals and controls the driving of the peripheral circuits such as the row scanner 1310, the horizontal selector 1330, and the column scanner 1340 on the basis of the variety of timing signals generated by the timing generator.The sensor device 100 described above is applicable, for example, to any type of electronic apparatus having an imaging function including a camera system such as a digital still camera and a video camera, a mobile phone having an imaging function, and the like. Fig. 16 illustrates a schematic configuration of an electronic apparatus 1000 (camera) as an example thereof. This electronic apparatus 1000 is, for example, a video camera that makes it possible to shoot a still image or a moving image. The electronic apparatus 1000 includes the sensor device 100, with sensor optics 120 that includes for example anoptical system (optical lens) 2100, a shutter device 2110, a driver 2130, and a signal processor 2120, which might also be part of the control unit 130. The driver 2130 drives the sensor device 100 and the shutter device 2110.The optical system 2100 guides image light (incident light) from an object toward the pixel section of the sensor device 100. This optical system 2100 may include a plurality of optical lenses. The shutter device 2110 controls a period in which the sensor device 100 is irradiated with the light and a period in which the light is blocked. The driver 2130 controls a transfer operation of the sensor device 100 and a shutter operation of the shutter device 2110. The signal processor 2120 performs various kinds of signal processing on signals outputted from the sensor device 100. An image signal Dout subjected to the signal processing is stored in a storage medium such as a memory or outputted to a monitor or the like.Further, the sensor device 100 described above is also applicable to electronic apparatuses (a capsule type endoscope 10100 and a mobile body such as a vehicle) described below.Further, the technology (the present technology) according to the present disclosure is applicable to a variety of products. For example, the technology according to the present disclosure may be applied to an endoscopic surgery system.Fig. 17 is a block diagram depicting an example of a schematic configuration of an in-vivo information acquisition system of a patient using a capsule type endoscope, to which the technology according to an embodiment of the present disclosure (present technology) can be applied.The in-vivo information acquisition system 10001 includes a capsule type endoscope 10100 and an external controlling apparatus 10200.The capsule type endoscope 10100 is swallowed by a patient at the time of inspection. The capsule type endoscope 10100 has an image pickup function and a wireless communication function and successively picks up an image of the inside of an organ such as the stomach or an intestine (hereinafter referred to as in-vivo image) at predetermined intervals while it moves inside of the organ by peristaltic motion for a period of time until it is naturally discharged from the patient. Then, the capsule type endoscope 10100 successively transmits information of the in-vivo image to the external controlling apparatus 10200 outside the body by wireless transmission.The external controlling apparatus 10200 integrally controls operation of the in-vivo information acquisition system 10001. Further, the external controlling apparatus 10200 receives information of an in-vivo image transmitted thereto from the capsule type endoscope 10100 and generates image data for displaying the in-vivo image on a display apparatus (not depicted) on the basis of the received information of the in-vivo image.In the in-vivo information acquisition system 10001, an in-vivo image imaged a state of the inside of the body of a patient can be acquired at any time in this manner for a period of time until the capsule typeendoscope 10100 is discharged after it is swallowed.A configuration and functions of the capsule type endoscope 10100 and the external controlling apparatus 10200 are described in more detail below.The capsule type endoscope 10100 includes a housing 10101 of the capsule type, in which a light source unit 10111, an image pickup unit 10112, an image processing unit 10113, a wireless communication unit 10114, a power feeding unit 10115, a power supply unit 10116 and a control unit 10117 are accommodated.The light source unit 10111 includes a light source such as, for example, a light emitting diode (LED) and irradiates light on an image pickup field-of-view of the image pickup unit 10112.The image pickup unit 10112 includes an image pickup element and an optical system including a plurality of lenses provided at a preceding stage to the image pickup element. Reflected light (hereinafter referred to as observation light) of light irradiated on a body tissue which is an observation target is condensed by the optical system and introduced into the image pickup element. In the image pickup unit 10112, the incident observation light is photoelectrically converted by the image pickup element, by which an image signal corresponding to the observation light is generated. The image signal generated by the image pickup unit 10112 is provided to the image processing unit 10113.The image processing unit 10113 includes a processor such as a central processing unit (CPU) or a graphics processing unit (GPU) and performs various signal processes for an image signal generated by the image pickup unit 10112. The image processing unit 10113 provides the image signal for which the signal processes have been performed thereby as RAW data to the wireless communication unit 10114.The wireless communication unit 10114 performs a predetermined process such as a modulation process for the image signal for which the signal processes have been performed by the image processing unit 10113 and transmits the resulting image signal to the external controlling apparatus 10200 through an antenna 10114A. Further, the wireless communication unit 10114 receives a control signal relating to driving control of the capsule type endoscope 10100 from the external controlling apparatus 10200 through the antenna 10114A. The wireless communication unit 10114 provides the control signal received from the external controlling apparatus 10200 to the control unit 10117.The power feeding unit 10115 includes an antenna coil for power reception, a power regeneration circuit for regenerating electric power from current generated in the antenna coil, a voltage booster circuit and so forth. The power feeding unit 10115 generates electric power using the principle of non-contact charging.The power supply unit 10116 includes a secondary battery and stores electric power generated by the power feeding unit 10115. In Fig. 17, in order to avoid complicated illustration, an arrow mark indicative of a supply destination of electric power from the power supply unit 10116 and so forth areomited. However, electric power stored in the power supply unit 10116 is supplied to and can be used to drive the light source unit 10111, the image pickup unit 10112, the image processing unit 10113, the wireless communication unit 10114 and the control unit 10117.The control unit 10117 includes a processor such as a CPU and suitably controls driving of the light source unit 10111, the image pickup unit 10112, the image processing unit 10113, the wireless communication unit 10114 and the power feeding unit 10115 in accordance with a control signal transmited thereto from the external controlling apparatus 10200.The external controlling apparatus 10200 includes a processor such as a CPU or a GPU, a microcomputer, a control board or the like in which a processor and a storage element such as a memory are mixedly incorporated. The external controlling apparatus 10200 transmits a control signal to the control unit 10117 of the capsule type endoscope 10100 through an antenna 10200A to control operation of the capsule type endoscope 10100. In the capsule type endoscope 10100, an irradiation condition of light upon an observation target of the light source unit 10111 can be changed, for example, in accordance with a control signal from the external controlling apparatus 10200. Further, an image pickup condition (for example, a frame rate, an exposure value or the like of the image pickup unit 10112) can be changed in accordance with a control signal from the external controlling apparatus 10200. Further, the substance of processing by the image processing unit 10113 or a condition for transmiting an image signal from the wireless communication unit 10114 (for example, a transmission interval, a transmission image number or the like) may be changed in accordance with a control signal from the external controlling apparatus 10200.Further, the external controlling apparatus 10200 performs various image processes for an image signal transmited thereto from the capsule type endoscope 10100 to generate image data for displaying a picked up in-vivo image on the display apparatus. As the image processes, various signal processes can be performed such as, for example, a development process (demosaic process), an image quality improving process (bandwidth enhancement process, a super-resolution process, a noise reduction (NR) process and / or image stabilization process) and / or an enlargement process (electronic zooming process). The external controlling apparatus 10200 controls driving of the display apparatus to cause the display apparatus to display a picked up in-vivo image on the basis of generated image data. Alternatively, the external controlling apparatus 10200 may also control a recording apparatus (not depicted) to record generated image data or control a printing apparatus (not depicted) to output generated image data by printing.The above has described the example of the in-vivo information acquisition system to which the technology according to the present disclosure may be applied. The technology according to the present disclosure may be applied, for example, to the image pickup unit 10112 among the components described above. This increases the detection accuracy.The technology (the present technology) according to the present disclosure is applicable to a variety of products. For example, the technology according to the present disclosure may be applied to anendoscopic surgery system.Fig. 18 is a view depicting an example of a schematic configuration of an endoscopic surgery system to which the technology according to an embodiment of the present disclosure (present technology) can be applied.In Fig. 18, a state is illustrated in which a surgeon (medical doctor) 11131 is using an endoscopic surgery system 11000 to perform surgery for a patient 11132 on a patient bed 11133. As depicted, the endoscopic surgery system 11000 includes an endoscope 11100, other surgical tools 11110 such as a pneumoperitoneum tube 11111 and an energy device 11112, a supporting arm apparatus 11120 which supports the endoscope 11100 thereon, and a cart 11200 on which various apparatus for endoscopic surgery are mounted.The endoscope 11100 includes a lens barrel 11101 having a region of a predetermined length from a distal end thereof to be inserted into a body cavity of the patient 11132, and a camera head 11102 connected to a proximal end of the lens barrel 11101. In the example depicted, the endoscope 11100 is depicted which includes as a rigid endoscope having the lens barrel 11101 of the hard type. However, the endoscope 11100 may otherwise be included as a flexible endoscope having the lens barrel 11101 of the flexible type.The lens barrel 11101 has, at a distal end thereof, an opening in which an objective lens is fitted. A light source apparatus 11203 is connected to the endoscope 11100 such that light generated by the light source apparatus 11203 is introduced to a distal end of the lens barrel 11101 by a light guide extending in the inside of the lens barrel 11101 and is irradiated toward an observation target in a body cavity of the patient 11132 through the objective lens. It is to be noted that the endoscope 11100 may be a forward-viewing endoscope or may be an oblique-viewing endoscope or a side-viewing endoscope.An optical system and an image pickup element are provided in the inside of the camera head 11102 such that reflected light (observation light) from the observation target is condensed on the image pickup element by the optical system. The observation light is photo-electrically converted by the image pickup element to generate an electric signal corresponding to the observation light, namely, an image signal corresponding to an observation image . The image signal is transmitted as RAW data to a CCU 11201.The CCU 11201 includes a central processing unit (CPU), a graphics processing unit (GPU) or the like and integrally controls operation of the endoscope 11100 and a display apparatus 11202. Further, the CCU 11201 receives an image signal from the camera head 11102 and performs, for the image signal, various image processes for displaying an image based on the image signal such as, for example, a development process (demosaic process).The display apparatus 11202 displays thereon an image based on an image signal, for which the image processes have been performed by the CCU 11201, under the control of the CCU 11201.The light source apparatus 11203 includes a light source such as, for example, a light emitting diode (LED) and supplies irradiation light upon imaging of a surgical region to the endoscope 11100.An inputting apparatus 11204 is an input interface for the endoscopic surgery system 11000. A user can perform inputting of various kinds of information or instruction inputting to the endoscopic surgery system 11000 through the inputting apparatus 11204. For example, the user would input an instruction or a like to change an image pickup condition (type of irradiation light, magnification, focal distance or the like) by the endoscope 11100.A treatment tool controlling apparatus 11205 controls driving of the energy device 11112 for cautery or incision of a tissue, sealing of a blood vessel or the like. A pneumoperitoneum apparatus 11206 feeds gas into a body cavity of the patient 11132 through the pneumoperitoneum tube 11111 to inflate the body cavity in order to secure the field of view of the endoscope 11100 and secure the working space for the surgeon. A recorder 11207 is an apparatus capable of recording various kinds of information relating to surgery. A printer 11208 is an apparatus capable of printing various kinds of information relating to surgery in various forms such as a text, an image or a graph.It is to be noted that the light source apparatus 11203 which supplies irradiation light when a surgical region is to be imaged to the endoscope 11100 may include a white light source which includes, for example, an LED, a laser light source or a combination of them. Where a white light source includes a combination of red, green, and blue (RGB) laser light sources, since the output intensity and the output timing can be controlled with a high degree of accuracy for each color (each wavelength), adjustment of the white balance of a picked up image can be performed by the light source apparatus 11203. Further, in this case, if laser beams from the respective RGB laser light sources are irradiated time-divisionally on an observation target and driving of the image pickup elements of the camera head 11102 are controlled in synchronism with the irradiation timings. Then images individually corresponding to the R, G and B colors can be also picked up time-divisionally. According to this method, a color image can be obtained even if color filters are not provided for the image pickup element.Further, the light source apparatus 11203 may be controlled such that the intensity of light to be outputted is changed for each predetermined time. By controlling driving of the image pickup element of the camera head 11102 in synchronism with the timing of the change of the intensity of light to acquire images time-divisionally and synthesizing the images, an image of a high dynamic range free from underexposed blocked up shadows and overexposed highlights can be created.Further, the light source apparatus 11203 may be configured to supply light of a predetermined wavelength band ready for special light observation. In special light observation, for example, by utilizing the wavelength dependency of absorption of light in a body tissue to irradiate light of a narrow band in comparison with irradiation light upon ordinary observation (namely, white light), narrow band observation (narrow band imaging) of imaging a predetermined tissue such as a blood vessel of a superficial portion of the mucous membrane or the like in a high contrast is performed. Alternatively, in special light observation, fluorescent observation for obtaining an image from fluorescent light generatedby irradiation of excitation light may be performed. In fluorescent observation, it is possible to perform observation of fluorescent light from a body tissue by irradiating excitation light on the body tissue (autofluorescence observation) or to obtain a fluorescent light image by locally injecting a reagent such as indocyanine green (ICG) into a body tissue and irradiating excitation light corresponding to a fluorescent light wavelength of the reagent upon the body tissue. The light source apparatus 11203 can be configured to supply such narrow-band light and / or excitation light suitable for special light observation as described above.Fig. 19 is a block diagram depicting an example of a functional configuration of the camera head 11102 and the CCU 11201 depicted in Fig. 18.The camera head 11102 includes a lens unit 11401, an image pickup unit 11402, a driving unit 11403, a communication unit 11404 and a camera head controlling unit 11405. The CCU 11201 includes a communication unit 11411, an image processing unit 11412 and a control unit 11413. The camera head 11102 and the CCU 11201 are connected for communication to each other by a transmission cable 11400.The lens unit 11401 is an optical system, provided at a connecting location to the lens barrel 11101. Observation light taken in from a distal end of the lens barrel 11101 is guided to the camera head 11102 and introduced into the lens unit 11401. The lens unit 11401 includes a combination of a plurality of lenses including a zoom lens and a focusing lens.The number of image pickup elements which is included by the image pickup unit 11402 may be one (single-plate type) or a plural number (multi -plate type). Where the image pickup unit 11402 is configured as that of the multi-plate type, for example, image signals corresponding to respective R, G and B are generated by the image pickup elements, and the image signals may be synthesized to obtain a color image. The image pickup unit 11402 may also be configured so as to have a pair of image pickup elements for acquiring respective image signals for the right eye and the left eye ready for three dimensional (3D) display. If 3D display is performed, then the depth of a living body tissue in a surgical region can be comprehended more accurately by the surgeon 11131. It is to be noted that, where the image pickup unit 11402 is configured as that of stereoscopic type, a plurality of systems of lens units 11401 are provided corresponding to the individual image pickup elements.Further, the image pickup unit 11402 may not necessarily be provided on the camera head 11102. For example, the image pickup unit 11402 may be provided immediately behind the objective lens in the inside of the lens barrel 11101.The driving unit 11403 includes an actuator and moves the zoom lens and the focusing lens of the lens unit 11401 by a predetermined distance along an optical axis under the control of the camera head controlling unit 11405. Consequently, the magnification and the focal point of a picked up image by the image pickup unit 11402 can be adjusted suitably.The communication unit 11404 includes a communication apparatus for transmitting and receivingvarious kinds of information to and from the CCU 11201. The communication unit 11404 transmits an image signal acquired from the image pickup unit 11402 as RAW data to the CCU 11201 through the transmission cable 11400.In addition, the communication unit 11404 receives a control signal for controlling driving of the camera head 11102 from the CCU 11201 and supplies the control signal to the camera head controlling unit 11405. The control signal includes information relating to image pickup conditions such as, for example, information that a frame rate of a picked up image is designated, information that an exposure value upon image picking up is designated and / or information that a magnification and a focal point of a picked up image are designated.It is to be noted that the image pickup conditions such as the frame rate, exposure value, magnification or focal point may be designated by the user or may be set automatically by the control unit 11413 of the CCU 11201 on the basis of an acquired image signal. In the latter case, an auto exposure (AE) function, an auto focus (AF) function and an auto white balance (AWB) function are incorporated in the endoscope 11100.The camera head controlling unit 11405 controls driving of the camera head 11102 on the basis of a control signal from the CCU 11201 received through the communication unit 11404.The communication unit 11411 includes a communication apparatus for transmitting and receiving various kinds of information to and from the camera head 11102. The communication unit 11411 receives an image signal transmitted thereto from the camera head 11102 through the transmission cable 11400.Further, the communication unit 11411 transmits a control signal for controlling driving of the camera head 11102 to the camera head 11102. The image signal and the control signal can be transmitted by electrical communication, optical communication or the like.The image processing unit 11412 performs various image processes for an image signal in the form of RAW data transmitted thereto from the camera head 11102.The control unit 11413 performs various kinds of control relating to image picking up of a surgical region or the like by the endoscope 11100 and display of a picked up image obtained by image picking up of the surgical region or the like. For example, the control unit 11413 creates a control signal for controlling driving of the camera head 11102.Further, the control unit 11413 controls, on the basis of an image signal for which image processes have been performed by the image processing unit 11412, the display apparatus 11202 to display a picked up image in which the surgical region or the like is imaged. Thereupon, the control unit 11413 may recognize various objects in the picked up image using various image recognition technologies. For example, the control unit 11413 can recognize a surgical tool such as forceps, a particular living bodyregion, bleeding, mist when the energy device 11112 is used and so forth by detecting the shape, color and so forth of edges of objects included in a picked up image. The control unit 11413 may cause, when it controls the display apparatus 11202 to display a picked up image, various kinds of surgery supporting information to be displayed in an overlapping manner with an image of the surgical region using a result of the recognition. Where surgery supporting information is displayed in an overlapping manner and presented to the surgeon 11131, the burden on the surgeon 11131 can be reduced and the surgeon 11131 can proceed with the surgery with certainty.The transmission cable 11400 which connects the camera head 11102 and the CCU 11201 to each other is an electric signal cable ready for communication of an electric signal, an optical fiber ready for optical communication or a composite cable ready for both of electrical and optical communications.Here, while, in the example depicted, communication is performed by wired communication using the transmission cable 11400, the communication between the camera head 11102 and the CCU 11201 may be performed by wireless communication.The above has described the example of the endoscopic surgery system to which the technology according to the present disclosure may be applied. The technology according to the present disclosure may be applied to the image pickup unit 11402 among the components described above. The application of the technology according to the present disclosure to the image pickup unit 11402 increases the detection accuracy.It is to be noted that the endoscopic surgery system has been described here as an example, but the technology according to the present disclosure may be additionally applied, for example, to a microscopic surgery system or the like.The technology according to the present disclosure is applicable to a variety of products. For example, the technology according to the present disclosure may be achieved as a device mounted on any type of mobile body such as a vehicle, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility, an airplane, a drone, a vessel, a robot, a construction machine, or an agricultural machine (tractor).Fig. 20 is a block diagram depicting an example of schematic configuration of a vehicle control system as an example of a mobile body control system to which the technology according to an embodiment of the present disclosure can be applied.The vehicle control system 12000 includes a plurality of electronic control units connected to each other via a communication network 12001. In the example depicted in Fig. 20, the vehicle control system 12000 includes a driving system control unit 12010, a body system control unit 12020, an outside -vehicle information detecting unit 12030, an in-vehicle information detecting unit 12040, and an integrated control unit 12050. In addition, a microcomputer 12051, a sound / image output section 12052, and a vehicle-mounted network interface (I / F) 12053 are illustrated as a functional configuration of theintegrated control unit 12050.The driving system control unit 12010 controls the operation of devices related to the driving system of the vehicle in accordance with various kinds of programs. For example, the driving system control unit 12010 functions as a control device for a driving force generating device for generating the driving force of the vehicle, such as an internal combustion engine, a driving motor, or the like, a driving force transmitting mechanism for transmitting the driving force to wheels, a steering mechanism for adjusting the steering angle of the vehicle, a braking device for generating the braking force of the vehicle, and the like.The body system control unit 12020 controls the operation of various kinds of devices provided to a vehicle body in accordance with various kinds of programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various kinds of lamps such as a headlamp, a backup lamp, a brake lamp, a turn signal, a fog lamp, or the like. In this case, radio waves transmitted from a mobile device as an alternative to a key or signals of various kinds of switches can be input to the body system control unit 12020. The body system control unit 12020 receives these input radio waves or signals, and controls a door lock device, the power window device, the lamps, or the like of the vehicle.The outside-vehicle information detecting unit 12030 detects information about the outside of the vehicle including the vehicle control system 12000. For example, the outside-vehicle information detecting unit 12030 is connected with an imaging section 12031. The outside-vehicle information detecting unit 12030 makes the imaging section 12031 image an image of the outside of the vehicle, and receives the imaged image. On the basis of the received image, the outside-vehicle information detecting unit 12030 may perform processing of detecting an object such as a human, a vehicle, an obstacle, a sign, a character on a road surface, or the like, or processing of detecting a distance thereto.The imaging section 12031 is an optical sensor that receives light, and which outputs an electric signal corresponding to a received light amount of the light. The imaging section 12031 can output the electric signal as an image, or can output the electric signal as information about a measured distance. In addition, the light received by the imaging section 12031 may be visible light, or may be invisible light such as infrared rays or the like.The in-vehicle information detecting unit 12040 detects information about the inside of the vehicle. The in-vehicle information detecting unit 12040 is, for example, connected with a driver state detecting section 12041 that detects the state of a driver. The driver state detecting section 12041, for example, includes a camera that images the driver. On the basis of detection information input from the driver state detecting section 12041, the in-vehicle information detecting unit 12040 may calculate a degree of fatigue of the driver or a degree of concentration of the driver, or may determine whether the driver is dozing.The microcomputer 12051 can calculate a control target value for the driving force generating device, thesteering mechanism, or the braking device on the basis of the information about the inside or outside of the vehicle which information is obtained by the outside-vehicle information detecting unit 12030 or the in-vehicle information detecting unit 12040, and output a control command to the driving system control unit 12010. For example, the microcomputer 12051 can perform cooperative control intended to implement functions of an advanced driver assistance system (ADAS) which functions include collision avoidance or shock mitigation for the vehicle, following driving based on a following distance, vehicle speed maintaining driving, a warning of collision of the vehicle, a warning of deviation of the vehicle from a lane, or the like.In addition, the microcomputer 12051 can perform cooperative control intended for automatic driving, which makes the vehicle to travel autonomously without depending on the operation of the driver, or the like, by controlling the driving force generating device, the steering mechanism, the braking device, or the like on the basis of the information about the outside or inside of the vehicle which information is obtained by the outside-vehicle information detecting unit 12030 or the in-vehicle information detecting unit 12040.In addition, the microcomputer 12051 can output a control command to the body system control unit 12020 on the basis of the information about the outside of the vehicle which information is obtained by the outside-vehicle information detecting unit 12030. For example, the microcomputer 12051 can perform cooperative control intended to prevent a glare by controlling the headlamp so as to change from a high beam to a low beam, for example, in accordance with the position of a preceding vehicle or an oncoming vehicle detected by the outside-vehicle information detecting unit 12030.The sound / image output section 12052 transmits an output signal of at least one of a sound and an image to an output device capable of visually or auditorily notifying information to an occupant of the vehicle or the outside of the vehicle. In the example of Fig. 20, an audio speaker 12061, a display section 12062, and an instrument panel 12063 are illustrated as the output device. The display section 12062 may, for example, include at least one of an on-board display and a head-up display.Fig. 21 is a diagram depicting an example of the installation position of the imaging section 12031.In Fig. 21, the imaging section 12031 includes imaging sections 12101, 12102, 12103, 12104, and 12105.The imaging sections 12101, 12102, 12103, 12104, and 12105 are, for example, disposed at positions on a front nose, sideview mirrors, a rear bumper, and a back door of the vehicle 12100 as well as a position on an upper portion of a windshield within the interior of the vehicle. The imaging section 12101 provided to the front nose and the imaging section 12105 provided to the upper portion of the windshield within the interior of the vehicle obtain mainly an image of the front of the vehicle 12100. The imaging sections 12102 and 12103 provided to the sideview mirrors obtain mainly an image of the sides of the vehicle 12100. The imaging section 12104 provided to the rear bumper or the back door obtains mainly an image of the rear of the vehicle 12100. The imaging section 12105 provided to the upper portion of the windshield within the interior of the vehicle is used mainly to detect a preceding vehicle, a pedestrian,an obstacle, a signal, a traffic sign, a lane, or the like.Incidentally, Fig. 21 depicts an example of photographing ranges of the imaging sections 12101 to 12104. An imaging range 12111 represents the imaging range of the imaging section 12101 provided to the front nose. Imaging ranges 12112 and 12113 respectively represent the imaging ranges of the imaging sections 12102 and 12103 provided to the sideview mirrors. An imaging range 12114 represents the imaging range of the imaging section 12104 provided to the rear bumper or the back door. A bird’s-eye image of the vehicle 12100 as viewed from above is obtained by superimposing image data imaged by the imaging sections 12101 to 12104, for example.At least one of the imaging sections 12101 to 12104 may have a function of obtaining distance information. For example, at least one of the imaging sections 12101 to 12104 may be a stereo camera constituted of a plurality of imaging elements, or may be an imaging element having pixels for phase difference detection.For example, the microcomputer 12051 can determine a distance to each three-dimensional object within the imaging ranges 12111 to 12114 and a temporal change in the distance (relative speed with respect to the vehicle 12100) on the basis of the distance information obtained from the imaging sections 12101 to 12104, and thereby extract, as a preceding vehicle, a nearest three-dimensional object in particular that is present on a traveling path of the vehicle 12100 and which travels in substantially the same direction as the vehicle 12100 at a predetermined speed (for example, equal to or more than 0 km / hour). Further, the microcomputer 12051 can set a following distance to be maintained in front of a preceding vehicle in advance, and perform automatic brake control (including following stop control), automatic acceleration control (including following start control), or the like. It is thus possible to perform cooperative control intended for automatic driving that makes the vehicle travel autonomously without depending on the operation of the driver or the like.For example, the microcomputer 12051 can classify three-dimensional object data on three-dimensional objects into three-dimensional object data of a two-wheeled vehicle, a standard-sized vehicle, a large-sized vehicle, a pedestrian, a utility pole, and other three-dimensional objects on the basis of the distance information obtained from the imaging sections 12101 to 12104, extract the classified three-dimensional object data, and use the extracted three-dimensional object data for automatic avoidance of an obstacle. For example, the microcomputer 12051 identifies obstacles around the vehicle 12100 as obstacles that the driver of the vehicle 12100 can recognize visually and obstacles that are difficult for the driver of the vehicle 12100 to recognize visually. Then, the microcomputer 12051 determines a collision risk indicating a risk of collision with each obstacle. In a situation in which the collision risk is equal to or higher than a set value and there is thus a possibility of collision, the microcomputer 12051 outputs a warning to the driver via the audio speaker 12061 or the display section 12062, and performs forced deceleration or avoidance steering via the driving system control unit 12010. The microcomputer 12051 can thereby assist in driving to avoid collision.At least one of the imaging sections 12101 to 12104 may be an infrared camera that detects infrared rays.The microcomputer 12051 can, for example, recognize a pedestrian by determining whether or not there is a pedestrian in imaged images of the imaging sections 12101 to 12104. Such recognition of a pedestrian is, for example, performed by a procedure of extracting characteristic points in the imaged images of the imaging sections 12101 to 12104 as infrared cameras and a procedure of determining whether or not it is the pedestrian by performing pattern matching processing on a series of characteristic points representing the contour of the object. When the microcomputer 12051 determines that there is a pedestrian in the imaged images of the imaging sections 12101 to 12104, and thus recognizes the pedestrian, the sound / image output section 12052 controls the display section 12062 so that a square contour line for emphasis is displayed so as to be superimposed on the recognized pedestrian. The sound / image output section 12052 may also control the display section 12062 so that an icon or the like representing the pedestrian is displayed at a desired position.The description has been given with reference to the first to fourth embodiments, the modification examples 1 to 3, the application examples, and the practical application examples, but the contents of the present disclosure are not limited to the embodiments or the like described above. It is possible to make a variety of modifications.It is to be noted that the effects described herein are merely examples, but not limitative. In addition, there may be other effects.Note that, the present technology can also take the following configurations.[1] A sensor device (100) for capturing images of a scene, the sensor device comprising a sensor substrate (110) having a light receiving surface (112) on which a plurality of light receiving elements (114) is arranged; sensor optics (120) that is configured to guide light from the scene onto the light receiving substrate (112); and a control unit (130) that is configured to control operation of the sensor device (100); wherein when the scene comprises a calibration chart (200) that shows a texture (T) with known geometrical features arranged with a predetermined pitch (P) that lead to a moire effect when imaged by the plurality of light receiving elements (114), the control unit (130) is configured to convert the image (S) showing the moire effect such that it shows the texture (T) of the calibration chart (200) without distortion, and to determine imaging properties of the sensor optics (120) based on the converted image (M).[2] The sensor device (100) according to [1], wherein the control unit (130) is configured to convert the image (S) showing the moire effect by the coordinate transformation<|>M= 4>T- <|>s=where<t>Tare normalized texture coordinates of the texture (T) of the calibration chart (200) that are normalized to the pitch (P) of the texture (T), <j>sare normalized sensor coordinates that are normalized to a pitch (D) of the light receiving elements (114) and are the coordinates of the converted image (M)[3] The sensor device (100) according to any one of [1] and [2], wherein the texture (T) of the calibration chart (200) is a two-dimensional array of dots of known size and a known pitch (P) that differs from the pitch (D) of the light receiving elements (114) such that a moire effect is generated during imaging, preferably such that the projection of the pitch (P) of the texture (T) onto the light receiving surface (112) differs from the pitch (D) of the light receiving elements (114) by 10%, 5%, 2% or 1%; the converted image (M) shows the dots with a blur caused by the sensor optics (120); and the control unit (130) is configured to determine the blur caused by the sensor optics from the converted image (M).[4] The sensor device (100) according to [3], wherein the control unit (130) is configured to determine for each dot in the converted image (M) the point spread function, PSF, of the light receiving elements (114) at the position of the sensor substrate (110) corresponding to the respective dot in the converted image (M).[5] The sensor device (100) according to [3] or [4], wherein the control unit is configured to determine for each dot in the converted image (M) at least one modulation transfer function, MTF, along a line through the respective dot.[6] The sensor device (100) according to any one of [1] to [5], wherein the control unit (130) is configured to generate data that allow visualizing, when rendered on a display (300), a correspondence of points on the image (S) showing the moire effect and the converted image (M).[7] The sensor device (100) according to any one of [3] to [6], wherein the control unit (130) is configured to generate data that allow visualizing, when rendered on a display (300), a correspondence of points on the image (S) showing the moire effect and the converted image (M), by indicating for each dot in the image (S) showing the moire effect and / or each dot in the converted image (M) the blur caused by the sensor optics (120) for the respective dot.[8] A calibration system (10) comprising the sensor device (100) according to any one of [1] to [7]; and the calibration chart (200).[9] The calibration system (10) according to [8], wherein the calibration chart (200) comprises a two-dimensional marker (210) that indicates information including at least one of a position of the marker (210) on the calibration chart (200), the geometrical features of the texture (T), and the pitch (P).

[0010] The calibration system (10) according to [8] or [9], whereinthe control unit (130) is configured to initially deduce coordinates on the calibration chart (200) and a mapping from the calibration chart (200) to the light receiving surface (112) from the information indicated by the marker (210), to convert the image (S) showing the moire effect based on said mapping, and to refine the mapping based on the detection of intensity peaks in the converted image (M).

[0011] The calibration system (10) according to any one of [8] to

[0010] , wherein the calibration chart (200) comprises different sections (220) with different textures.

[0012] The calibration system (10) according to any one of [8] to

[0011] , wherein the texture (T) comprises a two-dimensional array of dots that resemble the appearance of a regular two-dimensional array (R) of dots that is viewed under a slanting angle.

[0013] The calibration system (10) according to any one of [8] to

[0012] , further comprising a display (300) that is configured to display the calibration chart (200) on it; wherein the control unit (130) is configured to use display parameters of the display (300) to convert the image (S) showing the moire effect, the display parameters including at least one of a shape of pixels (310) of the display (300), an arrangement of the pixels (310) of the display (300), a distortion of the pixels (310) of the display, a refractive index of a cover glass of the display (300), and a thickness of the cover glass of the display (300).

[0014] The calibration system (10) according to

[0013] , wherein the display (300) comprises subpixels (312) of different colors; at different times the display (300) is configured to display the calibration chart (200) by using different colors of the display (300); and the control unit (130) is configured to take into account the positions of the subpixels (312) of different color of the display (300), when converting the image (S) showing the moire for the different colors.

[0015] A method for operating a sensor device (100) according to any one of [1] to [7] or for operating a calibration system (10) according to any one of [8] to

[0014] , the method comprising capturing an image (S) of the scene comprising a calibration chart (200) that shows a texture (T) with known geometrical features arranged with a predetermined pitch (P) that lead to a moire effect when imaged by the plurality of light receiving elements (114); converting, by the control unit (130), the image (S) showing the moire effect such that it shows the texture (T) of the calibration chart (200) without distortion; and determining, by the control unit (130), imaging properties of the sensor optics (120) based on the converted image (M).

Claims

CLAIMS1. A sensor device for capturing images of a scene, the sensor device comprising a sensor substrate having a light receiving surface on which a plurality of light receiving elements is arranged; sensor optics that is configured to guide light from the scene onto the light receiving substrate; and a control unit that is configured to control operation of the sensor device; wherein when the scene comprises a calibration chart that shows a texture with known geometrical features arranged with a predetermined pitch that lead to a moire effect when imaged by the plurality of light receiving elements, the control unit is configured to convert the image showing the moire effect such that it shows the texture of the calibration chart without distortion, and to determine imaging properties of the sensor optics based on the converted image.

2. The sensor device according to claim 1, wherein the control unit is configured to convert the image showing the moire effect by the coordinate transformation<|>M= 4>T- cf>s=where(|)Tare normalized texture coordinates of the texture of the calibration chart that are normalized to the pitch of the texture, <|>sare normalized sensor coordinates that are normalized to a pitch of the light receiving elements and (|)Iare the coordinates of the converted image3. The sensor device according to claim 1, wherein the texture of the calibration chart is a two-dimensional array of dots of known size and a known pitch that differs from the pitch of the light receiving elements such that a moire effect is generated during imaging, preferably such that the projection of the pitch of the texture onto the light receiving surface differs from the pitch of the light receiving elements by 10%, 5%, 2% or 1%; the converted image shows the dots with a blur caused by the sensor optics; and the control unit is configured to determine the blur caused by the sensor optics from the converted image.

4. The sensor device according to claim 3, wherein the control unit is configured to determine for each dot in the converted image the point spread function, PSF, of the light receiving elements at the position of the sensor substrate corresponding to the respective dot in the converted image.

5. The sensor device according to claim 3, wherein the control unit is configured to determine for each dot in the converted image at least one modulation transfer function, MTF, along a line through the respective dot.

6. The sensor device according to claim 1, whereinthe control unit is configured to generate data that allow visualizing, when rendered on a display, a correspondence of points on the image showing the moire effect and the converted image.

7. The sensor device according to claim 3, wherein the control unit is configured to generate data that allow visualizing, when rendered on a display, a correspondence of points on the image showing the moire effect and the converted image, by indicating for each dot in the image showing the moire effect and / or each dot in the converted image the blur caused by the sensor optics for the respective dot.

8. A calibration system comprising the sensor device according to claim 1 ; and the calibration chart.

9. The calibration system according to claim 8, wherein the calibration chart comprises a two-dimensional marker that indicates information including at least one of a position of the marker on the calibration chart, the geometrical features of the texture, and the pitch.

10. The calibration system according to claim 9, wherein the control unit is configured to initially deduce coordinates on the calibration chart and a mapping from the calibration chart to the light receiving surface from the information indicated by the marker, to convert the image showing the moire effect based on said mapping, and to refine the mapping based on the detection of intensity peaks in the converted image.

11. The calibration system according to claim 8, wherein the calibration chart comprises different sections with different textures.

12. The calibration system according to claim 8, wherein the texture comprises a two-dimensional array of dots that resemble the appearance of a regular two-dimensional array of dots that is viewed under a slanting angle.

13. The calibration system according to claim 8, further comprising a display that is configured to display the calibration chart on it; wherein the control unit is configured to use display parameters of the display to convert the image showing the moire effect, the display parameters including at least one of a shape of pixels of the display, an arrangement of the pixels of the display, a distortion of the pixels of the display, a refractive index of a cover glass of the display, and a thickness of the cover glass of the display.

14. The calibration system according to claim 13, wherein the display comprises subpixels of different colors; at different times the display is configured to display the calibration chart by using different colors of the display; andthe control unit is configured to take into account the positions of the subpixels of different color of the display, when converting the image showing the moire effect for the different colors.

15. A method for operating a sensor device according to claim 1, the method comprising capturing an image of the scene comprising a calibration chart that shows a texture with known geometrical features arranged with a predetermined pitch that lead to a moire effect when imaged by the plurality of light receiving elements; converting, by the control unit, the image showing the moire effect such that it shows the texture of the calibration chart without distortion; and determining, by the control unit, imaging properties of the sensor optics based on the converted image.

Images