Imaging device, inspection device, and imaging method

Abstract

To provide an imaging apparatus configured to acquire a multispectral image of a subject in a simple configuration, and provide an inspection apparatus, and an imaging method.SOLUTION: An imaging apparatus 11 includes a color image sensor 31 having a sensor unit 31S, N types (N≥2) of light sources 21-23 having different light emitting characteristics, a control unit 40, an adjacent pixel mixing processing unit 51, a resampling processing unit 52, and a rearrangement processing unit 53. The control unit 40 causes M types (2≤M≤N) of the light sources 21-23 to emit light in a time division manner to cause the color image sensor 31 to perform M image capturing operations. The adjacent pixel mixing processing unit 51 performs adjacent pixel mixing processing, by frame, on M frames output from the sensor unit 31S. The resampling processing unit 52 performs resampling processing, by frame, to shift a central position of a pixel, on frames obtained after the adjacent pixel mixing processing. The rearrangement processing unit 53 rearranges pixel data for resampled multiple frames into pixels in one frame.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an imaging device, an inspection device, and an imaging method for imaging a subject.

Background Art

[0002] In a manufacturing site such as a factory, an imaging device equipped with a camera may be used in a subsequent process including a manufacturing process or an inspection process. Usually, an imaging device equipped with a monochrome camera or a color camera is used. In this case, a subject is imaged using a monochrome camera or a color camera, and a predetermined process is performed on the subject or the subject is inspected based on the obtained monochrome image or color image. However, in addition to monochrome images and color images, when a pseudo-color image or a near-infrared image obtained by irradiating a subject with light of a specific frequency is used, specific features of the subject that cannot be obtained in monochrome images and color images can be observed. Therefore, a pseudo-color image or a near-infrared image of the subject is used according to the purpose of imaging.

[0003] For example, Patent Documents 1 to 3 disclose an imaging element or a camera module capable of imaging both visible light and near-infrared light. Patent Document 1 discloses an imaging device capable of acquiring a color image, an infrared light image, or a mixed image obtained by synthesizing both. The imaging unit of this imaging device includes a color filter group including three filters for visible light and one filter for near-infrared light, and a solid-state imaging device including three pixels for detecting visible light and one pixel for detecting near-infrared light. Further, Patent Document 2 discloses a solid-state imaging device in which a pixel sensor for detecting a color image of a document to be read and a pixel sensor for detecting near-infrared light recorded as invisible information are formed on the same substrate. Furthermore, Patent Document 3 discloses a camera module having a structure in which a filter unit having an IR cut characteristic and also serving as a diaphragm is disposed on the front surface of a lens.

[0004] In addition, an imaging device equipped with a multispectral camera may also be used. With a multispectral camera, black-and-white images, color images, pseudo-color images, and near-infrared images can be obtained. Also, with a multispectral camera, the optical frequency when acquiring pseudo-color images and near-infrared images can be adjusted as well.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, depending on the purpose of imaging, black-and-white images, color images, pseudo-color images, and near-infrared images are required. Black-and-white images and color images can be acquired by imaging with a single general-purpose color camera. Also, by using the special solid-state imaging devices (image sensors) or special camera modules described in Patent Documents 1 to 3, color images and near-infrared images can be obtained. However, since they are special imaging devices, the imaging device including the camera becomes expensive. In addition, an imaging device equipped with a multispectral camera has a complex structure and becomes even more expensive. For this reason, for example, for the purpose of acquiring an image of a subject in a manufacturing process or inspection process at a manufacturing site such as a factory, the imaging device becomes expensive, which causes an increase in the manufacturing cost of the product.

[0007] In addition, when a pseudo-color image or a near-infrared image captured in a pseudo-color other than a color image is required, a camera including a special solid-state imaging device or a special camera module is required instead of or in addition to a general-purpose color camera. For this reason, an imaging device equipped with a plurality of cameras is required. Therefore, there is a demand for a relatively inexpensive imaging device that can acquire multi-spectral images such as pseudo-color images or near-infrared images other than color images using a single inexpensive camera such as a general-purpose color camera.

Means for Solving the Problem

[0008] Hereinafter, means for solving the above problems and their effects will be described. The imaging device for solving the above problems is an imaging device that images a subject, and includes a color image sensor having a sensor unit that images the subject, N types of light sources having different light emission characteristics for illuminating the subject (where N is a natural number of 2 or more), and a control unit that causes the color image sensor to perform imaging M times by causing M types (where M is a natural number satisfying 2 ≤ M ≤ N) of the N types of light sources to emit light in a time-division manner, an adjacent pixel mixing processing unit that performs adjacent pixel mixing processing for each frame on the M frames output from the sensor unit, a resampling processing unit that performs resampling processing for shifting the center position of pixels for each frame on the frame after the adjacent pixel mixing processing, and a rearrangement processing unit that rearranges the pixel data for a plurality of resampled frames into pixels within one frame.

[0009] According to this configuration, a multi-spectral image can be acquired by imaging a subject using one general-purpose color image sensor. Therefore, a multi-spectral image of the subject can be acquired with a simple configuration.

[0010] In the above imaging device, a signal processing unit is provided for processing an imaging signal corresponding to the frame output from the sensor unit of the color image sensor. The adjacent pixel mixing processing unit is provided at least in the signal processing unit among the color image sensor and the signal processing unit. For each of the plurality of frames, either one or both of the color image sensor and the signal processing unit are selected, and the adjacent pixel mixing processing unit provided in the selected one or both performs the adjacent pixel mixing processing for each frame. In the adjacent pixel mixing processing, adjacent pixels to be mixed may be selected for each frame.

[0011] According to this configuration, the time required for the color image sensor to output an imaging signal is shortened, enabling high-speed processing. Also, even when there are limitations in the adjacent pixel mixing processing function of the color image sensor or when it does not have the function, adjacent pixel mixing processing can be performed, and resampling processing and rearrangement processing for reducing deterioration of resolution characteristics can be implemented.

[0012] In the above imaging device, the pixel arrangement of the color filter constituting the color image sensor is a Bayer array, and the rearrangement processing unit may rearrange the pixel data for the plurality of frames in a 3-color 4-pixel array same as the Bayer array within one frame.

[0013] According to this configuration, for a color camera having a color image sensor that employs a general-purpose pixel arrangement such as a Bayer array, a general-purpose pixel arrangement conversion unit (for example, a Bayer array conversion unit) for converting an imaging signal (frame) output from the color camera into a plurality of color-separated images is provided. Therefore, by rearranging the pixel data for a plurality of frames in a 3-color 4-pixel array same as the Bayer array within one frame to generate one-frame imaging data, an existing general-purpose pixel arrangement conversion unit can be used when converting this one-frame imaging data into a plurality of color-separated images.

[0014] In the above imaging device, the pixel arrangement of the color filter constituting the color image sensor is a Bayer array, and the control unit sequentially illuminates the subject with light from the light sources having different emission characteristics, causing the color image sensor to perform imaging a plurality of times in a time-division manner. For the plurality of frames, the rearrangement processing unit performs rearrangement processing of rearranging the pixel data for four frames obtained by performing the adjacent pixel mixing process and the resampling process for each frame into a four-color four-pixel array with the same four pixels as the Bayer array as a unit within one frame, thereby generating one-frame imaging data of four bands.

[0015] According to this configuration, it is possible to acquire one-frame imaging data of four bands, which is a larger number of bands than the number of colors of the Bayer array, which is "3". In the above imaging device, the N types of light sources include a light source capable of irradiating visible light and a light source capable of irradiating near-infrared light. The control unit selects the M types of light sources corresponding to the mode among the N types of light sources, and causes the selected M types of light sources to emit light in a time-division manner, causing the color image sensor to perform imaging a plurality of times in a time-division manner. The rearrangement processing unit outputs one-frame imaging data of a type corresponding to the selected mode. The mode may include at least one of a mode of outputting one-frame imaging data of three bands or four bands including one or more bands of pseudo-color pixel data having wavelengths within the light wavelength region from visible light to near-infrared light, and a mode of outputting one-frame imaging data of three bands or four bands including pixel data of at least one color among RGB pixel data and near-infrared light pixel data.

[0016] According to this configuration, the imaging device can output at least one of the 1-frame imaging data of 3 or 4 bands including at least 1 band of pseudo-color and the 1-frame imaging data of 3 or 4 bands including pixel data of at least one color among RGB and near-infrared light pixel data. Using an imaging device equipped with a single general-purpose color camera incorporating a single general-purpose color image sensor, a plurality of (multiple bands) images with different optical characteristics can be acquired from the 1-frame imaging data of 3 or 4 bands including at least 1 band of pseudo-color or near-infrared light band.

[0017] In the above imaging device, the N types of light sources include a visible light source and a near-infrared light source capable of covering the entire sensitivity wavelength range of the color image sensor. The control unit selects the M types of light sources corresponding to the mode among the N types of light sources, and causes the selected M types of light sources to emit light in a time-division manner to cause the color image sensor to perform imaging in a time-division manner and output a plurality of frames. The rearrangement processing unit may perform correction to reduce the influence of the afterglow of the light source during imaging of the previous frame on the next frame due to the entry of the afterglow of the light source during imaging of the previous frame into the next frame by arithmetic processing between frames.

[0018] According to this configuration, the influence of the illumination of the previous frame on the next frame can be reduced, and an image with high wavelength separation accuracy without mixing of emission wavelengths can be output. The inspection device for solving the above problems includes the above imaging device and an inspection processing unit for inspecting the subject based on the image output by the imaging device.

[0019] According to this configuration, various inspections can be performed on the subject using a plurality of types of captured images. The imaging method for solving the above problems is an imaging method for imaging a subject, including an illumination step of sequentially emitting a plurality of light sources with different light emission characteristics to illuminate the subject in a time-division manner, an imaging step of imaging the subject illuminated in a time-division manner with a color image sensor in a time-division manner to obtain a plurality of frames, an adjacent pixel mixing process step of performing an adjacent pixel mixing process for each of the plurality of frames, a resampling process step of performing a resampling process for each of the frames to shift the center position of the pixels for the frames after the adjacent pixel mixing process, and a rearrangement process step of rearranging the pixel data for the plurality of resampled frames into the pixels within one frame.

[0020] According to this imaging method, a multi-spectral image of a subject can be obtained with a simple configuration.

Effect of the Invention

[0021] According to the present invention, a multi-spectral image can be obtained by imaging a subject using one general-purpose color image sensor.

Brief Description of the Drawings

[0022]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Embodiments for Carrying Out the Invention

[0023] (First Embodiment) Hereinafter, an imaging device and an inspection device including the imaging device will be described with reference to the drawings. <Schematic Configuration of Imaging Device> The imaging device 11 shown in FIG. 1 is a device that images a subject 12 and outputs an image. The imaging device 11 includes an illumination unit 20 that illuminates the subject 12, a color image sensor 31 (hereinafter also simply referred to as "image sensor 31") that images the subject 12, and a lens 32 that condenses light from the subject 12. Further, the imaging device 11 includes a control unit 40 that controls the illumination unit 20, the image sensor 31, etc., and an image processing unit 50 that performs image processing on the imaging signal S1 output from the image sensor 31. The illumination unit 20 includes N types (where N is a natural number of 2 or more) of light sources 21 to 23 having different light emission characteristics. In FIG. 1, as an example, an example where N types are 3 types (N = 3) is shown, but N may be any natural number of 2 or more, and N = 2 or N = 4 may also be used.

[0024] The control unit 40 causes M types (where M is a natural number satisfying 2 ≤ M ≤ N) of the N types of light sources 21 to 23 to emit light in a time-division manner, so that the color image sensor 31 images the subject 12 irradiated with M types of different lights in a time-division manner M times. The image sensor 31 outputs imaging signals S1 corresponding to M frames.

[0025] The color image sensor 31 includes three types of pixels (picture elements) composed of an R light-receiving element 33R, a G light-receiving element 33G, and a B light-receiving element 33B. The three types of light-receiving elements 33R, 33G, and 33B can receive red light (R light), green light (G light), and blue light (B light), respectively. In the image sensor 31 of this example, the three types of light-receiving elements 33R, 33G, and 33B adopt a Bayer array in which three colors and four pixels (RGGB) are arranged in a 4-pixel square unit area, for example.

[0026] The color image sensor 31 receives the image of the subject 12 through the lens 32 and outputs an imaging signal S1 corresponding to the light-receiving result. The imaging signal S1 corresponds to M frames. The imaging signal S1 output by the image sensor 31 is input to the image processing unit 50.

[0027] As shown in FIG. 1, the imaging device 11 may be configured to output an image captured in an imaging format corresponding to a mode selected by an operator from among a plurality of modes. For example, the control unit 40 shown in FIG. 1 includes a mode setting unit 41, a trigger control unit 42, and a selection unit 43. The mode setting unit 41 sets the mode selected by the operator operating the input unit 75 (see FIG. 4). The mode setting unit 41 outputs a mode signal MS designating the mode to the trigger control unit 42 and the selection unit 43.

[0028] The trigger control unit 42 outputs a light emission control signal LC serving as a trigger for the light emission start timing to the light sources 21 to 23, and an imaging control signal IC serving as a trigger for starting imaging (exposure) of the image sensor 31 at an imaging timing synchronized with the light emission timing of the light sources 21 to 23. The trigger control unit 42 performs light emission control for causing M types of the light sources 21 to 23 corresponding to the mode designated by the mode signal MS among the N types of light sources 21 to 23 to emit light in a time-division manner using the light emission control signal LC as a trigger. Further, the trigger control unit 42 performs imaging control for causing the image sensor 31 to perform imaging M times with a predetermined exposure time at an imaging timing synchronized with the light emission timing of the light sources 21 to 23 using the imaging control signal IC as a trigger. Due to the M times of light emission and M times of imaging with respect to the subject 12, an imaging signal S1 corresponding to M frames is output from the image sensor 31.

[0029] The image processing unit 50 performs various image processes on the M frames based on the imaging signal S1. As a result of the image processing, the image processing unit 50 generates and outputs one-frame imaging data Ixyz of a plurality of bands (k bands) in which the pixel data of a plurality of frames (k frames) are rearranged in a predetermined pixel array within one frame. Here, k in this example is a natural number satisfying M ≤ k ≤ 4.

[0030] Further, the imaging device 11 may include a pixel array conversion unit 55. The pixel array conversion unit 55 converts one-frame imaging data Ixyz of a plurality of bands (k bands) having a predetermined pixel array into k pieces of image data for each color. The color here is not limited to colors other than RGB among visible light, but also includes pseudo colors representing near-infrared light.

[0031] The image processing unit 50 shown in FIG. 1 includes an adjacent pixel mixing processing unit 51, a resampling processing unit 52, and a rearrangement processing unit 53. In FIG. 1, the image sensor 31 shows only the sensor unit 31S which is a part of the imaging device. Also, in FIG. 1, the image sensor 31 and the adjacent pixel mixing processing unit 51 are drawn as separate entities, but the image sensor 31 may be configured to include at least a part of the adjacent pixel mixing processing unit 51 that performs adjacent pixel mixing processing on the imaging signal S1 output by the sensor unit 31S. Further, the sensor unit 31S may be configured to include at least a part of the adjacent pixel mixing processing unit 51.

[0032] The selection unit 43 performs selection of the processing mode and parameter setting for the image sensor 31 and the image processing unit 50 based on the selection setting signal. Among the image sensors 31 used in the imaging device 11, some have an adjacent pixel mixing processing unit 51 and some do not. Also, when the image sensor 31 includes an adjacent pixel mixing processing unit 51, some have limitations on its functions and some do not. The operator makes a setting to select the processing to be performed on the image sensor 31 and the image processing unit 50 according to the presence or absence of the adjacent pixel mixing processing unit 51 and the presence or absence of its functional limitations in the image sensor 31 used in the imaging device 11. As a result of the operator operating the input unit 75 to select this processing, the selection setting signal is input to the selection unit 43.

[0033] The selection unit 43 outputs an exposure signal ES determined from the selection setting signal and the mode signal MS based on the selection setting made by the operator operating the input unit 75 (see FIG. 4) to the image sensor 31. The exposure signal ES is a signal that specifies the exposure time every M times of imaging (exposure).

[0034] The selection unit 43 supplies a binning signal BS and a shift signal PS to the image processing unit 50. The binning signal BS is a signal that instructs adjacent pixel mixing processing for each frame. The binning signal BS includes horizontal and vertical pixel numbers that specify adjacent pixels to be mixed. Further, the shift signal PS is a signal that instructs the phase (binning phase) for shifting pixels for each frame to the resampling processing unit 52. The phase shift processing for shifting pixels to the specified phase for each frame is performed by selecting a combination of adjacent pixels to be mixed in the adjacent pixel mixing processing. In addition to the signals BS and PS, a plurality of setting signals (not shown) including a mode setting signal are input from the selection unit 43 to the image processing unit 50.

[0035] The adjacent pixel mixing processing unit 51 performs adjacent pixel mixing processing (binning processing) for each frame on the M frames (imaging signal S1) from the sensor unit 31S. Here, the adjacent pixel mixing processing is a process of enhancing sensitivity by integrating the charges of a plurality of pixels and outputting them as one pixel. However, the output resolution decreases according to the numerical value of the binning magnification, which is the number of adjacent pixels to be mixed. For example, by performing binning processing on 4 pixels of one color, which is the output of the image sensor 31, to 1 pixel of one color, the same sensitivity can be obtained with an exposure time 1 / 4 that before the binning processing. Therefore, the exposure time of the image sensor 31 can be 1 / 4. In other words, when K pixels (where K is a natural number of 2 or more) that receive light of the same color (wavelength) are subjected to adjacent pixel mixing processing (binning processing), the exposure time required to obtain the same sensitivity is only 1 / K.

[0036] The resampling processing unit 52 performs resampling processing for shifting the phase, which is the center position of the pixels, for each frame on the frame. The resampling processing unit 52 has an upsampling processing function for increasing the number of pixels per frame and a downsampling processing function for decreasing the number of pixels per frame in order to perform resampling processing. The downsampling processing function may constitute at least a part of the adjacent pixel mixing processing unit 51. That is, at least a part of the adjacent pixel mixing processing unit 51 and the resampling processing unit 52 may be configured by the same processing unit.

[0037] The rearrangement processing unit 53 inputs a plurality of frames after adjacent pixel mixing processing and phase shift processing, and rearranges the pixel data (pixel values) for the plurality of frames into pixels in one frame in a predetermined pixel arrangement. The rearrangement processing unit 53 outputs one-frame imaging data Ixyz of k bands as a processing result of rearranging the pixel data of k frames subjected to adjacent pixel mixing in a predetermined pixel arrangement within one frame. When k = 3, the pixel data of three frames subjected to adjacent pixel mixing are rearranged in a three-color four-pixel arrangement similar to the Bayer arrangement as an example of a predetermined pixel arrangement within one frame. In the example of FIG. 1, one-frame imaging data Ixyz of three bands generated by the rearrangement processing is output from the image processing unit 50. Hereinafter, one-frame imaging data is also referred to as a one-frame image.

[0038] The pixel array conversion unit 55 converts the multi-band one-frame image Ixyz output from the image processing unit 50 into a plurality of (multi-band) single-color images SIi. Here, the subscript i of SIi is i = 1, 2,..., k, and k is the number of bands. That is, the pixel array conversion unit 55 has a function of converting the k-band one-frame image Ixyz into k single-color images SI1 to SIk. For example, the three-band one-frame image Ixyz is converted by the pixel array conversion unit 55 into k (for example, three) images: an X pseudo-color image SI1, a Y pseudo-color image SI2, and a Z pseudo-color image SI3.

[0039] Generally, in a general-purpose color camera, a Bayer array is often used for the image sensor 31. Therefore, an existing pixel array conversion unit 55 that can convert a single-frame color image of three bands of the Bayer array obtained by imaging with a general-purpose color camera into a plurality of RGB monochromatic images (R image, G image, B image) is easily available. The image processing unit 50 of the present embodiment generates a one-frame image of a plurality of bands in a predetermined pixel array with, for example, four pixels of the same as the Bayer array as one unit so that the existing pixel array conversion unit 55 can be used. In this specification, when referring to the Bayer array for the predetermined pixel array, it may refer to the Bayer array not only limited to the pixel array of four pixels of three colors RGGB, but also the same pixel array of three colors and four pixels. In the present embodiment, the pixel array conversion unit 55 is, for example, a Bayer array conversion unit that converts a one-frame image of the Bayer array into a plurality of monochromatic images. Note that the pixel array conversion unit 55 is not limited to the Bayer array conversion unit, and may have an image separation and conversion function that converts a one-frame image of a predetermined pixel array in a form other than the Bayer array into a plurality of monochromatic images.

[0040] <Schematic Configuration of Camera and PC> Next, with reference to FIG. 2, the schematic configuration of the color camera 30 and the personal computer 70 (hereinafter also referred to as "PC 70") that constitute the imaging device 11 will be described. The color camera 30 shown in FIG. 2 (hereinafter also simply referred to as "camera 30") is based on a general-purpose color camera, and the IR cut filter 35 shown by the two-dot chain line in FIG. 2 is removed. The image sensor 31 has sensitivity that can image light in the wavelength band of visible light and near-infrared light without cutting the near-infrared light.

[0041] The camera 30 includes a lens 32 assembled to the lens barrel 30a and the aforementioned image sensor 31. The image sensor 31 includes a sensor unit 31S. The sensor unit 31S includes a color filter 34. The color filter 34 includes an R filter 34R, a G filter 34G, and a B filter 34B arranged in a Bayer array in this example.

[0042] The sensor unit 31S includes an R light-receiving element 33R, a G light-receiving element 33G, and a B light-receiving element 33B. The R light-receiving element 33R receives the light (mainly red light) transmitted through the R filter 34R and outputs an R imaging signal corresponding to the received light amount. The G light-receiving element 33G receives the light (mainly green light) transmitted through the G filter 34G and outputs a G imaging signal corresponding to the received light amount. The B light-receiving element 33B receives the light (mainly blue light) transmitted through the B filter 34B and outputs a B imaging signal corresponding to the received light amount. The light-receiving elements 33R, 33G, 33G, 33B are arranged in a Bayer array with three colors and four pixels of RGGB as one unit. Note that the light-receiving elements 33R, 33G, 33B constituting the sensor unit 31S can also be said to be the pixels of the image sensor 31.

[0043] Next, with reference to FIG. 2, the configuration of the adjacent pixel mixing processing unit 51 will be described. The imaging device 11 includes a signal processing unit 60 that processes an imaging signal corresponding to a frame output from the sensor unit 31S of the image sensor 31. The signal processing unit 60 is provided in at least one of the camera 30 and the PC 70. The adjacent pixel mixing processing unit 51 is provided in at least one of the image sensor 31 and the signal processing unit 60. In the present embodiment, the adjacent pixel mixing processing unit 51 is provided in at least the signal processing unit 60 among the image sensor 31 and the signal processing unit 60.

[0044] As shown in FIG. 2, the adjacent pixel mixing processing unit 51 also depends on the configuration of the camera 30. The adjacent pixel mixing processing unit 51 may include an adjacent pixel mixing processing unit 51S built in the image sensor 31 and an adjacent pixel mixing processing unit 51D built in the signal processing unit 60.

[0045] The adjacent pixel mixing processing unit 51S in the image sensor 31 may be at least one of the adjacent pixel mixing processing unit 51S built in the sensor unit 31S and the adjacent pixel mixing processing unit 51S mounted on the chip of the image sensor 31. The adjacent pixel mixing processing unit 51S built in the sensor unit 31S outputs the captured image signal S1 subjected to adjacent pixel mixing processing by mixing the charges for a plurality of adjacent light receiving elements (pixels) and outputting them as those for one pixel from the sensor unit 31S. In this way, the adjacent pixel mixing processing unit 51S uses, as a frame, the group of charges for one imaging stored in the many light receiving elements (pixels) constituting the sensor unit 31S, and performs, as adjacent pixel mixing processing, the processing of mixing and outputting the charges for a plurality of adjacent pixels for each frame. The adjacent pixel mixing processing unit 51S causes the image sensor 31 to output the imaging signal S1 having an analog value corresponding to the charges mixed by this adjacent pixel mixing processing. On the other hand, the adjacent pixel mixing processing unit 51S mounted on the chip of the image sensor 31 is constituted by an integrated circuit that performs substantially the same processing as the adjacent pixel mixing processing unit 51D in the signal processing unit 60. The adjacent pixel mixing processing unit 51S is provided by the sensor manufacturer. Whether or not to provide the function of the adjacent pixel mixing processing unit 51S depends on the design specifications of the camera 30 to be adopted.

[0046] The adjacent pixel mixing processing unit 51D built in the signal processing unit 60 is, for example, an integrated circuit. The adjacent pixel mixing processing unit 51D mounted on the signal processing unit 60 in the camera 30 is provided by the camera manufacturer. Whether or not the camera 30 includes the adjacent pixel mixing processing unit 51D depends on the configuration of the camera 30 to be adopted. The adjacent pixel mixing processing unit 51D mounted on the signal processing unit 60 in the PC 70 is provided, for example, by the manufacturer of the imaging device 11.

[0047] As shown in FIG. 2, the adjacent pixel mixing processing unit 51 mainly has three forms. In the first form, the image sensor 31 includes an adjacent pixel mixing processing unit 51S mounted on at least one of the sensor unit 31S and the chip. And the signal processing unit 60 provided in at least one of the camera 30 and the PC 70 mounts the adjacent pixel mixing processing unit 51D. It is sufficient that the signal processing unit 60 is provided in either the camera 30 or the PC 70.

[0048] As described above, among the cameras 30, some have the sensor unit 31S incorporating the adjacent pixel mixing processing unit 51S and some do not. Also, even if it is incorporated in the sensor unit 31S, there may be limitations in the function of the adjacent pixel mixing processing unit 51S. For this reason, in the first form, at least the signal processing unit 60 among the image sensor 31 and the signal processing unit 60 is configured to be provided with the adjacent pixel mixing processing unit 51D. By providing the signal processing unit 60 with the adjacent pixel mixing processing unit 51D, it is possible to perform adjacent pixel mixing processing on the M frames output from the sensor unit 31S regardless of the presence or absence of the adjacent pixel mixing processing unit 51S in the image sensor 31 or the presence or absence of its function limitation.

[0049] In the present embodiment, in particular, as the first form, it includes at least the adjacent pixel mixing processing unit 51S incorporated in the sensor unit 31S and the adjacent pixel mixing processing unit 51D mounted on the signal processing unit 60 in the PC 70. The manufacturer of the imaging device 11 mounts the signal processing unit 60 capable of performing necessary adjacent pixel mixing processing on the PC 70 regardless of the configuration of the camera 30. When the sensor unit 31S includes the adjacent pixel mixing processing unit 51S, the adjacent pixel mixing processing unit 51S is made to perform adjacent pixel mixing processing preferentially. This is because when adjacent pixel mixing processing is performed by the adjacent pixel mixing processing unit 51S in the sensor unit 31S, relatively high sensitivity can be obtained for the processed pixels, and the number of pixels per frame can be reduced, thereby shortening the output time required for outputting the imaging signal.

[0050] The second form is a form in which the image sensor 31 does not include the adjacent pixel mixing processing unit 51S and the signal processing unit 60 includes the adjacent pixel mixing processing unit 51D. The third form is a form in which the image sensor 31 includes the adjacent pixel mixing processing unit 51S and the signal processing unit 60 does not include the adjacent pixel mixing processing unit 51D.

[0051] <Output image of the imaging device> Next, with reference to FIG. 3, the image output by the image processing unit 50 according to the mode will be described. The image processing unit 50 can output at least one of the images in FIGS. 3(a) to (d). Note that FIGS. 3(e) and (f) are images that can also be output by a conventional general-purpose color camera. That is, FIGS. 3(e) and (f) are images of a comparative example. FIG. 3(e) is an RGB image Irgb of one frame of the Bayer array, and FIG. 3(f) is a monochrome image Iwb.

[0052] FIG. 3(a) is a 3-band 1-frame image Ixyz output by the image processing unit 50 in the first mode. This 1-frame image Ixyz is 1-frame imaging data in which pixels of three colors, X pixels, Y pixels, and Z pixels, are arranged in a 3-color 4-pixel array similar to the Bayer array in one frame. This 3-band 1-frame image Ixyz is converted by the pixel array conversion unit 55 into a 3-band pseudo-color image (X, Y, Z image) composed of three pseudo-color images.

[0053] FIG. 3(b) is a 4-band 1-frame image Ixzgb output by the image processing unit 50 in the second mode. This 1-frame image Ixzgb is 1-frame imaging data in which pixels of four colors, X pixels, Z pixels, G pixels, and B pixels, are assigned to each pixel of a square 4-pixel array similar to one unit of the Bayer array in one frame. This 4-band 1-frame image Ixzgb is converted by the pixel array conversion unit 55 into a 4-band image (X, R, G, B image) composed of two pseudo-color images (X image, Z image) and two color images (G image, B image).

[0054] FIG. 3(c) is a 4-band 1-frame image Ixrgb output by the image processing unit 50 in the third mode. This 1-frame image Ixrgb is 1-frame imaging data of 4 bands in which pixels of four colors, i.e., X pixels, R pixels, G pixels, and B pixels, are assigned to a square of 4 pixels similar to 1 unit of the Bayer array. This 4-band 1-frame image Ixrgb is converted by the pixel array conversion unit 55 into a 4-band image (X, R, G, B image) composed of one pseudo-color image (X image) and three color images (R image, G image, B image).

[0055] FIG. 3(d) is an image output by the image processing unit 50 in the fourth mode, and includes one 3-band 1-frame image Ixyz and one RGB image Irgb of a Bayer array 1 frame. This 3-band 1-frame image Ixyz is 1-frame imaging data similar to FIG. 3(a). This 3-band 1-frame image Ixyz is converted by the pixel array conversion unit 55 into a 3-band pseudo-color image composed of three pseudo-color images. The RGB image Irgb is converted by the pixel array conversion unit 55 into three color images (R image, G image, B image).

[0056] Note that the image processing unit 50 of the present embodiment can output the RGB image Irgb of a Bayer array 1 frame shown in FIG. 3(e) and the monochrome image Iwb shown in FIG. 3(f) in the fifth mode and the sixth mode, respectively. Details of each image process according to the first to fourth modes of the image processing unit 50, which is a feature of the present embodiment, will be described in the first to fifth examples described later.

[0057] <Configuration of an imaging device including a signal processing unit at a different position> Next, with reference to FIGS. 4 and 5, a specific configuration of the imaging device 11 will be described. FIGS. 4 and 5 show an example in which the imaging device 11 is applied to the inspection device 10. FIG. 4 is an example of a first configuration in which the above-described signal processing unit 60 is built in the PC 70. FIG. 5 is an example of a second configuration in which the signal processing unit 60 is built in the camera 30 and the PC 70. Note that the signal processing unit 60 may be built in only the camera 30.

[0058] The signal processing unit 60 shown in FIGS. 4 and 5 is constituted by an FPGA (field-programmable gate array). An FPGA is an integrated circuit whose configuration can be set by a purchaser or a designer after manufacture. It is a kind of PLD (programmable logic device) and is a gate array programmable on-site. Note that the signal processing unit 60 may be constituted by an ASIC (Application Specific Integrated Circuit). Further, the FPGA may have a configuration incorporating a CPU core and peripheral circuits.

[0059] Hereinafter, with reference to FIG. 4, the basic configurations of the imaging device 11 and the inspection device 10 will be described. As shown in FIG. 4, the illumination unit 20 includes a first light source 21, a second light source 22, and a third light source 23. The first light source 21 can emit X-rays having an emission spectrum in a frequency band different from, for example, RGB light. The second light source 22 can emit Y-rays having an emission spectrum in a frequency band different from, for example, RGB light and X-rays. The third light source 23 can emit Z-rays having an emission spectrum in a frequency band different from, for example, RGB light, X-rays, and Y-rays. That is, the first to third light sources 21 to 23 can emit X-rays, Y-rays, and Z-rays having emission spectra in mutually different frequency bands, respectively.

[0060] Note that the types of light sources constituting the illumination unit 20 are not limited to three types and may be N types. Here, N is a natural number of 2 or more. The example shown in FIGS. 4 and 5 is an example of N = 3. Further, it is not necessary to use all of the N types of light sources for time-division emission, and time-division emission may be performed using M types of light sources selected according to the mode among the N types of light sources. Here, M is a natural number satisfying 2 ≤ M ≤ N. Note that when imaging is performed by emitting X-rays, Y-rays, and Z-rays in time division, N = M.

[0061] The lighting unit 20 may be configured to include light sources other than the three types of light sources 21 to 23. For example, the lighting unit 20 may include three types of light sources (R light source, G light source, and B light source) that emit RGB light respectively. For example, the lighting unit 20 may include a white light source 22W capable of emitting white light. In this case, the white light source 22W may be composed of an R light source, a G light source, and a B light source capable of emitting RGB light simultaneously. Thus, in this embodiment, one type of light source used for one-time light emission when emitting light in multiple times in a time-division manner may be composed of a plurality of light sources that emit light simultaneously. In this case, one type of light source has an emission spectrum obtained by combining all the individual emission spectra of the plurality of light sources constituting it. Note that the first light source 21 (X light source) of this embodiment is a near-infrared light source capable of emitting near-infrared light as X-ray.

[0062] Thus, the lighting unit 20 may be configured to include at least two types of light sources having different emission spectra within the visible light frequency band and the near-infrared light frequency band. For example, N = 2 may be used, or N = 4 may be used.

[0063] Also, as long as the plurality of types of light sources 21 to 23 can irradiate light on the subject 12, the positional relationship between the subject 12 and the color camera 30 is not particularly limited. The position of the light source may be set according to the image to be captured. For example, when irradiating light on the subject 12 from the front and capturing the image of the reflected light with the camera 30, the light source is arranged at a position where it can irradiate light on the subject 12 from the side of the camera 30. Also, for example, when the subject 12 has a light-transmissive portion and the camera 30 captures the image of the transmitted light that has passed through the subject 12, the light source is arranged at a position on the opposite side of the camera 30 with the subject 12 interposed therebetween. Thus, the position of the plurality of types of light sources 21 to 23 in the lighting unit 20 can be set as appropriate, and they may be configured to be incorporated into one lighting body, or may be configured to be arranged separately at different positions.

[0064] Also, as shown in FIGS. 4 and 5, the PC 70 includes a CPU 71 (Central Processing Unit), an image input board 72, a VRAM 73 (Video RAM), and an inspection processing unit 74. The PC 70 includes an input unit 75 such as a keyboard and a display unit 76 such as a monitor.

[0065] The control unit 40 (see FIG. 1) is constituted by a part of the CPU 71 and the image input board 72. The CPU 71 outputs a control signal for controlling the illumination unit 20 and the camera 30 to the signal processing unit 60 based on a detection signal from a sensor (not shown). Further, the image input board 72 stores imaging data corresponding to a frame input from the camera 30 and image data output from the signal processing unit 60 in the VRAM 73. The inspection processing unit 74 inspects the subject 12 based on the image data temporarily stored in the VRAM 73. Instead of the inspection processing unit 74, an image analysis unit (not shown) may be provided, and a configuration may be adopted in which a predetermined process is performed on the subject 12 based on the image analysis result by the image analysis unit. Examples of the image analysis unit include a configuration in which a mark or a specific shape formed on the subject 12 is detected by image analysis, and a predetermined process is performed on a predetermined position of the subject 12 using the detected position as a reference position. Examples of the predetermined process include various processes such as a processing process (such as a laser processing process), a printing process, a marking process, a heat treatment process, an ultraviolet irradiation process, and a gripping process.

[0066] <Detailed Configuration of Signal Processing Unit 60> Next, with reference to FIG. 6, the detailed configuration of the signal processing unit 60 will be described. The signal processing unit 60 shown in FIG. 6 is constituted by the aforementioned FPGA.

[0067] As shown in FIG. 6, the signal processing unit 60 includes a plurality (for example, (M - 1) pieces) of frame memories 61 and 62 that temporarily store the previously input frames in order to synchronously process M frames (imaging signals) serially input by time-division imaging. FIG. 6 is an example of a configuration capable of supporting up to a maximum of M = 3, and the signal processing unit 60 includes a first frame memory 61 and a second frame memory 62. Note that the number of frame memories may be three or more, and a configuration capable of supporting M = 4 or more may be adopted.

[0068] In addition, the signal processing unit 60 includes an adjacent pixel mixing processing unit 51D, a resampling processing unit 52, and a rearrangement processing unit 53. The adjacent pixel mixing processing unit 51D is provided so that adjacent pixel mixing processing can be performed on all M frames output from the sensor unit 31S even when the adjacent pixel mixing processing unit 51S is not provided in the image sensor 31 or when there are functional limitations in the adjacent pixel mixing processing unit 51S. In this regard, the adjacent pixel mixing processing unit 51D constitutes a part of the adjacent pixel mixing processing unit 51.

[0069] M frames are serially transmitted from the camera 30 to the signal processing unit 60. The M frames are sequentially stored in both or one of the frame memories 61 and 62 for the previously input frames, and are input to the resampling processing unit 52 together at the timing when the last (Mth) frame is input. Note that the frame memories 61 and 62 may be disposed outside the FPGA in the signal processing unit 60.

[0070] The resampling processing unit 52 includes a first selector 63, four upsampling processing units 64, and four subsequent low-pass filters 65 (hereinafter referred to as "subsequent LPF 65"). Further, the resampling processing unit 52 includes a second selector 66, four preceding low-pass filters 67 (hereinafter referred to as "preceding LPF 67"), and four downsampling processing units 68. In the present embodiment, the adjacent pixel mixing processing unit 51D is composed of four preceding low-pass filters 67 (hereinafter referred to as "preceding LPF 67") and four downsampling processing units 68. That is, the adjacent pixel mixing processing unit 51D, which is a part of the adjacent pixel mixing processing unit 51, is also a part of the resampling processing unit 52.

[0071] The four upsampling processing units 64 that are the selection targets of the first selector 63 are connected to the second selector 66 via four subsequent LPF 65 connected in series therewith. Also, the four preceding LPF 67 that are the selection targets of the second selector 66 are connected to the rearrangement processing unit 53 via four downsampling processing units 68 connected in series therewith.

[0072] The four upsampling processing units 64 perform upsampling processing for increasing the number of pixels. The upsampling processing unit 64 enables a phase shift that shifts the phase, which is the center position of each pixel, by doubling or quadrupling the number of pixels per frame (number of pixels / frame). Here, when the adjacent pixel mixing processing unit 51S in the image sensor 31 does not have a phase shift function, it is necessary to perform a phase shift process on the frame subjected to the adjacent pixel mixing process by the adjacent pixel mixing processing unit 51S. For this purpose, it is necessary to increase the number of pixels in the frame subjected to the adjacent pixel mixing process. Therefore, the resampling processing unit 52 includes the upsampling processing unit 64. Note that a processing coefficient corresponding to the mode specified by the mode setting signal is individually set for the upsampling processing unit 64.

[0073] The four subsequent LPFs 65 remove high-frequency components exceeding the first frequency with a period equal to two pixel periods of the image sensor 31 from the frame after the upsampling process. The four preceding LPFs 67 remove high-frequency components exceeding the second frequency with a period equal to four pixel periods of the image sensor 31 from the frame from which the high-frequency components exceeding the first frequency have been removed by the subsequent LPF 65. Note that processing coefficients corresponding to the mode specified by the mode setting signal are individually set for the subsequent LPF 65 and the preceding LPF 67.

[0074] The downsampling processing unit 68 performs a downsampling process for reducing the number of pixels in the frame. The downsampling processing unit 68 can shift the phase of each pixel before the upsampling process by selecting the phase of the downsampling. That is, the phase of the pixel after the adjacent mixing process can be selected by selecting the combination of four adjacent pixels to be mixed by the downsampling process. The downsampling processing unit 68 performs an adjacent four-pixel mixing process of mixing four adjacent pixels of the combination selected based on the shift signal PS, thereby performing the phase shift process together. At this time, whether or not to cause the downsampling processing unit 68 to perform the adjacent four-pixel mixing process is individually instructed by an ON / OFF command based on the setting signal from the selection unit 43. Note that processing coefficients corresponding to the mode specified by the mode setting signal are individually set for the downsampling processing unit 68.

[0075] The rearrangement processing unit 53 inputs a plurality of (four) frames that have been subjected to adjacent four-pixel mixing processing by the adjacent pixel mixing processing unit 51S in the camera 30 or the adjacent pixel mixing processing unit 51D in the signal processing unit 60. The rearrangement processing unit 53 performs a rearrangement process of rearranging the pixel data (pixel values) of the input plurality of frames in a predetermined pixel arrangement within one frame. In the present embodiment, the predetermined pixel arrangement differs depending on the number of bands of the one-frame imaging data (one-frame image) generated by the signal processing unit 60. For example, when one-frame image is three bands, the predetermined pixel arrangement is a pixel arrangement of three colors and four pixels similar to the Bayer arrangement. For example, when the three colors are XYZ, instead of the four pixels of the Bayer arrangement RGGB when the three colors are RGB, it is arranged in four pixels of XYYZ, which is a pixel arrangement similar to the Bayer arrangement when the three colors are XYZ. Also, for example, when one-frame image is four bands, the predetermined pixel arrangement is a pixel arrangement of four colors and four pixels in which four-color pixels are arranged in four pixels similar to the Bayer arrangement. In this way, by setting the predetermined pixel arrangement to a pixel arrangement of three colors and four pixels or four colors and four pixels, an existing Bayer array conversion unit can be adopted as the pixel array conversion unit 55 (see FIGS. 1 and 3). Note that the predetermined pixel arrangement to be rearranged by the rearrangement processing unit 53 is specified based on a setting signal input from the selection unit 43 (see FIG. 1) to the signal processing unit 60.

[0076] FIG. 6 shows an example in which the rearrangement processing unit 53 outputs a three-band one-frame image Ixyz in the first mode. The rearrangement processing unit 53 outputs one-frame imaging data, which is a three-band or four-band one-frame image shown in FIGS. 3(a) to (d) according to the mode at that time. Further, the resampling processing unit 52 has a function of transmitting the input frame to the rearrangement processing unit 53 as it is without performing any processing on the basis of the selection of set values based on the binning signal BS, the shift signal PS, etc. Specifically, based on the setting signal from the selection unit 43, the ON / OFF of the upsampling processing unit 64 and the ON / OFF of the downsampling processing unit 68 are selected. The resampling processing unit 52 has a function of passing, as it is, a frame in which necessary adjacent four-pixel mixing processing and phase shift processing have been performed by the adjacent pixel mixing processing unit 51S in the image sensor 31, for example, by selecting a processing coefficient. Note that this function may be configured such that each selector 63, 66 can select a four-column signal path that does not pass through the upsampling processing unit 64, the subsequent-stage LPF 65, the preceding-stage LPF 67, and the downsampling processing unit 68. Also, in the second configuration shown in FIG. 5, the signal processing unit 60 in the camera 30 may not include the rearrangement processing unit 53.

[0077] <Characteristics of Illumination Unit 20 and Image Sensor 31> Next, with reference to FIG. 7, the characteristics of the light sources 21 to 23 and the image sensor 31 will be described. In the present embodiment, the illumination unit 20 has a configuration including the light sources 21 to 23 having the light emission characteristics shown in FIG. 7(a) and a configuration including the light sources 21 to 23 having the light emission characteristics shown in FIG. 7(b). FIG. 7(a) shows an example in which the light sources 21 to 23 are an X light source 21 capable of emitting X-rays, a Y light source 22 capable of emitting Y-rays, and a Z light source 23 capable of emitting Z-rays. Further, FIG. 7(b) shows an example in which the light sources 21 to 23 are an X light source 21 capable of emitting X-rays, a white light source 22W (also referred to as a "W light source 22W") capable of emitting white light (also referred to as "W light"), and a Z light source 23Z capable of emitting Z-rays. Note that the illumination unit 20 may include the X light source 21, the Y light source 22, and the Z light source 23 having the emission spectrum shown in FIG. 7(a) and the W light source 22W and the Z light source 23Z having the emission spectrum characteristics shown in FIG. 7(b).

[0078] Figures 7(a) and 7(b) show the characteristics of different illumination units 20 and image sensors 31, respectively. In Figures 7(a) and 7(b), the left graph shows the emission spectra of a plurality of light sources 21 to 23. The middle graph shows the spectral sensitivity of the image sensor 31. This spectral sensitivity shows the spectral sensitivity for pixels obtained by mixing adjacent four pixels. Also, the right graph shows the spectral output characteristics of the imaging signal of the image sensor 31 when imaging is performed using the above illumination.

[0079] The left graph in Figure 7(a) shows the emission spectra of the X light source 21, the Y light source 22, and the Z light source 23. In this graph, the horizontal axis represents the wavelength (nm), and the vertical axis represents the emission intensity. The X light of the X light source 21 has an emission intensity of 0.3 or more in the wavelength band of 830 to 980 nm. The Y light of the Y light source 22 has an emission intensity of 0.3 or more in the wavelength band of 620 to 660 nm. The Z light of the Z light source 23 has an emission intensity of 0.3 or more in the wavelength band of 510 to 560 nm.

[0080] As shown by the sensitivity curve S in the middle graph of Figure 7(a), when the pixels obtained by mixing adjacent four pixels of the image sensor 31 are taken as pixel units, the spectral sensitivity has a relative sensitivity of a predetermined value (for example, 0.2) or more with respect to light in a wide range from visible light to near-infrared light. In this graph, the horizontal axis represents the wavelength (nm), and the vertical axis represents the relative sensitivity. Since the image sensor 31 is configured to be provided in the color camera 30 from which the IR cut filter 35 is removed, the image sensor 31 has a relative sensitivity of, for example, 0.2 or more with respect to light in a wide range of the visible light wavelength region VA and the near-infrared wavelength region NIRA, which is a range from visible light to near-infrared light. Specifically, the image sensor 31 has a spectral sensitivity such that the relative intensity is, for example, 0.2 or more in the wavelength band of 400 to 950 nm. In particular, it has a spectral sensitivity such that the relative sensitivity is 0.5 or more in the wavelength band of 480 to 890 nm. Note that the image sensor 31 may have a relative sensitivity of, for example, 0.1 or more with respect to light in the ranges of the visible light wavelength region VA and the near-infrared wavelength region NIRA.

[0081] Therefore, as long as the light sources 21 to 23 have a wavelength band of the emission spectrum within a wide range from visible light to near-infrared light, the image sensor 31 that receives the light can output an imaging signal with a relative output of 0.2 or more.

[0082] Therefore, when the X-ray of the X light source 21, the Y light of the Y light source 22, and the Z light of the Z light source 23 expose the image sensor 31, the spectral output characteristics of the X-ray, Y light, and Z light shown in the left graph of FIG. 7(a) can be obtained. The spectral output characteristics of the X-ray, Y light, and Z light are shown as the product of the emission spectra of the light sources 21 to 23 shown in the left graph of FIG. 7(a) and the relative sensitivity of the image sensor 31.

[0083] That is, the right graph of FIG. 7(a) shows the relative output of the image sensor 31. In this graph, the horizontal axis represents the wavelength (nm), and the vertical axis represents the relative output. The image sensor 31 has spectral output characteristics including X-ray having a peak at about 890 nm, Y light having a peak at about 670 nm, and Z light having a peak at about 560 nm. The image sensor 31 can obtain a relative output of 0.2 or more for the X-ray, Y light, and Z light. In particular, in this example, a relative output of 0.5 or more can be obtained for the X-ray, Y light, and Z light.

[0084] Next, FIG. 7(b) will be described. The left graph of FIG. 7(b) shows the emission spectra of the X light source 21, the W light source 22W, and the Z light source 23Z. The X-ray of the X light source 21 has an emission intensity of 0.3 or more in the wavelength band of 830 to 880 nm. The W light of the W light source 22W has an emission intensity of 0.3 or more in the wavelength band of 450 to 660 nm. The Z light has an emission intensity of 0.3 or more in the wavelength band of 930 to 980 nm.

[0085] As shown in the middle graph of FIG. 7(b), the spectral sensitivity in pixel units considering 4 adjacent pixels of the image sensor 31 as 1 pixel has a relative sensitivity of a predetermined value (for example, 0.2) or more for light in a wide range from visible light to near-infrared light. The spectral sensitivity of this image sensor 31 is the same as that shown in FIG. 7(a).

[0086] Therefore, when the X-ray of the X light source 21, the W light of the W light source 22W, and the Z light of the Z light source 23Z expose the image sensor 31, the spectral output characteristics of the X-ray, W light, and Z light shown in the left graph of FIG. 7(b) can be obtained. The spectral output characteristics of the X-ray, W light, and Z light are shown as the product of the emission spectra of the X-ray, W light, and Z light of the light sources 21, 22W, 23Z shown in the left graph of FIG. 7(b) and the relative sensitivity of the image sensor 31.

[0087] That is, as shown in the right graph of FIG. 7(b), it has spectral output characteristics including X-ray having a peak at about 890 nm, W light having two peaks at about 500 nm and 600 nm, and Z light having a peak at about 960 nm. The image sensor 31 can obtain a relative output of 0.2 or more with respect to the X-ray, W light, and Z light.

[0088] Hereinafter, the first to fifth embodiments will be described. The first to fourth embodiments are examples of configurations in which the adjacent pixel mixing processing unit 51 is incorporated in the image sensor 31. The first embodiment is an example of a first mode for generating a three-band one-frame image Ixyz of a three-color four-pixel array similar to the Bayer array. The second embodiment is an example of a second mode for generating a four-band one-frame image Ixzgb of a four-color four-pixel array. The third embodiment is an example of a third mode for generating a four-band one-frame image Ixrgb of a four-color four-pixel array. The fourth embodiment is an example of a fourth mode for generating a three-band one-frame image Ixyz of a three-color four-pixel array and an RGB image Irgb of one frame of the Bayer array. The fifth embodiment is an example of a configuration in which the adjacent pixel mixing processing unit 51 is not provided in the image sensor 31 but is provided in the signal processing unit 60, and will be described as an example of the first mode.

[0089] (First Embodiment) First, with reference to FIGS. 8 to 11, a first embodiment which is an example of a first mode for generating a three-band one-frame image Ixyz of a three-color four-pixel array will be described. The illumination unit 20 of the first embodiment includes an X light source 21, a Y light source 22, and a Z light source 23. Since there is no functional limitation on the adjacent pixel mixing processing unit 51 in the image sensor 31, the image sensor 31 supports adjacent two-pixel mixing processing, adjacent four-pixel mixing processing, and a phase shift function.

[0090] FIG. 8 shows a timing chart from the imaging by the color camera 30 until the image processing unit 50 outputs an image. In FIG. 8, the horizontal axis represents time. In FIG. 8, in order from the top, the time-division light emission timing of the illumination unit 20, the image sensor 31 exposed by the illumination unit 20, the output image (sensor output image) output from the image sensor 31, and the FPGA image processing are shown. The FPGA image processing is image processing executed by the FPGA constituting the signal processing unit 60.

[0091] As the first to third light sources 21 to 23, the illumination unit 20 uses an X light source 21 capable of irradiating X-rays shown in FIG. 7(a), a Y light source 22 capable of irradiating Y light, and a Z light source 23 capable of irradiating Z light. The control unit 40 instructs the adjacent pixel mixing processing unit 51 in the image sensor 31 to perform adjacent pixel mixing processing on a plurality of imaging signals (M frames) sequentially input from the sensor unit 31S. At this time, the control unit 40 designates adjacent pixels to be mixed and the binning magnification.

[0092] As shown in FIG. 8, the control unit 40 performs light emission control to emit light from the X light source 21, the Y light source 22, and the Z light source 23 in order in time division. The control unit 40 performs imaging control to expose the image sensor 31 for a predetermined exposure period in accordance with the time-division light emission timing of the light sources 21 to 23.

[0093] As shown in FIG. 8, frames IX, IY0, and IZ are sequentially output from the image sensor 31. The first output frame IX is output as a result of the adjacent pixel mixing processing unit 51 in the image sensor 31 performing adjacent 4-pixel mixing processing (4-pixel binning processing) on the X imaging signal output as a result of the exposure of X-rays from the sensor unit 31S.

[0094] The second output frame IY0 is output as a result of the adjacent pixel mixing process (2-pixel binning process) performed by the adjacent pixel mixing unit 51 in the image sensor 31 on the Y imaging signal output as a result of the exposure of Y light from the sensor unit 31S. In this example, the adjacent pixel mixing process is performed on two horizontally adjacent pixels. The adjacent pixel mixing unit 51 in the image sensor 31 also has a phase shift function for shifting the phase (center position) of the binned (adjacent pixel mixed) pixels in addition to the binning process. By this phase shift function, the pixels binned in two pixels in the horizontal direction are shifted in pixel position by 180 degrees (one pixel) in line units every other line.

[0095] The third output frame IZ is output as a result of the adjacent pixel mixing process (4-pixel binning process) performed by the adjacent pixel mixing unit 51 in the image sensor 31 on the Z imaging signal output as a result of the exposure of Z light from the sensor unit 31S. Also, by the phase shift function provided in the image sensor 31, it is binned in four pixels and output with a specification of shifting by one original pixel in both the horizontal and vertical directions.

[0096] In the FPGA image processing shown in FIG. 8, the frame IY0 after the adjacent pixel mixing process is upsampled and converted into a frame IYF. Then, the adjacent pixel mixing process with a horizontal phase shift process and the adjacent pixel mixing process with a vertical phase shift process are separately performed on the frame IYF. As a result, a Y image IY1 adjacent pixel mixed with a phase shift of one original pixel in the horizontal direction and a Y image IY2 adjacent pixel mixed with a phase shift of one original pixel in the vertical direction are generated. The process until four frames IZ, IX, IY1, and IY2 are generated in the FPGA image processing corresponds to the resampling process.

[0097] Then, by rearranging the pixel values (pixel data) of the four frames IZ, IX, IY1, and IY2 to the corresponding pixels that are the rearrangement destinations indicated by the arrows in FIG. 8, one-frame imaging data Ixyz is generated. In this example, one-frame imaging data Ixyz of a three-color four-pixel array is generated by performing a pixel rearrangement process of assigning the pixel values of the four frames IZ, IX, IY1, and IY2 to the pixel positions corresponding to a three-color four-pixel array similar to the Bayer array.

[0098] <Image Processing of the First Embodiment> Next, with reference to FIG. 9, the control of the control unit 40 and the image processing of the image processing unit 50 will be described in detail.

[0099] The operator has previously operated the input unit 75 of the PC 70 to input necessary input information into the PC 70. Based on the input information, the PC 70 performs trigger setting, binning setting, exposure time setting, and signal processing setting. When a sensor (not shown) detects the subject 12 that has been conveyed and is approaching the imaging position, an imaging trigger based on this detection is input to the trigger control unit 42 in the signal processing unit 60 shown in FIG. 9. The trigger control unit 42 outputs a light emission control signal LC based on the imaging trigger to the light sources 21 to 23, and outputs an imaging control signal IC based on the imaging trigger to the image sensor 31. As a result, the X light source 21, the Y light source 22, and the Z light source 23 emit light sequentially in a time-division manner. The sensor unit 31S of the image sensor 31 is sequentially exposed by the X light, Y light, and Z light from the subject 12. Specifically, the sensor unit 31S sequentially outputs an X imaging signal based on the charges accumulated by the exposure of the X light, a Y imaging signal based on the charges accumulated by the exposure of the Y light, and a Z imaging signal based on the charges accumulated by the exposure of the Z light.

[0100] As shown in FIG. 9, the selection unit 43 includes a binning setting unit 44 and an exposure time setting unit 45. The binning setting unit 44 sets the setting content related to binning based on the binning setting set by the operator. The binning setting unit 44 in this example outputs a binning signal BS to the image sensor 31, thereby instructing the adjacent pixel mixing processing unit 51 in the image sensor 31 about the content of the binning process to be performed on the X, Y, and Z imaging signals output from the sensor unit 31S.

[0101] Also, the exposure time setting unit 45 outputs an exposure signal ES instructing the exposure time based on the exposure time setting to the image sensor 31. The exposure time is set according to the content of the binning process for the imaging signal generated by the exposure due to the light emission for each light emission unit that emits light in a time-division manner. For example, when adjacent 2-pixel mixing processing (2-pixel binning processing) is performed, since the 2 pixels to be mixed can receive twice as much light per unit time as 1 pixel, the exposure time can be 1 / 2. Also, when adjacent 4-pixel mixing processing (4-pixel binning processing) is performed, since the 4 pixels to be mixed can receive four times as much light per unit time as 1 pixel, the exposure time can be 1 / 4.

[0102] The sensor unit 31S of the image sensor 31 exposes the X-ray at a predetermined exposure time within the first light emission period when the X light source 21 is emitting light, and outputs an X imaging signal. During the second light emission period when the next Y light source 22 is emitting light, the sensor unit 31S exposes the Y light at a predetermined exposure time and outputs a Y imaging signal. Further, during the third light emission period when the next Z light source 23 is emitting light, the sensor unit 31S exposes the Z light at a predetermined exposure time and outputs a Z imaging signal.

[0103] The X imaging signal, Y imaging signal, and Z imaging signal output from the sensor unit 31S are sequentially input to the adjacent pixel mixing processing unit 51S within the same image sensor 31. The adjacent pixel mixing processing unit 51S sequentially performs predetermined processing including adjacent pixel mixing processing on the input X imaging signal, Y imaging signal, and Z imaging signal. Specifically, the adjacent pixel mixing processing unit 51S performs adjacent 4-pixel mixing processing on the X imaging signal to output an X image (frame X) in which adjacent 4 pixels are mixed and the number of frame pixels becomes 1 / 4. Next, the adjacent pixel mixing processing unit 51S performs horizontal adjacent 2-pixel mixing processing on the Y imaging signal by alternately changing the presence or absence of horizontal phase shift in units of one line, thereby outputting a Y image (frame Y) in which the number of frame pixels becomes 1 / 2. Further, the adjacent pixel mixing processing unit 51S performs adjacent 4-pixel mixing processing and phase shift processing on the Z imaging signal to output a Z image (frame Z) in which the phase, which is the center position of the pixel, is shifted in the horizontal and vertical directions and the number of frame pixels is 1 / 4 (see FIG. 8).

[0104] Here, the phase shift of the pixel is performed in advance to shift the phase (center coordinates) of the pixel to the phase of the rearrangement destination so that the pixel data of M images (for example, 3 images) X, Y, Z can be rearranged in a 3-color 4-pixel array similar to the Bayer array within one frame. In this way, the X image, Y image, and Z image subjected to adjacent pixel mixing processing and, if necessary, phase shift processing are sequentially output from the image sensor 31 to the signal processing unit 60. In the signal processing unit 60, the X image mixed with adjacent 4 pixels is written into the frame memories 61 and 62 and held until the output of the Z image.

[0105] The Y image is temporarily held in the frame memory 61, and then, when it is time to start image processing, it is read out from the frame memory 61, and image processing is performed on the Y image by the upsampling processing unit 64, the subsequent-stage LPF 65, and the preceding-stage LPF 67. Further, the Y image output from the preceding-stage LPF 67 is subjected to image processing by separate downsampling processing units 68, and Y images Y1 and Y2 are respectively output from the separate downsampling processing units 68.

[0106] The rearrangement processing unit 53 rearranges the pixel values (pixel data) of the images X, Y1, Y2, and Z input at the same timing to pixel positions corresponding to a Bayer array of 4 pixels per unit, thereby generating and outputting a 3-band 1-frame image Ixyz of a three-color 4-pixel array.

[0107] Here, referring to FIG. 10, the images (frames) generated at each of the steps (a) to (g) shown in the image processing unit 50 in FIG. 9 and the image processing for generating the images will be described. FIG. 10(a) is an X image IX (frame IX) that has been subjected to adjacent 4-pixel mixing processing and input from the image sensor 31 to the signal processing unit 60. Since the X image IX is an image after adjacent 4-pixel mixing processing, it is an image in which the number of frame pixels is 1 / 4 of the resolution of the image sensor 31.

[0108] FIG. 10(b) is a Y image IY0 (frame IY0) that has been subjected to horizontal adjacent 2-pixel mixing processing. The Y image IY0 is subjected to horizontal adjacent 2-pixel mixing processing so as to shift alternately by 1 pixel in the horizontal direction for each line. Since the Y image IY0 is an image after adjacent 2-pixel mixing processing, it is an image in which the number of frame pixels is 1 / 2 of the resolution of the image sensor 31.

[0109] The next FIG. 10(c) is a Y image IYU generated by subjecting the Y image IY0 to upsampling processing by the upsampling processing unit 64. The Y image IYU is image data in which the number of frame pixels is set to be twice by the upsampling processing. However, since zero values are inserted into the image data for each horizontal pixel, the resolution remains unchanged.

[0110] Figure 10(d) shows the Y image IYF generated by subjecting the Y image IYU to a subsequent LPF process by the subsequent LPF 65. This Y image IYF is input to two downsampling processing units 68, and by subjecting each Y image IYF to downsampling processing individually, the Y image IY1 shown in Figure 10(e) and the Y image IY2 shown in Figure 10(f) are generated. The Y image IY1 and the Y image IY2 are images with 1 / 2 the number of frame pixels with respect to the original Y image IY0 and 1 / 4 the number of frame pixels with respect to the Y image IYF.

[0111] Here, as shown in Figures 10(a), (e), (f), and (g), assume that pixel values indicated by numbers are obtained within each pixel of the X image IX, the Y images IY1 and IY2, and the Z image IZ. Each pixel of the images IX, IY1, IY2, and IZ is shifted to a pixel position corresponding to the Bayer array. Specifically, the grid-like broken lines shown in Figures 10(a), (e), (f), and (g) are the positions of the original pixels corresponding to the pixels of the image sensor 31. Each pixel of the images IX, IY1, IY2, and IZ has its phase shifted so that the central position relationship between pixels is not disrupted when rearranged into the same 3-color 4-pixel array as the Bayer array. That is, the X image IX shown in Figure 10(a) has no pixel phase shift. The Y image IY1 shown in Figure 10(e) has its pixel phase shifted by one pixel of the image sensor 31 in the horizontal direction with respect to the X image IX. The Y image IY2 shown in Figure 10(c) has its pixel phase shifted by one pixel of the image sensor 31 in the vertical direction with respect to the X image IX. Furthermore, the Z image IZ shown in Figure 10(d) has its pixel phase shifted by one pixel of the image sensor 31 in both the horizontal and vertical directions with respect to the X image IX. And all the pixels of the four images IX, IY1, IY2, and IZ have different phases (central coordinates).

[0112] Then, by assigning the pixel values of all the pixels of the four images IX, IY1, IY2, and IZ to the pixel positions corresponding to the Bayer array within one frame of the rearrangement destination, pixel rearrangement is performed without impairing the positional relationship of the XYZ pixel data. In this way, the 3-band 1-frame image Ixyz shown in Figure 10 is generated.

[0113] Next, with reference to FIG. 11, the timing chart of the imaging device 11 of the first embodiment will be described. In FIG. 11, the horizontal axis represents the time axis. FIG. 11(a) shows the timing chart of an imaging device of a comparative example including a general-purpose color camera. FIG. 11(b) shows the timing chart of the imaging device 11 of the first embodiment.

[0114] First, the imaging device of the comparative example will be described with reference to FIG. 11(a). As shown in FIG. 11(a), when an imaging trigger Trg is input, white light (W light) is irradiated onto the subject by the light source, and the subject is imaged by the image sensor. For this imaging, the image sensor is exposed for a predetermined exposure time. An RGB color image is output from a general-purpose color camera incorporating the image sensor.

[0115] The required time Trgb required for this comparative example imaging device to output an RGB color image is approximately given by the sum of the exposure time Texp of the image sensor and the image output time Tout from the image sensor. That is, the required time Trgb is Trgb = Texp + Tout.

[0116] Next, the imaging device 11 of the first embodiment will be described with reference to FIG. 11(b). As shown in FIG. 11(b), when imaging triggers Trg_x, Trg_y, Trg_z are sequentially input, the light sources 21 to 23 constituting the illumination unit 20 emit light in a time-division manner, and the subject 12 is sequentially irradiated with X light, Y light, and Z light.

[0117] The image sensor 31 sequentially captures the subject 12 in accordance with the emission timings of X light, Y light, and Z light. That is, the image sensor 31 is exposed for a predetermined exposure time Texp_x within the emission period of X light, then exposed for a predetermined exposure time Texp_y within the emission period of Y light, and further exposed for a predetermined exposure time Texp_z within the emission period of Z light. Since the pixel mixing process is performed on the imaging signal S1 output from the sensor unit 31S, it is sufficient to obtain the light reception amount required for one pixel with the number of pixels (2 pixels or 4 pixels) to be pixel-mixed. Therefore, the required exposure time when imaging with the irradiation of light to be pixel-mixed is 1 / P with respect to the exposure time Texp of the comparative example, where P is the number of pixels to be pixel-mixed. In the example of Fig. 11(b), there is a relationship of Texp_y = 2 * Texp_x. Therefore, there is a relationship of Texp_x = Texp / 4, Texp_y = Texp / 2, and Texp_z = Texp / 4.

[0118] Also, the X image and the Z image are converted into images with 1 / 4 of the frame pixel number compared to the comparative example by the adjacent 4-pixel mixing process, and the data amount thereof is 1 / 4 of the comparative example. Also, the Y image is converted into an image with a lower resolution than the comparative example by the adjacent 2-pixel mixing process, and the data amount thereof is 1 / 2 of the comparative example.

[0119] As shown in the sensor output image of Fig. 11(b), the X image, the Y image, and the Z image are sequentially output from the image sensor 31. At this time, the output times required for the output of the X image, the Y image, and the Z image depend on the data amount of each image. Therefore, the output times have a relationship of Tout_x = Tout / 4, Tout_y = Tout / 2, and Tout_z = Tout / 4.

[0120] The output X image is sequentially written into the frame memories 61 and 62, the Y image is written into the frame memory 61, and the X image and the Y image are output from the frame memories 61 and 62 in synchronization with the output timing of the Z image. Then, image processing is performed on the X image, the Y image, and the Z image at the same timing. As a result, a 3-band 1-frame image Ixyz of a 3-color 4-pixel array is output from the signal processing unit 60 (FPGA) at substantially the same timing as the image processing.

[0121] The required time Txyz of the imaging device 11 in the first embodiment is approximately given by Txyz = Texp_x + Texp_y + Tout_y + Tout_z. Here, Texp_x = Texp / 4, Texp_y = Texp / 2, Tout_y = Tout / 2, and Tout_z = Tout / 4. Therefore, Txyz < Trgb, and the required time Txyz of the imaging device 11 in the first embodiment can be shorter than the required time Trgb of the imaging device in the comparative example.

[0122] In the first embodiment, a corresponding example when the horizontal 2-pixel mixing function for generating the Y image IY0 is not mounted on the image sensor 31 will be described below. The image sensor 31 is caused to generate an image obtained by performing the same adjacent 4-pixel mixing process as the Y image IY1. The Y image IY1 with the frame pixel number reduced to 1 / 4 is upsampled to restore the frame pixel number to the original pixel number, thereby generating an image equivalent to the Y image IYF. Thereafter, the Y images IY1 and IY2 are generated by the same process as in the first embodiment, and further, a rearrangement process is performed to generate and output one-frame image Ixyz.

[0123] In this case, the output time of the Y image IY1 from the image sensor 31 can be shortened to 1 / 2 compared to the first embodiment, and the required time Txyz required for the total process until the one-frame image Ixyz after rearrangement is generated can be further shortened.

[0124] (Second Embodiment) Next, the second embodiment will be described with reference to FIGS. 12 to 15. The second embodiment is an example of the second mode, and generates a 4-band one-frame image Ixzgb of a 4-color 4-pixel array. The image sensor 31 supports 4-pixel mixing processing but does not support the phase shift function. The light sources 21 to 23 used are the X light source 21, the W light source 22W, and the Z light source 23Z having the emission spectrum characteristics shown in FIG. 7(b).

[0125] As shown in FIGS. 12 and 13, the illumination unit 20 causes the X light source 21, the W light source 22W, and the Z light source 23Z to emit light in time division in this order based on the imaging trigger of the light emission control signal LC from the control unit 40. The image sensor 31 captures (exposes) the subject 12 irradiated with the X light, the W light, and the Z light in sequence three times (M = 3) in accordance with the light emission timing. From the sensor unit 31S, the X image IX in which four pixels are mixed, the RGB image Irgb in which pixels are not mixed, and the Z image IZ0 in which four pixels are mixed are sequentially output. In the signal processing unit 60, the X image IX is sequentially written into two frame memories 61 and 62, and the RGB image Irgb is written into one frame memory 61. Note that the two frame memories 61 and 62 may be arranged in parallel, and the X image IX may be written into the first frame memory 61 and the RGB image Irgb may be written into the second frame memory 62.

[0126] At the timing when the three images are aligned, the signal processing unit 60 generates the G image IG and the B image IB by performing adjacent four-pixel mixing processing and phase shift processing on the RGB image Irgb. Further, the Z image IZ is generated by performing phase shift processing on the Z image IZ0 to shift the phase by one pixel in the horizontal direction and the vertical direction.

[0127] Specifically, from the RGB image Irgb, the G image IGF in which the number of frame pixels is quadrupled by upsampling processing and the subsequent LPF, and the B image IBF in which the number of frame pixels is also quadrupled are generated. Further, the G image IGF and the B image IBF are respectively subjected to the preceding LPF and downsampling processing through two different signal paths, so that the G image IG in which the phase is shifted by one pixel in the horizontal direction and the B image IB in which the phase is shifted by one pixel in the vertical direction are generated. Also, the Z image IZ0 is subjected to upsampling processing and the subsequent LPF to generate the Z image IZF in which the number of frame pixels is quadrupled. Further, the Z image IZF is subjected to the preceding LPF and downsampling processing, so that the Z image IZ having the same number of frame pixels as the X image IX and having the phase shifted by one pixel of the image sensor 31 in the horizontal direction and the vertical direction with respect to the X image IX is generated.

[0128] Then, the rearrangement processing unit 53 in the signal processing unit 60 arranges the pixel values (pixel data) of the pixels of the four images IX, IG, IB, and IZ in order according to the positions on one frame corresponding to their respective phases. As a result, a four-band one-frame image Ixzgb in which the pixel data of the four images IX, IG, IB, and IZ are rearranged in a predetermined pixel arrangement (four-color four-pixel arrangement) within one frame is generated and output from the signal processing unit 60.

[0129] <Image Processing of the Second Embodiment> Next, with reference to FIG. 13, the control of the control unit 40 and the image processing of the image processing unit 50 will be described in detail. The signal processing unit 60 shown in FIG. 13 shows only the configuration used in the second embodiment. For this reason, in FIG. 13, the illustration of these processing units 64, 68, etc. is omitted from the signal path where the upsampling processing unit 64 and the downsampling processing unit 68 are turned off.

[0130] In the sensor unit 31S, imaging is performed M times in a time-division manner. The adjacent pixel mixing processing unit 51S (see FIG. 2) mounted on the image sensor 31 performs adjacent pixel mixing processing on the imaging signal S1 of the sensor unit 31S as necessary. From the image sensor 31, an X image, an RGB image, and a Z image are sequentially input to the signal processing unit 60 as M images corresponding to the imaging signal S1 in series.

[0131] The M frames serially input from the image sensor 31 to the signal processing unit 60 are, as shown in FIG. 12, an X image IX subjected to adjacent pixel mixing processing, an RGB image Irgb not subjected to adjacent pixel mixing processing, and a Z image IZ0 subjected to adjacent pixel mixing processing. The X image IX does not require adjacent pixel mixing processing and phase shift processing. The RGB image Irgb requires adjacent pixel mixing processing and phase shift processing. The Z image IZ0 is an image subjected to adjacent pixel mixing processing, but phase shift processing is required. Therefore, the Z image IZ0 requires adjacent pixel mixing processing for the purpose of phase shift processing.

[0132] Therefore, as shown in FIG. 13, the X image is input to the rearrangement processing unit 53 after being written into the frame memories 61 and 62. The RGB image is temporarily held in the frame memory 61 and then undergoes image processing by the upsampling processing unit 64, the subsequent-stage LPF 65, the preceding-stage LPF 67, and the downsampling processing unit 68 through two signal paths. As a result, from the RGB image, a G image and a B image in which four adjacent pixels are mixed in a state where the phase is shifted by one pixel of the image sensor 31 in the horizontal and vertical directions with respect to the X image IX are generated and input to the rearrangement processing unit 53. Also, the Z image undergoes image processing by the upsampling processing unit 64, the subsequent-stage LPF 65, the preceding-stage LPF 67, and the downsampling processing unit 68. By this image processing, the Z image is converted into an adjacent four-pixel mixed Z image in which the phase is shifted by one pixel of the image sensor 31 in the horizontal and vertical directions with respect to the X image IX.

[0133] The rearrangement processing unit 53 rearranges the pixel data (pixel values) of the X image, G image, B image, and Z image input at the same timing into a predetermined pixel array (four-color four-pixel array) within one frame. As a result, the rearrangement processing unit 53 generates and outputs a four-band one-frame image Ixzgb.

[0134] Here, referring to FIG. 14, the images (frames) generated at each of the stages (a) to (l) shown in the image processing unit 50 in FIG. 13 and the image processing for generating the images will be described. FIG. 14(a) is an X image IX (frame IX) subjected to adjacent four-pixel mixing processing and input from the image sensor 31 to the signal processing unit 60.

[0135] FIG. 14(b) is an RGB image Irgb. The next FIG. 14(c) is a G image IGU in which upsampling processing is performed on the RGB image Irgb and the pixel value of the G pixel among each pixel is extracted, and the number of frame pixels is equal to the number of pixels of the image sensor 31.

[0136] FIG. 14(d) is a G image IGF generated by subjecting the G image IGU to a subsequent LPF process. The G image IGF is an image having the same number of frame pixels as the number of pixels of the image sensor 31.

[0137] FIG. 14(e) shows that when this G image IGF is subjected to a preceding LPF process and a downsampling process, a G image IG is obtained in which the phase is shifted by one pixel of the image sensor 31 in the horizontal direction with respect to the X image IX, and four adjacent pixels are mixed in a state where they are adjacent to each other, and the number of frame pixels is 1 / 4 of the number of pixels of the image sensor 31.

[0138] Also, FIG. 14(f) is a B image IBU in which an upsampling process is performed on the RGB color image Irgb and the pixel value of the B pixel among each pixel is extracted, and the number of frame pixels is equal to the number of pixels of the image sensor 31.

[0139] FIG. 14(g) is a B image IBF obtained by subjecting the B image IBU to a subsequent LPF process. The B image IBF is an image having the same number of frame pixels as the number of pixels of the image sensor 31. FIG. 14(h) shows that when this B image IBF is subjected to a preceding LPF process and a downsampling process, a B image IB is generated in which the phase is shifted by one pixel of the image sensor 31 in the vertical direction with respect to the X image IX, and four adjacent pixels are mixed in a state where they are adjacent to each other. The B image IB is an image having 1 / 4 of the number of frame pixels of the image sensor 31.

[0140] Also, FIG. 14(i) is an adjacent four-pixel mixed Z image IZ0 input from the image sensor 31 to the signal processing unit 60. FIG. 14(j) is a Z image IZU obtained by subjecting the Z image IZ0 to an upsampling process and having the same number of frame pixels as the number of pixels of the image sensor 31.

[0141] FIG. 14(k) is a Z image IZF generated by subjecting the Z image IZU to a subsequent LPF process. The Z image IZF is an image having the same number of frame pixels as the number of pixels of the image sensor.

[0142] As shown in FIG. 14(l), the Z image IZF is subjected to the preceding LPF processing and downsampling processing, so that the phase is shifted by one pixel of the image sensor 31 in both the horizontal and vertical directions with respect to the X image IX, and the adjacent four pixels are mixed to generate the Z image IZ. The Z image IZ is an image with a frame pixel number that is 1 / 4 of the pixel number of the image sensor 31.

[0143] Here, as shown in FIGS. 14(a), (e), (h), and (l), it is assumed that pixel values (pixel data) indicated by numbers are obtained in each pixel of the X image IX, G image IG, B image IB, and Z image IZ. The centers of each pixel of the images IX, IG, IB, and IZ are shifted. Specifically, the grid-like broken lines shown in FIGS. 14(a), (e), (h), and (l) are the positions of the original pixels corresponding to the pixels of the image sensor 31. Each pixel of the images IX, IG, IB, and IZ shifts the phase so that the central position relationship of each pixel does not collapse when rearranged into a 4-color 4-pixel array. All the pixels of the four images IX, IG, IB, and IZ have different phases (central coordinates).

[0144] Then, by assigning the pixel values of all the pixels of the four images IX, IG, IB, and IZ to the pixel positions corresponding to the Bayer array within one frame of the rearrangement destination, the rearrangement of the pixel data is performed. Thus, the 4-band 1-frame image Ixzgb shown in FIG. 14 is generated.

[0145] As shown in FIG. 15, the required time Txzgb for generating the 1-frame image Ixzgb is slightly longer than the required time Trgb of the comparative example for generating the RGB color image. This is due to the fact that the exposure time Texp_w requires about 4 times of Texp_x, and the output time Tout_w requires about 4 times of Tout_x. However, the required time Txzgb is less than 1.5 times the required time Trgb of the comparative example.

[0146] When the emission intensity of at least the W light source 22W among the plurality of light sources constituting the illumination unit 20 is individually adjusted, the relationship of the exposure time Texp_w = 4 * Texp_x is lost, and it becomes possible to set Texp_w = Texp_x. In this case, the required timeTxzgb can be shortened compared to the required time Trgb in the comparative example.

[0147] (Third Embodiment) Next, the third embodiment will be described with reference to FIGS. 16 and 17. The third embodiment is an example of the third mode, and generates a 4-band 1-frame image Ixrgb of a 4-color 4-pixel array. The image sensor 31 corresponds to 4-pixel mixing processing, but is an example that does not correspond to the phase shift function. Two types (M = 2) of light sources 21 and 22 are used. Specifically, they are the X light source 21 and the W light source 22W having the emission spectrum characteristics of FIG. 7(b).

[0148] As shown in FIGS. 16 and 17, the illumination unit 20 causes the X light source 21 and the W light source 22W to emit light in time division in this order based on the imaging trigger of the light emission control signal LC from the control unit 40. The image sensor 31 images (exposes) the subject 12 irradiated with X light and W light sequentially twice (M = 2) in accordance with the light emission timing. From the sensor unit 31S, a 4-pixel mixed X image IX and an RGB image Irgb that are not pixel-mixed are sequentially output. The signal processing unit 60 writes the X image IX into the frame memory 61.

[0149] At the timing when the two images are aligned, the signal processing unit 60 performs adjacent 4-pixel mixing processing and phase shift processing on the RGB image Irgb to generate a G image IG, a B image IB, and an R image IR. Specifically, when the RGB image Irgb passes through three signal paths of the signal processing unit 60, each RGB image Irgb is subjected to upsampling and subsequent LPF processing, thereby generating a G image IGF, a B image IBF, and an R image IRF whose frame pixel numbers are equal to the pixel number of the image sensor 31, respectively.

[0150] Furthermore, when the G image IGF, B image IBF, and R image IRF each pass through three different signal paths, each of them is subjected to the processes of a front-stage LPF and downsampling. As a result, a G image IG, a B image IB, and an R image IR are generated, in which adjacent four pixels are mixed in a state where the phase is shifted with respect to the X image IX in different directions. The G image IG is an image in which adjacent four pixels are mixed in a state where the phase is shifted by one pixel of the image sensor 31 in the horizontal direction with respect to the X image IX. The B image IB is an image in which adjacent four pixels are mixed in a state where the phase is shifted by one pixel of the image sensor 31 in the vertical direction with respect to the X image IX. The R image IR is an image in which adjacent four pixels are mixed in a state where the phase is shifted by one pixel of the image sensor 31 in both the horizontal and vertical directions with respect to the X image IX.

[0151] Then, the rearrangement processing unit 53 rearranges the pixel values (pixel data) of each pixel of the four images IX, IG, IB, and IR in order to the coordinate positions within one frame according to their respective phases. As a result, a 4-band one-frame image Ixrgb is generated in which each pixel of the four images IX, IG, IB, and IR is rearranged in a 4-color 4-pixel array within one frame, and is output from the signal processing unit 60.

[0152] As shown in FIG. 17, the required time Txrgb for generating the one-frame image Ixrgb is slightly longer than the required time Trgb of the comparative example for generating an RGB color image. This is due to the fact that the exposure time Texp_w requires about four times that of Texp_x, and the output time Tout_w requires about four times that of Tout_x. However, the required time Txrgb is less than 1.5 times the required time Trgb of the comparative example. When the emission intensity of at least the light source 22W among the plurality of light sources constituting the illumination unit 20 is individually adjusted, the relationship of the exposure time Texp_w = 4 * Texp_x is eliminated, and it becomes possible to set Texp_w = Texp_x. In this case, the required time Txrgb can be shortened compared to the required time Trgb of the comparative example.

[0153] (Fourth Embodiment) Next, in the fourth mode, a fourth embodiment will be described in which the signal processing unit 60 outputs a three-band one-frame image Ixyz and an RGB image Irgb shown in FIG. 3(d).

[0154] In the first embodiment, a W light source 22W capable of emitting white light is added to the illumination unit 20 shown in FIGS. 1, 4, and 5. The control unit 40 causes the X light source 21, the Y light source 22, the Z light source 23, and the W light source 22W to emit light in a time-division manner, and causes the image sensor 31 to perform imaging four times. The signal processing unit 60 outputs the RGB image Irgb input from the image sensor 31 simultaneously with the three-band one-frame image Ixyz (see FIG. 8) generated by the same image processing as in the first embodiment.

[0155] (Fifth Embodiment) Next, the fifth embodiment will be described with reference to FIGS. 18 and 19. The fifth embodiment is an example of the first mode similar to the first embodiment, but the image sensor 31 does not include the adjacent pixel mixing processing unit 51S, and the signal processing unit 60 performs adjacent pixel mixing processing and phase shift processing. The imaging device 11 of this embodiment includes a signal processing unit 60 that processes an imaging signal corresponding to a frame output from the sensor unit 31S of the image sensor 31. The adjacent pixel mixing processing unit 51 is provided in at least the signal processing unit 60 among the camera 30 and the signal processing unit 60. That is, in FIG. 2, the adjacent pixel mixing processing unit 51 is provided in at least one of the signal processing unit 60 in the camera 30 and the signal processing unit 60 in the PC 70. When the camera 30 does not include the signal processing unit 60, the PC 70 includes the signal processing unit 60.

[0156] The control unit 40 causes the adjacent pixel mixing processing unit 51D provided in the signal processing unit 60 to perform adjacent pixel mixing processing for each of a plurality of frames. The adjacent pixel mixing processing unit 51D performs the selected adjacent pixel mixing processing for each frame.

[0157] In this embodiment, the signal processing unit 60 is selected by the control unit 40, and the adjacent pixel mixing processing unit 51 in the signal processing unit 60 performs adjacent pixel mixing processing for each frame on a plurality of frames output from the sensor unit 31S.

[0158] As shown in FIGS. 18 and 19, the illumination unit 20 causes the X light source 21, the Y light source 22, and the Z light source 23 to emit light in a time-division manner in this order based on the imaging trigger of the light emission control signal LC from the control unit 40. The image sensor 31 captures (exposes) the subject 12 irradiated with the X light, the Y light, and the Z light in sequence three times (M = 3) in accordance with the light emission timing. From the sensor unit 31S, the X image IXrgb, the Y image IYrgb, and the Z image IZrgb in which adjacent pixels are not mixed are sequentially output. The signal processing unit 60 sequentially writes the X image IXrgb into two frame memories 61 and 62, and writes the Y image IYrgb into one frame memory 61.

[0159] At the timing when the three images are aligned, the signal processing unit 60 performs image processing on the three images IXrgb, IYrgb, and IZrgb. The three images IXrgb, IYrgb, and IZrgb are input to the upsampling processing unit 64 through three different signal paths respectively. Since the number of frame pixels is the same as the number of pixels of the image sensor 31, it is output as it is without upsampling processing. Next, frequency components with a period of two pixels or more of the image sensor 31 are removed by the subsequent LPF 65. As a result, the X image IXF, the Y image IYF, and the Z image IZF are generated. These three images IXF, IYF, and IZF are input to the downsampling processing unit 68 via the preceding LPF 67 respectively. In the downsampling processing unit 68, the number of frame pixels is reduced to 1 / 4 by adjacent pixel mixing processing. As a result, the X image IX is generated from the X image IXF, two Y images IY1 and IY2 are generated from the Y image IYF, and the Z image IZ is generated from the Z image IZF.

[0160] Then, the rearrangement processing unit 53 rearranges the pixel data (pixel values) of the four images IX, IY1, IY2, and IZ in a three-color four-pixel array within one frame. Thereby, a three-band one-frame image Ixyz is generated and output from the signal processing unit 60.

[0161] As shown in FIG. 19, the required time Txyz for generating one-frame image Ixyz requires approximately twice the required time Trgb of the comparative example for generating an RGB image. This is because the exposure times Texp_x, Texp_y, and Texp_z each require approximately four times the exposure time Texp_x of the first embodiment, and the output times Tout_x, Tout_y, and Tout_z require approximately four times the output time Tout_x of the first embodiment. Although the required time Txyz becomes longer than that of the comparative example in this way, the imaging device 11 can acquire a multi-band image of the subject 12 imaged by one camera 30. Note that the light emission intensities of the plurality of light sources 21 to 23 constituting the illumination unit 20 may be individually adjustable, and the required time Txrgb may be shortened by adjusting the light emission intensity upward.

[0162] (Operation of the Embodiment) Next, the operations of the imaging device 11 and the inspection device 10 will be described. As shown in FIG. 1, for example, when the conveyed subject 12 is detected by a sensor, based on this detection, the control unit 40 causes M types (2 ≤ M ≤ N) of light sources corresponding to the mode among N types (N ≥ 2) of light sources having different light emission characteristics constituting the illumination unit 20 to emit light in a time-division manner (light emission step). Further, the control unit 40 causes the image sensor 31 to perform imaging (exposure) M times in accordance with the light emission timings of the M types of light sources 21 to 23 to acquire an imaging signal S1 corresponding to a plurality of frames (imaging step). This imaging signal S1 is output from the sensor unit 31S of the image sensor 31.

[0163] For M frames, adjacent pixel mixing processing by the adjacent pixel mixing processing unit 51 is performed on one or both of the image sensor 31 and the signal processing unit 60 according to the mode, settings, etc. The adjacent pixel mixing processing unit 51 performs adjacent pixel mixing processing for each frame on a plurality of frames (adjacent pixel mixing processing step). Further, resampling processing by the resampling processing unit 52 is performed on the M frames after the adjacent pixel mixing processing. In the resampling processing, the center position of the pixels is shifted with respect to the frames after the adjacent pixel mixing processing (resampling processing step). Next, rearrangement processing by the rearrangement processing unit 53 is performed on the M frames after the resampling processing. In the rearrangement processing, the pixel data (pixel values) for a plurality of frames on which the resampling processing has been performed are rearranged in a predetermined pixel arrangement within one frame (rearrangement processing step). Thus, a multi-band one-frame image is output as one-frame imaging data from the signal processing unit 60.

[0164] For example, in the first mode, a 3-band one-frame image Ixyz (first and fifth embodiments) is output from the signal processing unit 60. In the second mode, a 4-band one-frame image Ixzgb (second embodiment) is output from the signal processing unit 60. In the third mode, a 4-band one-frame image Ixrgb (third embodiment) is output from the signal processing unit 60. In the fourth mode, a one-frame image Ixyz and an RGB image Irgb (fourth embodiment) are output from the signal processing unit 60.

[0165] The multi-band one-frame image from the signal processing unit 60 is output to the pixel array conversion unit 55. The pixel array conversion unit 55 converts a k-band one-frame image (k≥2) into a plurality of (k) images. The k images are multi-spectral images. The k images are temporarily stored in the VRAM 73 (see FIGS. 4 and 4).

[0166] When the object 12 is an article handled at a manufacturing factory or the like, the imaging device 11 is applied to various devices. In the example shown in FIG. 1, the imaging device 11 is applied to the inspection device 10. The k images converted by the pixel array conversion unit 55 are read from the VRAM 73 and sent to the inspection processing unit 74. The inspection processing unit 74 inspects the object 12 based on the k images input from the imaging device 11. At least one of the k images can inspect contents that cannot be inspected with an RGB color image. The at least one image refers to an image captured with X-ray, Y-ray, Z-ray having a light frequency other than RGB, etc. In particular, the X-ray shown in FIG. 7(a), the X-ray and Z-ray shown in FIG. 7(b) are near-infrared light.

[0167] According to the first embodiment described in detail above, the following effects can be obtained. (1-1) The imaging device 11 includes a color image sensor 31 having a sensor unit 31S that images a subject 12, N types (where N is a natural number of 2 or more) of light sources 21 to 23 with different light emission characteristics for illuminating the subject 12, and a control unit 40. The control unit 40 causes the color image sensor 31 to perform imaging M times by causing M types (where M is a natural number satisfying 2 ≦ M ≦ N) of the N types of light sources 21 to 23 to emit light in a time-division manner. Further, the imaging device 11 includes an adjacent pixel mixing processing unit 51, a resampling processing unit 52, and a rearrangement processing unit 53. The adjacent pixel mixing processing unit 51 performs adjacent pixel mixing processing on each of the M frames output from the sensor unit 31S. The resampling processing unit 52 performs resampling processing for shifting the center position of pixels on the frame after the adjacent pixel mixing processing on each frame. The rearrangement processing unit 53 rearranges the pixel data for a plurality of frames that have been resampled in a predetermined pixel arrangement within one frame. According to this configuration, a multi-spectral image can be acquired by imaging the subject 12 using one general-purpose color image sensor 31. Therefore, a multi-spectral image of the subject 12 can be acquired with a simple configuration. For example, with a color camera 30 equipped with a general-purpose color image sensor 31, in addition to RGB color imaging and monochrome imaging, multi-spectral imaging is possible. For example, since a normal RGB color image that is not time-division imaging can also be output, compatibility with existing inspection processes can be maintained. Therefore, with a configuration compatible with existing devices, multi-spectral image characteristics different from the image characteristics obtained by existing devices can be selected, and it becomes easy to change the device characteristics.

[0168] (1-2) It includes a signal processing unit 60 that processes an imaging signal S1 corresponding to a frame output from the sensor unit 31S of the color image sensor 31. The adjacent pixel mixing processing unit 51 is provided at least in the signal processing unit 60 among the color image sensor 31 and the signal processing unit 60. For a plurality of frames, either one or both of the color image sensor 31 and the signal processing unit 60 are selected for each frame, and the adjacent pixel mixing processing unit 51 provided in the selected one or both performs adjacent pixel mixing processing for each frame. In the adjacent pixel mixing processing, the adjacent pixels to be mixed are selected for each frame. According to this configuration, the required time for the imaging signal S1 to be output from the color image sensor 31 is shortened, enabling high-speed processing. Also, even when there are limitations in the adjacent pixel mixing processing function of the color image sensor 31 or when it does not have an adjacent pixel mixing processing function, adjacent pixel mixing processing, resampling processing for reducing deterioration of resolution characteristics, and rearrangement processing can be performed.

[0169] (1-3) The pixel array of the color filter 34 constituting the color image sensor 31 is a Bayer array. The rearrangement processing unit 53 rearranges the pixel data for a plurality of frames into a 3-color 4-pixel array same as the Bayer array within one frame. For the color camera 30 having the color image sensor 31 that employs a general-purpose pixel array such as the Bayer array, a general-purpose pixel array conversion unit 55 (for example, a Bayer array conversion unit) for converting the imaging signal (frame) output from the color camera 30 into a plurality of color-separated images is provided. Therefore, by rearranging the pixel data for a plurality of frames into a 3-color 4-pixel array same as the Bayer array within one frame, a 3-band 1-frame image Ixyz, which is an example of 1-frame imaging data, is generated. Thus, when converting the 3-band 1-frame image Ixyz, etc. into three images SI1 to SI3, the existing pixel array conversion unit 55 can be used.

[0170] (1-4) The pixel array of the color filter 34 that constitutes the color image sensor 31 is a Bayer array. The control unit 40 causes the color image sensor 31 to perform imaging a plurality of times in a time-division manner by sequentially illuminating the subject 12 with the light from the light sources 21 to 23 having different emission characteristics. For each of the plurality of frames output from the sensor unit 31S of the color image sensor 31, adjacent pixel mixing processing and resampling processing are performed frame by frame. The rearrangement processing unit 53 performs rearrangement processing to rearrange the pixel data for four frames obtained by performing these processes into a four-color four-pixel array with the same four pixels as the Bayer array as a unit within one frame, thereby generating one-frame imaging data of four bands. According to this configuration, it is possible to acquire one-frame imaging data of four bands, which is a larger number of bands than the number of colors of the Bayer array, which is "3".

[0171] (1-5) The N types of light sources 21 to 23 etc. include a plurality of light sources among the visible light irradiating light sources 22, 23, 22W and the near-infrared light irradiating light sources 21, 23Z. The control unit 40 selects M types of light sources corresponding to the mode from among the N types of light sources 21 to 23 etc., and causes the selected M types of light sources to emit light in a time-division manner, thereby causing the color image sensor 31 to perform imaging a plurality of times in a time-division manner. The rearrangement processing unit 53 outputs one-frame imaging data of a type corresponding to the selected mode. The mode includes at least one of a mode for outputting 3-band or 4-band one-frame imaging data including one or more bands of pseudo-color pixel data having wavelengths within the light wavelength region from visible light to near-infrared light, and a mode for outputting 3-band or 4-band one-frame imaging data including pixel data of at least one color among RGB pixel data and near-infrared light pixel data. According to this configuration, the imaging device 11 can output at least one of 3-band or 4-band one-frame imaging data including one or more bands of pseudo-color and 3-band or 4-band one-frame imaging data including pixel data of at least one color among RGB and near-infrared light pixel data. Using the imaging device 11 including one general-purpose color camera 30 incorporating one general-purpose color image sensor 31, a plurality of types of images SI1 to SIk having different optical characteristics can be acquired from 3-band or 4-band one-frame imaging data including one or more bands of pseudo-color or near-infrared light.

[0172] For example, according to the first, fourth, and fifth embodiments, as an example of one-frame imaging data including pixel data of X, Y, Z pseudo-color images, a 3-band one-frame image Ixyz can be acquired. According to the second embodiment, as an example of one-frame imaging data including G, B pixel data and pixel data of X, Z pseudo-color images, a 4-band one-frame image Ixzgb can be acquired. According to the third embodiment, as an example of one-frame imaging data including RGB pixel data and pixel data of X pseudo-color images, a 4-band one-frame image Ixrgb can be acquired. By converting the 3-band or 4-band one-frame images Ixyz, Ixzgb, Ixrgb of the first to fifth embodiments by the pixel array conversion unit 55, three or four images SI1 to SIk can be acquired.

[0173] (1-6) The inspection device 10 includes an imaging device 11 and an inspection processing unit 74 that inspects the subject 12 based on the image output by the imaging device 11. According to this configuration, various inspections can be performed on the subject 12 using a plurality of types of captured images.

[0174] (1-7) The imaging method for imaging the subject 12 includes an illumination step, an imaging step, an adjacent pixel mixing process step, a resampling process step, and a rearrangement process step. In the illumination step, a plurality of light sources 21 to 23 with different emission characteristics are sequentially emitted to illuminate the subject 12 in a time-division manner. In the imaging step, the subject 12 illuminated in a time-division manner is imaged by the color image sensor 31 in a time-division manner to obtain a plurality of frames. In the adjacent pixel mixing process step, adjacent pixel mixing processing is performed for each frame on the plurality of frames. In the resampling process step, resampling processing for shifting the center position of the pixels is performed for each frame on the frames after the adjacent pixel mixing process. In the rearrangement process step, the pixel data for the plurality of frames after the resampling process is rearranged in a predetermined pixel arrangement within one frame. According to this imaging method, by imaging the subject 12 using one imaging device 11, a plurality of types of captured images with different optical characteristics can be obtained.

[0175] (1-8) According to the first to fifth embodiments, the adjacent pixel mixing processing unit 51S built in the image sensor 31 performs adjacent pixel mixing processing on at least one of the M frames. Therefore, the exposure time and image output time of the sensor unit 31S can be shortened compared to the full-pixel imaging under the same imaging conditions, and a high frame rate can be realized. Thus, although the image sensor 31 is configured to perform imaging a plurality of times in a time-division manner, the required times Txyz, Txzgb, and Txrgb are short.

[0176] The signal processing unit 60 processes and synthesizes a plurality of frames acquired by time-division imaging, rearranges pixel data for a plurality of frames in a predetermined pixel array within one frame, and generates and outputs a multi-band one-frame image. Therefore, pseudo-color processing and inspection processing compatible with conventional processing can be performed without being conscious of time-division imaging.

[0177] (1-10) By switching the light sources that emit light in time division among N types of light sources 21 to 23 that make up the illumination unit 20, etc., and only changing the settings of the signal processing unit 60, the multi-spectral imaging characteristics can be changed.

[0178] (1-11) By performing resampling processing, phase shift processing, and rearrangement processing on at least one frame out of M frames (images) by the signal processing unit 60, it is possible to reduce the deterioration of resolution characteristics due to adjacent 4-pixel mixing processing (4-pixel binning processing).

[0179] (Second Embodiment) Next, the second embodiment will be described with reference to FIGS. 20 and 21. When imaging is performed multiple times by time-division light emission, light from the previous imaging may enter the image of the next imaging. As a countermeasure, in the second embodiment, the influence of the light from the previous imaging is removed by image processing. Note that FIGS. 20 and 21 show an example of the first mode, but basically the same applies to other modes.

[0180] As shown in FIG. 20, the signal processing unit 60 includes a matrix operation unit 69 that performs matrix operations on the input M types of frames (imaging signals). The matrix operation unit 69 performs operations using a 3×3 matrix, and values that can remove the components of light (for example, X-rays) illuminated in the previous imaging are set as the coefficients of the matrix. Correction is performed by the operation processing between frames by matrix operations, and the components of the light remaining from the previous imaging are removed from the imaging signal.

[0181] For example, X image, Y image, and Z image, which are imaging signals output from the matrix operation unit 69, are input to the resampling processing unit 52. The resampling processing unit 52 performs the same image processing as in the first embodiment on the X image, Y image, and Z image to generate an X image IX, Y images IY1 and IY2, and a Z image IZ. The rearrangement processing unit 53 rearranges the pixel data of the X image IX, Y images IY1 and IY2, and Z image IZ in a three-color four-pixel array within one frame to generate a three-band one-frame image Ixyz.

[0182] As shown in FIG. 21, even when the emission of the X light source 21 stops, the X light remains for a while. Therefore, when the emission of the Y light starts and the next imaging starts, a small amount of X light is mixed in the Y light. At the initial stage of the second imaging start, the image sensor 31 is exposed not only by the Y light but also by the X light. Therefore, the X light affects the Y image, which is the sensor output image, at a ratio p. That is, the pixel values of the Y image increase by the amount affected by the X light. This Y image is denoted as image Y + p*X. The X image and the image Y + p*X written in the memory are read out from the frame memories 61 and 62 at the timing when the Z image is output, and image processing X, Y, and Z is performed on each image.

[0183] At this time, before the resampling process is performed by the resampling processing unit 52, a 3×3 matrix operation is performed on the X image, the image Y + p*X, and the Z image by the matrix operation unit 69. As a result, for the Y image, its pixel value is corrected to a smaller value by the ratio p affected by the X light. That is, the X-ray component p*X is removed from the pixel value Y + p*X of the Y image.

[0184] The matrix operation unit 69 is configured, for example, as a part of an FPGA. Therefore, a special illumination unit 20 equipped with a shutter is not required, and a multi-band one-frame image Ixyz, which is an example of high-quality one-frame imaging data, can be obtained.

[0185] (2-1) The N light sources 21 to 23, etc. include the light sources 22, 22W, 23Z which are visible light sources capable of covering the entire sensitivity wavelength range of the color image sensor 31, and the light sources 21, 23Z which are near-infrared light sources. The control unit 40 selects M light sources corresponding to the mode from among the N light sources 21 to 23, etc., and causes the selected M light sources to emit light in a time-division manner, thereby causing the color image sensor 31 to perform imaging in a time-division manner and output a plurality of frames. The rearrangement processing unit 53 performs correction by arithmetic processing between frames to reduce the influence of the afterglow of the light source on the previous frame entering the next frame on the next frame. According to this configuration, the influence of the afterglow of the illumination of the previous frame on the next frame can be reduced, and an image with high wavelength separation accuracy without mixing of emission wavelengths can be output.

[0186] The embodiment is not limited to the above, and may be changed to the following aspects. · In the first embodiment, when the phase shift function of the adjacent pixel mixing process is not installed, the following corresponding example can be adopted. Hereinafter, this corresponding example will be described with reference to FIGS. 8 and 10 of the first embodiment and FIGS. 12 to 15 of the second embodiment, etc. In the first embodiment, when the image sensor 31 does not have the adjacent two-pixel mixing process function and the phase shift function, the output from the image sensor 31 has the same adjacent four-pixel mixed pixel configuration as the X image IX (see FIG. 12) for both the X image, the Y image, and the Z image. Here, let the Y image which is the output from the image sensor 31 be the Y image IY0.

[0187] In this case, a relationship similar to that of the images IX and IZ0 output from the image sensor 31 in the second embodiment is established, and the Z image IZ can be generated from the Z image IZ0 via the Z image IZF. By performing the same type of processing on the Y image IY0 as the processing for this IZ0, the Y image IYF (FIGS. 8 and 10) can be generated from the Y image IY0. Then, by performing an adjacent 4-pixel pixel mixing process with a phase shift of one pixel of the image sensor 31 in the horizontal and vertical directions on the Y image IYF in the same manner as in the first embodiment, the Y images IY1 and IY2 can be generated. Then, in the same manner as in the first embodiment, if the pixel data of the X image IX, the Z image IZ with the pixel phase shifted, and the Y images IY1 and IY2 are rearranged in a 3-color 4-pixel array within one frame, a 3-band 1-frame image Ixyz can be obtained.

[0188] · In the second embodiment, as a measure to suppress the entry of light from the previous imaging into the image of the next imaging, a shutter may be provided in the illumination unit 20. The imaging device 11 includes a plurality of shutters that open and close the optical paths for each of the light sources 21 to 23. The control unit 40 controls the light emission of the light sources 21 to 23 and the opening and closing of each shutter based on the light emission control signal LC. The illumination unit 20 emits one light source indicated by the light emission control signal LC, and opens one shutter corresponding to the light source to be emitted among the plurality of shutters at the start of light emission and closes it at the end of light emission.

[0189] · The configuration of the imaging device 11 is not limited to the configurations of the above embodiments. For example, an imaging device 11 that does not have a plurality of modes and only outputs a plurality of band 1-frame images according to one mode may be used.

[0190] · The resampling process performed by the resampling processing unit 52 may be the nearest neighbor interpolation method (nearest neighbor), the bilinear interpolation method (bilinear), or the bicubic convolution interpolation method (bicubic).

[0191] · The adjacent pixel mixing processing unit 51D is not limited to a configuration that is part of the resampling processing unit 52 and may be a separate processing unit. · The arrangement positions of the plurality (N types) of light sources 21 to 23 etc. that make up the illumination unit 20 can be changed as appropriate. For example, a configuration in which the plurality (N types) of light sources are arranged adjacent to each other may be used, or a configuration in which the plurality of light sources are arranged separately at a position where the camera 30 can receive reflected light by irradiating from the front of the subject 12 and a position where the camera 30 can receive transmitted light that has passed through the subject 12 may also be used.

[0192] · The N types of light sources that make up the illumination unit 20 are not limited to 2, 3, or 4 types, and may be 5 types or more. Also, the M types are not limited to 2, 3, or 4 types, and may be 5 types or more. Here, M may be M < N. Also, M is not limited to "3" which is the same number as the number of colors in the Bayer array or "4" which is the number of pixels in the Bayer array, and M = 2 or M = 5 may also be used. In this case, the pixel data for a plurality of frames may be rearranged in a predetermined pixel arrangement other than the Bayer array.

[0193] · All of the M types of light sources may be near-infrared light sources. Also, all of the M types of light sources may be light sources that can emit light other than RGB among the light in the visible light region. Note that it is sufficient if at least one of the M types of light sources is a light source that can emit light other than R light, G light, B light, and light that is a combination of these three types of light.

[0194] · The signal processing unit 60 may be configured to include only the rearrangement processing unit 53. For example, in the first embodiment, the adjacent pixel mixing processing unit 51S of the image sensor 31 generates the X image IX, the Y image IY1 (or IY2), and the Z image IZ. The rearrangement processing unit 53 rearranges the pixel data of the three images IX, IY1 (or IY2), and IZ into 3 out of 4 pixels similar to the Bayer array. In this case, the pixel data may not be rearranged into the unused 1 pixel out of the 4 pixels, or dummy pixel values may be arranged. In this configuration, the adjacent pixel mixing processing unit 51 and the resampling processing unit 52 are incorporated in the image sensor 31.

[0195] · One of the time-division light emissions may be one in which a plurality of light sources are combined and emitted simultaneously. · The light source may be configured to emit light having an emission spectrum of a desired frequency by irradiating the subject 12 with the emitted light through an optical band-pass filter.

[0196] · Save the data of an image (for example, an X image, a Y image, and a Z image) based on the imaging signal captured by the color camera 30 including the image sensor 31 in a removable memory such as a USB memory. Let the personal computer read the image data stored in the removable memory, and the signal processing unit 60 of the personal computer may perform image processing on the X image, the Y image, and the Z image to generate one-frame imaging data of a plurality of bands. That is, the device including the camera 30 that performs the imaging step and the device including the signal processing unit 60 that performs the image processing step may be separate devices. Also by this imaging method, one-frame imaging data of a plurality of bands can be acquired.

[0197] · The arrangement pattern of the color filter 34 constituting the image sensor 31 is not limited to the RGB Bayer array, and may be any arrangement pattern such as a stripe array. Also, the number of colors of the color filter 34 constituting the image sensor 31 is not limited to three colors, and may be four colors or five colors.

[0198] · A color camera 30 including a color image sensor 31 having a complementary color filter 34 may be used. In this case, the color filter 34 may be a complementary color filter of magenta (Mg), yellow (Ye), and cyan (Cy) instead of the RGB primary color filter. The complementary colors may be four colors: yellow, cyan, magenta, and green. Further, the color filter 34 may have a configuration combining a complementary color filter and a primary color filter.

[0199] · The application range of the imaging device 11 is not limited to inspection devices, and may be applied to various devices used in the manufacturing process. For example, the imaging device 11 may be used for various devices that perform positioning, removal of adherents and foreign objects, component assembly, etc.

[0200] · The imaging device 11 may be used as the eyes of a robot, and the robot may be configured to recognize the subject 12 based on a multi-band 1-frame image output from the imaging device 11. · A configuration may be adopted in which an inspector visually checks a plurality of images obtained by converting the multi-band 1-frame image output from the imaging device 11 by the pixel array conversion unit 55, and performs processing or inspection on the subject 12.

[0201] · The subject 12 may be, for example, a container, food, beverage, electronic component, household electrical appliance, daily necessity, part, member, raw material such as powder or liquid, etc. Further, the subject 12 may be, for example, a container containing liquid, food ingredients such as fruits and vegetables, plants such as flowers, semi-processed or processed food of plant or animal origin, organisms, etc. Further, the photograph (image) of the subject may be a photograph of a building, a topographic photograph of aerial photography, a photograph of the sky or celestial bodies, a microscopic photograph, etc.

[0202] · At least one of the control unit 40, the image processing unit 50, and the inspection processing unit 74 may be partially or entirely configured by software executed by a computer that executes a program, or may be configured by hardware such as an electronic circuit. At least a part of the image processing unit 50 is not limited to hardware such as an FPGA.

Explanation of Reference Numerals

[0203] 10…Inspection device, 11…Imaging device, 12…Object, 20…Illumination unit, 21…First light source (X light source) as an example of a light source, 22…Second light source (Y light source) as an example of a light source, 23…Third light source (Z light source) as an example of a light source, 22W…White light source (W light source) as an example of a light source, 23Z…Z light source as an example of a light source, 30…Color camera (camera), 30a…Lens barrel, 31…Color image sensor (image sensor), 31S…Sensor unit, 32…Lens, 33R…R light receiving element, 33G…G light receiving element, 33B…B light receiving element, 34…Color filter, 34R…R filter, 34G…G filter, 34B…B filter, 35…IR cut filter, 40…Control unit, 41…Mode setting unit, 42…Trigger control unit, 43…Selection unit, 44…Binning setting unit, 45…Exposure time setting unit, 50…Image processing unit, 51, 51S,51D... Adjacent pixel mixing processing unit, 52... Resampling processing unit, 53... Rearrangement processing unit, 55... Pixel array conversion unit, 60... Signal processing unit, 61... First frame memory (frame memory), 62... Second frame memory (frame memory), 63... First selector, 64... Upsampling processing unit, 65... Post-stage LPF, 66... Second selector, 67... Pre-stage LPF, 68... Downsampling processing unit, 70... PC, 71... CPU, 72... Image input board, 73... VRAM, 74... Inspection processing unit, 75... Input unit, 76... Display unit, S1... Imaging signal (frame), IC... Imaging control signal, PS... Shift signal, VA... Visible light wavelength region, NIRA... Near-infrared wavelength region, S... Sensitivity curve, Ixyz... 3-band 1-frame image as an example of 1-frame imaging data, Ixzgb... 4-band 1-frame image as an example of 1-frame imaging data, Ixrgb... 4-band 1-frame image as an example of 1-frame imaging data, Ixyz... 3-band 1-frame image as an example of 1-frame imaging data, Irgb... RGB image, Iwb... Black-and-white image, SI1... X pseudo-color image, SI2... Y pseudo-color image, SI3... Z pseudo-color image, IX... X image (frame), IY0... Y image (frame), IY1... Y image (frame), IY2... Y image (frame), IZ... Z image (frame), IZ0... Z image (frame), IG... G image (frame), IB... B image (frame), IR... R image (frame), Irgb... RGB image (frame), IXrgb... X image (frame), IYrgb... Y image (frame), IZrgb... Z image (frame), Txyz, Txzgb, Txrgb... Required time, Texp, Texp_x, Texp_y, Texp_z... Exposure time, Tout, Tout_x, Tout_y, Tout_z... Output time.,

Claims

1. An imaging device for imaging a subject, comprising: a color image sensor having a sensor unit for imaging the subject; N types (where N is a natural number of 2 or more) of light sources with different light emission characteristics for illuminating the subject; a control unit that causes the color image sensor to perform imaging M times by causing M types (where M is a natural number satisfying 2 ≤ M ≤ N) of the N types of light sources to emit light in a time-division manner; an adjacent pixel mixing processing unit that performs adjacent pixel mixing processing for each of the M frames output from the sensor unit; a resampling processing unit that performs resampling processing for shifting the center position of pixels for each frame on the frames after the adjacent pixel mixing processing; a rearrangement processing unit that rearranges pixel data for a plurality of frames subjected to the resampling processing into pixels within one frame; An imaging device characterized by comprising the above.

2. Comprising a signal processing unit that processes an imaging signal corresponding to the frame output from the sensor unit of the color image sensor, The adjacent pixel mixing processing unit is provided at least in the signal processing unit among the color image sensor and the signal processing unit, For the plurality of frames, either one or both of the color image sensor and the signal processing unit are selected for each frame, and the adjacent pixel mixing processing unit provided in the selected one or both performs the adjacent pixel mixing processing for each frame, The imaging device according to claim 1, wherein in the adjacent pixel mixing processing, adjacent pixels to be mixed are selected for each frame.

3. The pixel arrangement of the color filter constituting the color image sensor is a Bayer array, The imaging device according to claim 1 or 2, wherein the rearrangement processing unit rearranges the pixel data for the plurality of frames into a 3-color 4-pixel arrangement same as the Bayer array within one frame.

4. The pixel arrangement of the color filter constituting the color image sensor is a Bayer array, The control unit causes the color image sensor to perform imaging a plurality of times in a time-division manner by sequentially illuminating the subject with light from the light sources having different light emission characteristics. The rearrangement processing unit performs rearrangement processing of rearranging, within one frame, the pixel data for four frames obtained by performing the adjacent pixel mixing process and the resampling process for each of the plurality of frames into a four-color four-pixel array with the same four pixels as the Bayer array as a unit, thereby generating one-frame imaging data of four bands. The imaging device according to any one of claims 1 to 3, characterized in that.

5. The N types of light sources include a light source capable of irradiating visible light and a light source capable of irradiating near-infrared light. The control unit selects the M types of light sources corresponding to the mode among the N types of light sources, and causes the color image sensor to perform imaging a plurality of times in a time-division manner by causing the selected M types of light sources to emit light in a time-division manner. The rearrangement processing unit outputs one-frame imaging data of a type corresponding to the selected mode. The mode includes at least one of a mode of outputting one-frame imaging data of three bands or four bands including one or more bands of pseudo-color pixel data having a wavelength within the light wavelength region from visible light to near-infrared light, and a mode of outputting one-frame imaging data of three bands or four bands including pixel data of at least one color among RGB pixel data and near-infrared light pixel data. The imaging device according to any one of claims 1 to 4, characterized in that.

6. The N types of light sources include a visible light source and a near-infrared light source capable of covering the entire sensitivity wavelength region of the color image sensor. The control unit selects the M types of light sources corresponding to the mode among the N types of light sources, and causes the color image sensor to perform imaging in a time-division manner by causing the selected M types of light sources to emit light in a time-division manner, thereby outputting a plurality of frames. The rearrangement processing unit performs correction for reducing the influence of the afterglow of the light source at the time of imaging of the previous frame on the next frame due to the entry of the afterglow of the light source at the time of imaging of the previous frame into the next frame by arithmetic processing between frames. The imaging device according to any one of claims 1 to 5, characterized in that.

7. An imaging device according to any one of claims 1 to 6, An inspection processing unit that inspects the subject based on an image output by the imaging device An inspection device, characterized in that it comprises.

8. An imaging method for imaging a subject, An illumination step of sequentially emitting a plurality of light sources having different emission characteristics to illuminate the subject in a time-division manner. An imaging step of causing a color image sensor to image the subject illuminated in a time-division manner in a time-division manner to obtain a plurality of frames; An adjacent pixel mixing process step of performing an adjacent pixel mixing process for each of the plurality of frames; A resampling process step of performing a resampling process for shifting the center position of a pixel for each of the frames on the frame after the adjacent pixel mixing process; A rearrangement process step of rearranging pixel data for a plurality of frames subjected to the resampling process into pixels within one frame; An imaging method characterized by including the above steps.

Images