Image pickup device, image pickup system and image pickup method

Abstract

The present invention provides an imaging device, comprising an imaging unit (130) and a synthesis unit (140). The imaging unit (130) generates a frame of image by sequentially receiving reflected lights reflected by a moving subject when the subject is intermittently and sequentially irradiated with irradiation lights having different wavelengths according to the position of the moving subject, temporarily and sequentially retaining signal information based on the reflected lights of each wavelength, and reading out the retained signal information together. The synthesis unit (140) generates a synthesized image by cutting out a subject image corresponding to the reflected lights of each wavelength from the frame of image, and superimposing a plurality of cut subject images.

Description

Technical Field

[0001] The invention relates to an imaging device, an imaging system and an imaging method. Background Art

[0002] At a production site or the like, when a product is shipped, an inspection device that checks the appearance of the product based on a captured image is used. For example, the inspection device can check whether the product is a good product or a bad product based on a captured image of the appearance of the product obtained by an RGB camera. However, for an RGB camera that detects light with a wide range of wavelengths such as the three primary colors of RGB, it may be difficult to accurately capture the appearance of the product from the captured image obtained by the RGB camera. Therefore, in order to more accurately capture the appearance of the product, a technology using a spectral image has been proposed, which is obtained by finely dividing the wavelength of light related to a subject image into multiple wavelengths and thereby detecting the wavelength. More specifically, its purpose is to more accurately capture the appearance of the product by synthesizing the multiple spectral images obtained as described above into one image and using the synthesized image. For example, as an example of a camera device for acquiring a spectral image, the camera devices disclosed in Patent Documents 1 and 2 can be cited.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2015-41784A

[0006] Patent Document 2: Japanese Patent Application Laid-Open No. 2015-126537A Summary of the invention

[0007] [Technical issues to be solved]

[0008] However, in the imaging devices disclosed in Patent Document 1 and Patent Document 2, it is difficult to avoid complicating the configuration, and further, it is difficult to suppress an increase in the processing time taken when obtaining one composite image.

[0009] Therefore, the present invention proposes an imaging device, an imaging system, and an imaging method that can obtain a composite image of spectroscopic images with a simple configuration and at high speed.

[0010] [Solutions to technical problems]

[0011] According to the present invention, there is provided an imaging device, which includes an imaging unit and a synthesis unit. The imaging unit generates a frame of image by sequentially receiving each reflected light reflected by the subject when the subject is intermittently and sequentially irradiated with each irradiation light having different wavelengths according to the position of the moving subject, temporarily and sequentially retaining each signal information based on the reflected light of each wavelength, and reading out each retained signal information together. The synthesis unit generates a synthesized image by cutting out the subject image corresponding to the reflected light of each wavelength from the one frame of image, and superimposing the cut-out multiple subject images.

[0012] According to the present invention, there is provided an imaging device, which includes an imaging unit and a synthesis unit. The imaging unit generates a frame of image by sequentially receiving each reflected light reflected by the subject when the subject is intermittently and sequentially irradiated with each irradiation light having different wavelengths according to the position of the moving subject, temporarily and sequentially retaining each signal information based on the reflected light of each wavelength, and reading out a part of the signal information corresponding to a plurality of predetermined areas specified in advance among the retained each signal information. The synthesis unit generates a synthesized image by cutting out the subject image corresponding to the reflected light of each wavelength from the one frame of image, and superimposing the cut-out plurality of subject images.

[0013] In addition, according to the present invention, there is provided an imaging system, which includes a moving device, an irradiation device, an imaging device, and a synthesis device. The moving device moves a subject. The irradiation device intermittently and sequentially irradiates the subject with irradiation light having different wavelengths according to the position of the moving subject. The imaging device generates a frame of image by sequentially receiving each reflected light obtained by the irradiation being reflected by the subject, temporarily and sequentially retaining each signal information based on the reflected light of each wavelength, and reading out each retained signal information together. The synthesis device generates a synthesized image by cutting out a subject image corresponding to the reflected light of each wavelength from the one frame of image, and superimposing the cut-out multiple subject images.

[0014] In addition, according to the present invention, there is provided a camera method, which includes the steps of generating a frame of image and generating a composite image. The frame of image is generated by sequentially receiving each reflected light reflected by the subject when the subject is intermittently and sequentially irradiated with each irradiation light having different wavelengths according to the position of the moving subject, temporarily and sequentially retaining each signal information based on the reflected light of each wavelength, and reading out each retained signal information together. Moreover, the composite image is generated by cutting out the subject image corresponding to the reflected light of each wavelength from the frame of image, and superimposing the cut-out multiple subject images. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 1 is an explanatory diagram for explaining one example of the configuration of the imaging system 10 according to the first embodiment of the present invention.

[0016] Figure 2 is a block diagram showing one example of a functional configuration of the camera module 100 according to the first embodiment.

[0017] Figure 3 : is an explanatory diagram showing a planar configuration example of the image pickup element 134 according to the first embodiment.

[0018] Figure 4 is a flowchart for explaining one example of the imaging method according to the first embodiment.

[0019] Figure 5 is an explanatory diagram (one of) for explaining one example of the image capturing method according to the first embodiment.

[0020] Figure 6 This is an explanatory diagram (part 2) for explaining an example of the imaging method according to the first embodiment.

[0021] Figure 7 This is an explanatory diagram (part 3) for explaining an example of the imaging method according to the first embodiment.

[0022] Figure 8 This is an explanatory diagram (fourth) for explaining an example of the imaging method according to the first embodiment.

[0023] Fig. 9 is a flowchart for explaining an example of an image capturing method according to the second embodiment of the present invention.

[0024] Fig.10 is an explanatory diagram for explaining one example of an image capturing method according to the second embodiment.

[0025] Fig.11is a flowchart for explaining an example of an image capturing method according to the third embodiment of the present invention.

[0026] Fig.12 is an explanatory diagram (one of) for explaining one example of the image capturing method according to the third embodiment.

[0027] Fig.13 This is an explanatory diagram (part 2) for explaining an example of the imaging method according to the third embodiment.

[0028] Fig.14 FIG. 3 is an explanatory diagram for explaining an example of the imaging method according to the third embodiment.

[0029] Fig.15 is a flowchart for explaining an example of an image capturing method according to the fourth embodiment of the present invention.

[0030] Fig.16 is an explanatory diagram for explaining one example of an image capturing method according to the fourth embodiment.

[0031] Fig.17 is a flowchart for explaining an example of an image capturing method according to the fifth embodiment of the present invention.

[0032] Fig.18 is an explanatory diagram for explaining one example of an image capturing method according to the fifth embodiment.

[0033] Fig.19 : is an explanatory diagram showing one example of an electronic device 900 according to the sixth embodiment of the present invention.

[0034] Fig. 20 is a block diagram showing one example of a schematic configuration of a vehicle control system.

[0035] Fig.21 1 is an explanatory diagram showing an example of the installation positions of the vehicle exterior information detection unit and the imaging unit. DETAILED DESCRIPTION

[0036] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. Please note that in this specification and the accompanying drawings, components having substantially the same functional configuration are represented by the same reference numerals, and repeated descriptions of these components will be omitted. In addition, in this specification and the accompanying drawings, similar components in different embodiments can be distinguished by adding different letters after the same reference numerals. However, if there is no need to particularly distinguish each similar component, it is represented by only the same reference numeral.

[0037] Please note that the instructions will be given in the following order.

[0038] 1. Background when the inventor created the embodiments of the present invention 2. First embodiment

[0039] 2.1 Overview of the Camera System

[0040] 2.2 Detailed structure of the camera module

[0041] 2.3 Video Recording Method

[0042] 2.4 Modification 1

[0043] 2.5 Modification 2

[0044] 3. Second embodiment

[0045] 3.1 Detailed structure of the camera module

[0046] 3.2 Video Recording Method

[0047] 4. Third embodiment

[0048] 4.1 Detailed structure of the camera module

[0049] 4.2 Video Recording Method

[0050] 5. Fourth embodiment

[0051] 5.1 Detailed Structure of the Camera Module

[0052] 5.2 Video Recording Method

[0053] 6. Fifth embodiment

[0054] 6.1 Detailed Structure of the Camera Module

[0055] 6.2 Video Recording Method

[0056] 7. Sixth embodiment

[0057] 8. Application examples for mobile objects

[0058] 9. Summary

[0059] 10. Supplement

[0060] The embodiments described later will be described in the case of being applied to an inspection device that inspects the presence or absence of scratches, the presence or absence of foreign matter mixed in, and whether the appearance of the product is a qualified product suitable for shipment based on an image of the appearance of the manufactured product in a production line installed at a production site, etc. However, the embodiments of the present invention are not limited to being applied to the inspection device, and may also be applied to other devices or for other purposes.

[0061] In addition, the embodiments described later will be described in the case of being applied to a camera module that operates in a global shutter (GS) manner. Please note that in the later description, the global shutter method means a method of reading out the camera signals (signal information) obtained by each camera element of the camera module together and generating a frame of image based on the read camera signals. However, the embodiments of the present invention are not limited to being applied to camera modules of the global shutter type, and can also be applied to camera modules of other methods.

[0062] In the following description, one frame means one readout, and therefore one frame image is an image generated by performing one collective readout of the imaging signal.

[0063] [1. Background when the inventors created the embodiments of the present invention]

[0064] As described above, in order to more accurately capture the appearance of a product, for example, a technology has been proposed for synthesizing a plurality of spectral images into one image and using the synthesized image, wherein the plurality of spectral images are obtained by finely dividing the wavelength of light related to the subject image into a plurality of wavelengths and thereby detecting the wavelengths.

[0065] Specifically, as an example of a method for obtaining a spectroscopic image, a camera module uses optical components (such as a diffraction grating and a reflector) to split light in a vertical direction relative to a horizontal line and detect the light. In addition, the subject or the camera module is moved (scanned) at a constant speed in the horizontal direction to perform the above-mentioned spectroscopic and detection, thereby acquiring a two-dimensional image of each wavelength of the light.

[0066] Furthermore, in the above-mentioned Patent Document 1, strobe lights having different wavelengths are continuously irradiated to the subject, and reflected light from the spatially separated subjects is incident on different positions on a light receiving surface (on which a plurality of imaging elements of the imaging module are arranged), thereby detecting the light. However, in Patent Document 1, as described above, in order to spatially separate the reflected light, many optical components such as a diffraction grating and a reflector are required, and therefore, it is difficult to avoid the complication of the structure and the increase in the manufacturing cost of the imaging module.

[0067] In addition, in the above-mentioned patent document 2, for each frame, the wavelength of the light emitted from the light source is switched to detect images of each wavelength. Specifically, in the above-mentioned patent document 2, when it is desired to obtain images of three different wavelengths, it is necessary to spend three frames of shooting time. Therefore, in the above-mentioned patent document 2, it is difficult to suppress the increase in processing time used in obtaining images, and the real-time performance is poor.

[0068] Therefore, in view of this situation, we have created an embodiment of the present invention that can obtain a composite image of spectral images with a simple structure and high speed. Specifically, according to the embodiment of the present invention, many optical elements such as diffraction gratings and reflectors are not required, which can avoid the complication of the structure and the increase in the manufacturing cost of the camera module. Moreover, within the shooting time for one frame, an image obtained by synthesizing multiple spectral images into one image can be generated. Below, the embodiments of the present invention created by the inventors are described in detail one by one.

[0069] [2. First embodiment]

[0070] <2.1 Overview of Camera System>

[0071] First, refer to Figure 1 The configuration of the imaging system 10 according to the embodiment of the present invention will be described. Figure 1 1 is an explanatory diagram for explaining an example of the configuration of the imaging system 10 according to the present embodiment. Figure 1 As shown, the camera system 10 according to the present embodiment may mainly include, for example, a camera module 100, a control server 200, and a belt conveyor (mobile device) 300. That is, the belt conveyor 300 is provided on a production line and is used to convey the manufactured product (referred to as the subject 800 in the following description). In addition, the camera module 100 may be applied to an inspection device for inspecting the presence or absence of scratches, the presence or absence of foreign matter mixed in, and whether the appearance of the product is a qualified product suitable for shipment based on an image of the appearance of the manufactured product. Below, an overview of each device included in the camera system 10 will be described in sequence.

[0072] (Camera module 100)

[0073] The camera module 100 irradiates the subject 800 with light, receives reflected light from the subject 800, generates one frame of image, and generates a composite image based on the one frame of image. The detailed configuration of the camera module 100 will be described later.

[0074] (Control Server 200)

[0075] The control server 200 can control the camera module 100, and further, can monitor or control the travel speed (described later) of the belt conveyor 300, the position of the object 800 on the belt conveyor 300, etc. The control server 200 is implemented by hardware such as a CPU (central processing unit), a ROM (read only memory), and a RAM (random access memory).

[0076] (Belt conveyor 300)

[0077] The belt conveyor 300 is a moving device capable of moving the subject 800 according to the control of the control server 200. Alternatively, the belt conveyor 300 is a moving device whose travel speed is monitored by the control server 200 and which can move the subject 800. In the present embodiment, the moving device is not limited to the belt conveyor 300, and it is not particularly limited as long as it is a moving device capable of moving the subject 800.

[0078] Note that the various devices in the camera system 10 according to the present embodiment are connected to each other via a network (not shown) so as to be communicable. Specifically, for example, the camera module 100, the control server 200, and the belt conveyor 300 can be connected to the above network via a base station (not shown) or the like (for example, a base station of a mobile phone, an access point of a wireless LAN (wireless local area network), and the like). Note that as a communication method used in the above network, any method is applicable, whether it is wired or wireless (for example, WiFi (registered trademark), Bluetooth (registered trademark), etc.), but it is preferable to use a communication method that can maintain stable operation.

[0079] <2.2 Detailed Structure of Camera Module>

[0080] Next, we will refer to Figure 2 The configuration of the camera module 100 according to the present embodiment will be described. Figure 2 1 is a block diagram showing an example of the functional configuration of the camera module 100 according to the present embodiment. Figure 2 As shown, the camera module 100 according to the present embodiment may mainly include, for example, an irradiation unit 110 and an image pickup device 120. The details of each block included in the camera module 100 will be described below. In the following description, the camera module 100 in which the irradiation unit 110 and the image pickup device 120 are constructed as an integrated structure will be described as an example, but the present embodiment is not limited to the structure in which they are integrated as described here. That is, in the present embodiment, the irradiation unit 110 and the image pickup device 120 may also be constructed in the form of an individual.

[0081] (Irradiation part 110)

[0082] The irradiation unit 110 may intermittently and sequentially irradiate the subject 800 with irradiation lights having different wavelengths (for example, wavelengths λ1 to λ7) according to the position of the moving subject 800 (pulse irradiation). Figure 2As shown, the irradiation unit 110 has a plurality of light emitting elements (light emitting diodes; LEDs) 112, which are arranged at different positions (specifically, at different positions along the traveling direction of the belt conveyor 300) and can emit light with different wavelengths. In addition, in the present embodiment, the plurality of light emitting elements 112 sequentially irradiate light with corresponding wavelengths using the corresponding light emitting elements 112 according to the position of the object 800 (in other words, in synchronization with the object 800 reaching the position where each light emitting element 112 can irradiate). For example, the plurality of light emitting elements 112 may include a plurality of LED light emitting diodes capable of emitting near-infrared light (having a wavelength of approximately 800nm ​​to 1700nm). More specifically, in Figure 2 In the example, the light emitting element 112a can emit near-infrared light with a wavelength of 900 nm, the light emitting element 112b can emit near-infrared light with a wavelength of 1200 nm, and the light emitting element 112c can emit near-infrared light with a wavelength of 1500 nm.

[0083] (Image pickup device 120)

[0084] like Figure 2 As shown, the imaging device 120 includes a single imaging apparatus and may mainly include an imaging section 130, a synthesis section 140, and a control section 150. Figure 2 In the example shown, the imaging unit 130, the synthesis unit 140 and the control unit 150 are configured as an integrated device, but the present embodiment is not limited thereto, and the above components may also be configured as individual components. The details of each functional unit included in the imaging device 120 will be described in turn.

[0085] (Camera unit 130)

[0086] The imaging unit 130 can sequentially receive reflected light having various wavelengths (for example, wavelengths λ1 to λ7) reflected by the moving subject 800. In addition, the imaging unit 130 can generate a frame of image by temporarily and sequentially retaining various imaging signals (signal information) based on the received reflected light of various wavelengths, and then reading out the retained imaging signals together. Specifically, the imaging unit 130 has an optical system mechanism (not shown) including a lens unit 132, an aperture mechanism (not shown), a zoom lens (not shown), and a focusing lens (not shown). In addition, the imaging unit 130 also includes: a plurality of imaging elements 134, which perform photoelectric conversion on the light obtained using the above-mentioned optical system mechanism to generate imaging signals; a plurality of storage units 136, which temporarily retain the generated imaging signals; and a readout unit 138, which reads out the imaging signals from the plurality of storage units 136 together. Note that although in Figure 2Although one imaging element 134 and one storage unit 136 are shown in FIG. 1 , a plurality of imaging elements 134 and a plurality of storage units 136 may be provided in the imaging unit 130 according to the present embodiment.

[0087] More specifically, the optical system mechanism uses the lens unit 132 and the like to focus the reflected light from the subject 800 as an optical image onto a plurality of imaging elements 134. For example, the imaging element 134 may be a compound sensor such as an InGaAs photodiode (InGaAs imaging element) capable of detecting near-infrared light, or may be a silicon photodiode capable of detecting visible light. In addition, the plurality of imaging elements 134 are arranged in a matrix on the light receiving surface (imaging surface), and each imaging element 134 performs photoelectric conversion on a pixel basis (unit of imaging element) for the optical image formed, thereby generating a signal of each pixel as an imaging signal. In addition, the plurality of imaging elements 134 outputs the generated imaging signal to, for example, a storage unit 136 provided in pixel basis. The output imaging signal may be temporarily retained by the storage unit 136. In addition, the readout unit 138 may output a frame of image to the synthesis unit 140 by reading out the imaging signals from the plurality of storage units 136 together. That is, in the present embodiment, the imaging section 130 can operate in a global shutter method in which imaging signals held in the respective storage sections 136 are read out collectively.

[0088] In this embodiment, for example, the irradiation by the irradiation unit 110 and the global shutter imaging (multiple exposure) by the imaging unit 130 are performed in synchronization with each other, triggered by the object 800 arriving at the imaging position by the belt conveyor 300 .

[0089] In addition, refer to Figure 3 , an example of a planar structure of the above-mentioned plurality of imaging elements 134 will be described below. Figure 3 1 is an explanatory diagram showing an example of a planar configuration of the image pickup element 134 according to the present embodiment. Figure 3 As shown, a plurality of imaging elements 134 according to the present embodiment are arranged in a matrix on a light receiving surface of a semiconductor substrate 500 made of, for example, silicon. Specifically, the imaging module 100 according to the present embodiment has: a pixel array section 410 in which a plurality of imaging elements 134 are arranged; and a peripheral circuit section 480 provided in a manner surrounding the pixel array section 410. The peripheral circuit section 480 includes a vertical drive circuit section 432, a column signal processing circuit section 434, a horizontal drive circuit section 436, an output circuit section 438, and a control circuit section 440, etc. The details of the pixel array section 410 and the peripheral circuit section 480 will be described below.

[0090] The pixel array unit 410 has a plurality of imaging elements (pixels) 134 arranged two-dimensionally in a matrix on a semiconductor substrate 500. In addition, the plurality of pixels 134 may include normal pixels and a pair of phase difference detection pixels, wherein the normal pixels are used to generate pixel signals for image generation, and the pair of phase difference detection pixels are used to generate pixel signals for focus detection. Each pixel 134 may include a plurality of InGaAs imaging elements (photoelectric conversion elements) and a plurality of pixel transistors (e.g., MOS (metal oxide semiconductor) transistors) (not shown). More specifically, the pixel transistors may include, for example, transfer transistors, selection transistors, reset transistors, and amplifier transistors.

[0091] The vertical drive circuit section 432 includes, for example, a shift register. The vertical drive circuit section 432 selects the pixel drive wiring 442, supplies a pulse for driving the pixel 134 to the selected pixel drive wiring 442, and drives the pixel 134 in units of rows. That is, the vertical drive circuit section 432 sequentially shifts each pixel 134 of the pixel array section 410 in the vertical direction ( Figure 3 The pixel signal is selectively scanned in the up and down directions in the vertical direction, and a pixel signal based on the charge generated corresponding to the amount of light received by the photoelectric conversion element of each pixel 134 is provided to the column signal processing circuit unit 434 described later via the vertical signal line 444.

[0092] The column signal processing circuit section 434 is arranged for each column of the pixels 134, and performs signal processing such as noise removal for each pixel column for pixel signals output from a row of pixels 134. For example, the column signal processing circuit section 434 can perform signal processing such as CDS (correlated double sampling) for removing fixed pattern noise inherent to pixels and AD (analog-digital) conversion.

[0093] The horizontal driving circuit section 436 includes, for example, a shift register and sequentially outputs horizontal scanning pulses to sequentially select the column signal processing circuit sections 434 , and can output pixel signals from the column signal processing circuit sections 434 to the horizontal signal lines 446 .

[0094] The output circuit section 438 can perform signal processing on the pixel signals sequentially provided from each of the above-mentioned column signal processing circuit sections 434 via the horizontal signal line 446, and then can output these signals. The output circuit section 438 can play the role of a functional section for performing buffering, for example, or can perform processing such as black level adjustment, column difference correction, and various digital signal processing. Please note that buffering refers to temporarily storing pixel signals in order to compensate for the difference in processing speed and transmission speed when pixel signals are exchanged. In addition, the input / output terminal 448 is a terminal for signal exchange with an external device, and this terminal does not necessarily have to be set in this embodiment.

[0095] The control circuit section 440 can receive an input clock and data for indicating an operation mode and the like, and can output data such as internal information of the pixel 134. That is, the control circuit section 440 generates a clock signal or a control signal to be a reference for the operation of the vertical drive circuit section 432, the column signal processing circuit section 434, the horizontal drive circuit section 436, and the like based on the vertical synchronization signal, the horizontal synchronization signal, and the main clock. Then, the control circuit section 440 outputs the generated clock signal or the control signal to the vertical drive circuit section 432, the column signal processing circuit section 434, the horizontal drive circuit section 436, and the like.

[0096] Note that the planar configuration example of the image pickup element 134 according to the present embodiment is not limited to Figure 3 The examples shown are not particularly limited and may include, for example, other circuit parts, etc.

[0097] (Synthesis Department 140)

[0098] The synthesis unit 140 cuts out the subject images corresponding to the reflected light of each wavelength (for example, wavelengths λ1 to λ7) from the one frame image output from the imaging unit 130, and superimposes the cut-out plurality of subject images to generate a synthesized image. The synthesis unit 140 is implemented by hardware such as a CPU, ROM, and RAM, for example. Specifically, Figure 2 As shown, the synthesis section 140 mainly includes a binarization processing section 142, an imaging region specifying unit 144, and a synthesis processing section 146. The details of each functional section included in the synthesis section 140 will be described in turn.

[0099] The binarization processing unit 142 can generate a two-step color tone image (for example, a black and white image) by performing a binarization process of converting a frame image output from the camera unit 130 into a two-step color tone. For example, the binarization processing unit 142 generates a black and white image by comparing the camera signal of each pixel unit (specifically, each pixel) in a frame image with shading with a predetermined threshold value, and based on the threshold value, converting the pixel unit holding the camera signal within one range into white and converting the pixel unit holding the camera signal within another range into black. In the present embodiment, as described above, a binarization process is performed on a frame image with shading to convert the image into a black and white image. As a result, the contour of the imaging of the subject 800 in the frame image is clarified. Thereby, the subject image described later can be easily and accurately determined.

[0100] The imaging region determination unit 144 determines each subject image (e.g., ROI (Region of Interest)) corresponding to the reflected light of each wavelength (e.g., wavelengths λ1 to λ7) within the above-mentioned one frame of image. Specifically, the imaging region determination unit 144 can determine the center coordinates (e.g., X and Y coordinates) of the image of each subject 800 within the above-mentioned one frame of image, for example, by detecting the contours of the image of each subject 800 contained in the two-level tone image generated by the binarization processing unit 142. In addition, the imaging region determination unit 144 can determine the ROI, which is the area of ​​the image of the subject 800 corresponding to the reflected light of each wavelength, based on the determined center coordinates. For example, the imaging region determination unit 144 superimposes the center of a rectangular extraction frame having a predetermined size capable of containing the image of the subject 800 on each determined center coordinate, thereby determining each ROI within the one frame of image. Note that in this embodiment, the extraction frame is not limited to a rectangular shape, and may also have a polygonal shape, a circular shape, or a shape that is the same as or similar to the shape of the subject 800, as long as the extraction frame has a size that can contain the image of the subject 800. Further, in this embodiment, the determination of the ROI is not limited to being performed based on the above-mentioned center coordinates, and may be performed based on, for example, the contours of the detected images of each subject 800, and there is no particular limitation.

[0101] In addition, in this embodiment, the imaging region determination unit 144 may determine each ROI by determining the position of the image of the identification mark (not shown) provided on the surface of the object 800 in the above-mentioned one frame image instead of using the two-level tone image. In addition, in this embodiment, the imaging region determination unit 144 may determine each ROI based on a plurality of predetermined regions pre-specified by the user in the above-mentioned one frame image (for example, the coordinates of each vertex of such region are pre-set) instead of using the two-level tone image.

[0102] Then, the synthesis processing unit 146 cuts out each ROI from one frame image based on each ROI determined by the imaging region determination unit 144, and superimposes the cut-out multiple ROIs to generate a synthetic image. Specifically, the synthesis processing unit 146 performs position adjustment in a manner such that the center and the outline of the image of the object 800 contained in each ROI are consistent with each other, and superimposes the multiple ROIs, thereby generating a synthetic image. Please note that in the present embodiment, the synthesis processing unit 146 can superimpose the multiple ROIs in a state such that the images of the identification marks (not shown) provided on the surface of the object 800 contained in each ROI are consistent with each other, thereby generating a synthetic image, and the synthesis processing unit 146 is not particularly limited. In addition, when generating a synthetic image, the synthesis processing unit 146 can refer to the color information (for example, red, green, and blue in the visible light band are assigned) pre-assigned to each wavelength (for example, wavelengths λ1 to λ7) to generate a pseudo color image. In the present embodiment, by generating a synthetic image of the pseudo color image in the above manner, the visibility of the details in the image can be improved. The details of the generation of the pseudo-color image will be described later.

[0103] (Control Unit 150)

[0104] The control unit 150 can control the imaging unit 130 to receive reflected light in synchronization with the irradiation of the irradiation unit 110. For example, the control unit 150 is implemented by hardware such as a CPU, a ROM, and a RAM.

[0105] As described above, in the camera module 100 according to the present embodiment, many optical components such as diffraction gratings and reflection mirrors are not required, thereby avoiding complication of the structure and increase in the manufacturing cost of the camera module 100. That is, according to the present embodiment, the camera module 100 having a simple structure can be provided.

[0106] <2.3 Video Recording Method>

[0107] The configurations of the imaging system 10 and the respective devices included in the imaging system 10 according to the present embodiment have been described in detail above. Figures 4 to 8 The imaging method according to this embodiment will be described. Figure 4 is a flowchart for explaining an example of the imaging method according to the present embodiment. Figures 5 to 8 FIG. 1 is an explanatory diagram for explaining an example of the imaging method according to the present embodiment. Figure 4 As shown, the imaging method according to the present embodiment includes a plurality of steps from step S101 to step S121. The details of each step included in the imaging method according to the present embodiment will be described below.

[0108] (Step S101)

[0109] The control unit 150 monitors the travel speed of the belt conveyor 300 (for example, controls the travel speed to be constant) or the position of the subject 800 on the belt conveyor 300 , for example, by cooperating with the control server 200 .

[0110] (Step S103)

[0111] The control unit 150 determines whether the subject 800 has reached the photography start position. In the present embodiment, when the subject 800 has reached the photography start position, the processing proceeds to the next step S105, and when the subject 800 has not reached the photography start position, the processing returns to the previous step S101. That is, in the present embodiment, as described above, the irradiation / light receiving operation is performed synchronously with the travel of the belt conveyor 300. Note that in the present embodiment, the progress of the processing is not limited to the arrival of the subject 800 at the photography start position as a trigger, and other events may be used as triggers. In addition, in the present embodiment, the trigger event may be obtained from each device in the camera system 10, or may be obtained from a device outside the camera system 10, and this is not particularly limited.

[0112] (Step S105)

[0113] The control section 150 controls the light emitting element 112 corresponding to the position of the subject 800, and causes the light emitting element 112 to irradiate the subject 800 with light having a predetermined wavelength (e.g., wavelengths λ1 to λ7) (e.g., near-infrared light having a predetermined wavelength). Specifically, Figure 5 As shown, when the subject 800 arrives under each of the light emitting elements 112 a to 112 c , the light emitting element irradiates the subject 800 with light having a predetermined wavelength.

[0114] (Step S107)

[0115] The control unit 150 controls the plurality of imaging elements 134 in synchronization with the illumination of the light emitting element 112 in step S105 so that the plurality of imaging elements 134 receive reflected light from the subject 800. For example, Figure 5As shown, a plurality of imaging elements 134 (photodiodes; PD) receive light in synchronization with the irradiation of each light emitting element 112a to 112c, and generate an image 802 of the subject obtained by the light reception as an imaging signal, and output the generated imaging signal to each storage unit 136 (Memory: MEM). In addition, each storage unit 136 temporarily holds the imaging signal. Note that the imaging signal corresponds to an image 802 of the subject 800 corresponding to each wavelength (e.g., wavelengths λ1 to λ7) of the light irradiated by the light emitting element 112 in step S105, and each image 802 is included in one frame of image (i.e., multiple exposure) described later.

[0116] (Step S109)

[0117] The control unit 150 controls the corresponding light emitting element 112 to terminate the irradiation.

[0118] (Step S111)

[0119] The control unit 150 determines whether all the light emitting elements 112 have been irradiated. In this embodiment, if all the light emitting elements 112 have been irradiated, the process proceeds to the next step S113, and if not all the light emitting elements 112 have been irradiated, the process returns to the previous step S105.

[0120] That is, in this embodiment, if Figure 6 As shown in the upper part of FIG. 1 , the irradiation unit 110 sequentially performs pulse irradiation on the moving object 800 with light having different wavelengths (for example, λ1 to λ7). Then, in the present embodiment, as shown in FIG. Figure 6 As shown in the middle of FIG. 1 , the following operations are performed in sequence in synchronization with the illumination timing: the image pickup element 134 receives the reflected light from the object 800, the image pickup signal is transmitted to the storage unit 136, and the storage unit 136 temporarily stores the image pickup signal. Next, as shown in FIG. Figure 6 As shown in the lower part of , in the subsequent steps, by reading out the camera signals from each storage unit 136 together (in other words, by performing multiple exposures to capture the trajectory of the subject 800), a frame of image 802 (spectral image) of the subject 800 corresponding to each wavelength can be obtained.

[0121] (Step S113)

[0122] The control unit 150 controls the readout unit 138 so as to read out the video signals stored in the respective storage units 136 together, obtain a frame of image 802 (spectral image) of the subject 800 corresponding to each wavelength (for example, wavelengths λ1 to λ7), and output the obtained frame of image to the synthesis unit 140 (global shutter method).

[0123] (Step S115)

[0124] The control unit 150 controls the binarization processing unit 142 of the synthesis unit 140 to perform binarization processing in which the one-frame image acquired in step S113 is converted into a second-order tone to generate a second-order tone image. For example, in step S115, the following can be obtained: Figure 7 In addition, the control unit 150 controls the imaging area determination unit 144 of the synthesis unit 140 to detect the contours and the positions of the images 802 of the respective objects 800 contained in the image after the binarization process. For example, in step S115, Figure 7 As shown in the middle of FIG. 8 , the center coordinates (X, Y) of the image 802 of each subject 800 are detected from the black and white image.

[0125] (Step S117)

[0126] like Figure 7 As shown in the lower part of , the control unit 150 controls the synthesis processing unit 146 of the synthesis unit 140 to cut out each ROI having a predetermined extraction frame based on the position of each image 802 of the subject 800 detected in the above step S115.

[0127] (Step S119)

[0128] like Figure 7 As shown in the lower part of , the control unit 150 controls the synthesis processing unit 146 to adjust the positions so that the centers and contours of the images of the subject 800 included in the respective ROIs coincide with each other, and to superimpose the cut-out ROIs. In addition, the control unit 150 controls the synthesis processing unit 146 to generate a pseudo-color image as a synthesized image by referring to color information pre-assigned to each wavelength (e.g., wavelengths λ1 to λ7).

[0129] Here, for ease of understanding, when describing the pseudo-color image generation method, it is assumed that a frame of image 802 (spectral image) of the subject 800 corresponding to three wavelengths λ1 to λ3 is obtained.

[0130] In this example, as the above-mentioned color information, it is assumed that red is pre-assigned to wavelength λ1, green is pre-assigned to wavelength λ2, and blue is pre-assigned to wavelength λ3. In addition, in this embodiment, when generating a pseudo-color image, it is assumed that the plurality of imaging elements 134 of the camera module 100 can detect visible light. That is, it is assumed that: on the light receiving surface of the camera module 100 where the plurality of imaging elements 134 are arranged in a matrix, the imaging elements for detecting red, the imaging elements for detecting green, and the imaging elements for detecting blue are arranged in accordance with the Bayer array.

[0131] Therefore, in this embodiment, under the above-mentioned distribution and assumption conditions, the pseudo-color image can be synthesized as follows. Specifically, first, Figure 8 As shown in the upper left part of , the image data of the ROI corresponding to the wavelength λ1 is assigned to the position of red (R) on the Bayer array, thereby generating a pixel data group 804a. Figure 8 As shown in the middle left part of , the image data of the ROI corresponding to the wavelength λ2 is assigned to the position of green (G) on the Bayer array, thereby generating a pixel data group 804b. Figure 8 As shown in the lower left part of FIG. 1 , the image data of the ROI corresponding to the wavelength λ3 is allocated to the position of the blue (B) on the Bayer array, thereby generating a pixel data group 804c. Then, in this embodiment, the pixel data groups 804a, 804b, and 804c to which the image data is allocated to the positions of the respective colors can be synthesized. Figure 8 . In the present embodiment, by generating a composite image of the pseudo-color image 806 in the above-described manner, the visibility of details in the image can be improved.

[0132] Please note that in this embodiment, the synthesis of the pseudo-color image 806 is not limited to the above example, and can also be implemented by other methods. For example, other methods may be to perform additive averaging of each pixel such as color parameters.

[0133] (Step S121)

[0134] The control section 150 controls the synthesis processing section 146 to output the generated synthesized image to the control section 150. In addition, the output synthesized image is converted into an appropriate format by the control section 150 and output to, for example, the control server 200 or the like.

[0135] As described above, according to this embodiment, since a composite image is generated using one frame image, an image obtained by combining multiple spectral images into one image can be generated within the imaging time of one frame. That is, according to this embodiment, a composite image of spectral images can be obtained at a high speed.

[0136] <2.4 Modification 1>

[0137] In addition, in this embodiment, as a variation, instead of using a binary tone image as described above, each ROI may be determined based on a plurality of predetermined areas in the one frame image that are previously specified by the user.

[0138] <2.5 Modification 2>

[0139] Note that in the foregoing description, it has been described that a plurality of cut-out ROIs are superimposed to generate a composite image, but the present embodiment and modification 1 are not limited thereto. For example, in the present embodiment and modification 1, the above-mentioned one-frame image itself may be output, or the ROI cut out from the above-mentioned one-frame image may be output. For example, in this case, since images corresponding to the respective wavelengths of light can be acquired and analyzed respectively, the presence or absence and distribution of components for the respective corresponding wavelengths can be easily identified.

[0140] [3. Second embodiment]

[0141] In the first embodiment described above, a plurality of light emitting elements 112 are provided to emit light having different wavelengths. However, by using a plurality of optical filters 162 (see Fig.10 ), even if a light emitting element 112d emitting white light is used (see Fig.10 ) can also obtain a composite image in the same manner as in the above-mentioned embodiment. Thus, in this embodiment, since a plurality of filters 162 are used, it is possible to avoid complication of the structure of the irradiation unit 110 and increase in the manufacturing cost of the irradiation unit 110. Fig. 9 and Fig.10 , a filter unit 160 (see Fig.10 ) is described as the second embodiment of the present invention. Fig. 9 is a flowchart for explaining an example of the imaging method according to the present embodiment, Fig.10 is an explanatory diagram for explaining an example of the image capturing method according to the present embodiment.

[0142] <3.1 Detailed Structure of Camera Module>

[0143] First, the detailed configuration of the camera module 100 according to the second embodiment of the present invention will be described. In the following description, the description of the common points with the above-mentioned first embodiment will be omitted, and only the differences will be described. In this embodiment, the illumination unit 110 of the camera module 100 has a light emitting element 112d (see FIG. 1 ) that can illuminate the subject 800 with white light. Fig.10). It should be noted that the present embodiment is not limited to the use of the light emitting element 112d, and for example, an indoor light or the like may be used instead of the irradiation unit 110, or natural light may be used (in this case, the irradiation unit 110 itself becomes unnecessary). That is, according to the present embodiment, it is not necessary to have an irradiation unit 110 with a special structure, and a general lighting device or the like may be used.

[0144] Furthermore, the imaging section 130 of the imaging device 120 according to the present embodiment further includes a filter section 160 (see FIG. 1 ) on the object 800 side of the lens section 132. Fig.10 ). Specifically, Fig.10 As shown, the filter unit 160 has a plurality of filters 162a, 162b, and 162c arranged in sequence along the traveling direction (moving direction) of the belt conveyor 300 (note this figure). Fig.10 , three filters 162a to 162c are shown, but the present embodiment is not limited to three filters, but a plurality of filters may be provided. Each filter 162a, 162b, 162c includes a narrow-band OCCF (on chip color filter) or a plasma filter (a filter that uses a surface plasmon and transmits only a specific wavelength), and can transmit light having different wavelengths. For example, the filters 162a, 162b, 162c can transmit light having wavelengths λ1 to λ7 in the first embodiment.

[0145] In the present embodiment, for example, global shutter imaging (multiple exposure) is performed by the imaging unit 130 using the arrival of the subject 800 at the imaging position as a trigger.

[0146] As described above, in the present embodiment, since only one type of light emitting element 112d is required by using a plurality of filters 162, it is possible to avoid complication of the structure of the irradiation unit 110 and increase in the manufacturing cost of the irradiation unit 110. In addition, in the present embodiment, the irradiation unit 110 can be omitted by using a general indoor lamp, natural light, etc. That is, according to the present embodiment, it is not necessary to have the irradiation unit 110 of a special structure, but a general lighting device, etc. can be used.

[0147] <3.2 Video Recording Method>

[0148] The detailed configuration of the camera module 100 according to the present embodiment has been described above. Fig. 9 and Fig.10 The imaging method according to this embodiment is described below. Fig. 9As shown, the imaging method according to the present embodiment includes a plurality of steps from step S201 to step S217. The details of each step included in the imaging method according to the present embodiment will be described below.

[0149] First, in this embodiment, the light emitting element 112d starts irradiating light.

[0150] (Step S201 to Step S205)

[0151] Since steps S201 to S205 according to this embodiment are different from those according to Figure 4 Steps S101, S103 and S107 of the first embodiment shown are the same, and their description will be omitted here.

[0152] (Step S207)

[0153] The control unit 150 determines whether the imaging unit 130 has received reflected light of all wavelengths. In this embodiment, if reflected light of all wavelengths has been received, the process proceeds to the next step S209, and if reflected light of all wavelengths has not been received, the process returns to the previous step S205.

[0154] In this embodiment, if Fig.10 As shown in the upper part of , for example, white light is irradiated toward the moving subject 800. Then, as Fig.10 As shown in the middle of FIG. 1 , the following operations are sequentially performed in synchronization with the timing of the object 800 reaching above each filter 162a to 162c: the image pickup element 134 receives the reflected light from the object 800, the image pickup signal is transmitted to the storage unit 136, and the storage unit 136 temporarily stores the image pickup signal. Next, as in the first embodiment, Fig.10 As shown in the lower part of , in the subsequent steps, by reading the imaging signals from each storage unit 136 together, a frame of image including the image 802 of the object 800 corresponding to each wavelength can be obtained.

[0155] (Step S209 to Step S217)

[0156] Since steps S209 to S217 according to this embodiment are different from those according to Figure 4 Steps S113 to S121 of the first embodiment shown are the same, and their description is omitted here.

[0157] [4. Third embodiment]

[0158] In the above-mentioned first embodiment, a frame image containing an image 802 of the subject 800 corresponding to reflected light of each wavelength (for example, wavelengths λ1 to λ7) is converted into a two-order tone image, and the positions of the images 802 of the subject 800 within the one frame image are determined. On the other hand, in a third embodiment of the present invention described later, in order to further improve the accuracy of position detection, the position of the image 802 of the subject 800 corresponding to reflected light of each wavelength other than visible light can be determined by using a frame image containing an image 802 of the subject 800 corresponding to reflected light of visible light (reference light). Thus, in this embodiment, because a visible light image (visible light image) or the like in which the outline of the image 802 can be easily detected in the two-order tone image is used, the position of the image 802 of the subject 800 corresponding to reflected light such as near-infrared light other than visible light can be accurately detected. Therefore, the reference image 802 will be used as the reference image. Figures 11 to 14 Such a third embodiment will be described. Fig.11 is a flowchart for explaining an example of an imaging method according to the present embodiment, and Figure 12 to Figure 14 is an explanatory diagram for explaining an example of the image capturing method according to the present embodiment.

[0159] <4.1 Detailed Structure of Camera Module>

[0160] First, the detailed configuration of the camera module 100 according to the present embodiment will be described. In the following description, the description of the common points with the above-mentioned first embodiment will be omitted, and only the differences will be described. In the present embodiment, compared with the first embodiment, the irradiation unit 110 further has a light emitting element (reference light emitting element) 112f (see Fig.12 ). Specifically, in the present embodiment, for example, between a plurality of light emitting elements 112a and 112b capable of emitting near-infrared light having different wavelengths (e.g., wavelengths λ1 to λ7), an emitting element 112f capable of emitting visible light is also provided. Note that although Fig.12 , multiple light emitting elements 112f are drawn, but in this embodiment, it is not limited to using multiple light emitting elements 112f, and one light emitting element 112f may be used. In this embodiment, for example, the irradiation of near-infrared light with different wavelengths by the light emitting elements 112a and 112b and the irradiation of visible light by the light emitting element 112f are performed alternately. In addition, in the following description, the light emitting element 112f irradiates visible light (for example, a wavelength of λ) as a reference light. ref ) is used as an example for explanation, but in this embodiment, the light emitting element 112f is not limited to emitting visible light, for example, it can emit light with a predetermined wavelength other than near-infrared light.

[0161] In addition, in this embodiment, similarly, for example, with the subject 800 reaching the imaging position via the belt conveyor 300 as a trigger, the illumination of the illumination unit 110 including the light emitting element 112f and the imaging (multiple exposure) of the imaging unit 130 using the global shutter method are performed synchronously with each other.

[0162] <4.2 Video Recording Method>

[0163] The detailed configuration of the camera module 100 according to the present embodiment has been described above. Figures 11 to 14 The imaging method according to this embodiment is described below. Fig.11 As shown, the imaging method according to this embodiment includes a plurality of steps from step S301 to step S321. The details of each step included in the imaging method according to this embodiment will be described below. In this embodiment, it is assumed that the object 800 is moved by the belt conveyor 300 at a constant speed.

[0164] (Step S301 and Step S303)

[0165] Since steps S301 and S303 according to this embodiment are different from those according to Figure 4 Steps S101 and S103 of the first embodiment shown are the same, and their description will be omitted here.

[0166] (Step S305)

[0167] The control unit 150 controls the light emitting element 112 corresponding to the position of the subject 800 to emit light having a visible light wavelength (eg, wavelength λ ref ) and near-infrared light having a predetermined wavelength (for example, wavelengths λ1 to λ7) are alternately irradiated toward the subject 800. Specifically, as Fig.12 As shown, when the object 800 reaches below each light emitting element 112a, 112b, 112f, each light emitting element 112a, 112b, 112f irradiates the object 800 with visible light or near-infrared light.

[0168] (Step S307)

[0169] The control unit 150 controls the plurality of imaging elements 134 in synchronization with the illumination of the light emitting element 112 in step S305 so that the plurality of imaging elements 134 receive reflected light from the subject 800. For example, Fig.13As shown, in synchronization with the irradiation of the light emitting elements 112a, 112b, and 112f, the plurality of imaging elements 134 perform light reception, generate an image 802 of the subject obtained by the light reception as an imaging signal, and output the generated imaging signal to each storage unit 136. In addition, each storage unit 136 temporarily holds the imaging signal. Note that the imaging signal corresponds to each wavelength (e.g., wavelengths λ1 to λ7, λ8) of the light irradiated by the light emitting elements 112a, 112b, and 112f in step S305. ref ) corresponds to an image 802 of the subject 800, and each image 802 is included in a frame of image to be described later.

[0170] (Step S309 to Step S313)

[0171] Since steps S309 to S313 according to this embodiment are different from those according to Figure 4 Steps S109 to S113 of the first embodiment shown are the same, and their description will be omitted here.

[0172] (Step S315)

[0173] The control unit 150 controls the binarization unit 142 of the synthesis unit 140 to perform binarization processing (for example, black-and-white imaging) in which the one-frame image acquired in step S313 is converted into a binary tone to generate a binary tone image.

[0174] In addition, if Fig.14 As shown in the middle of , the control unit 150 controls the imaging region determination unit 144 of the synthesis unit 140 to detect the contours of each image 802 of the subject 800 corresponding to the visible light contained in the image after the binarization process. Next, the control unit 150 controls the imaging region determination unit 144 of the synthesis unit 140 to detect the center coordinates (X, Y) of the image 802 of the subject 800 corresponding to the near-infrared light sandwiched between the two images 802 from the positions of the images 802 of the subject 800 corresponding to the visible light. In the present embodiment, since the subject 800 is moved at a constant speed by the belt conveyor 300, in one frame of image, the image 802 of the subject 800 corresponding to the near-infrared light is located at the center between the two images 802 of the subject 800 corresponding to the visible light. Therefore, in this embodiment, by using two images 802 corresponding to visible light whose contours are easily detected in the second-order tone image, the centers of the two images 802 are calculated, thereby accurately detecting the center coordinates (X, Y) of the image 802 of the subject 800 corresponding to near-infrared light, etc.

[0175] (Step S317 to Step S321)

[0176] Since steps S317 to S321 according to this embodiment are different from those according to Figure 4 Steps S117 to S121 of the first embodiment shown are the same, and their description will be omitted here.

[0177] As described above, in this embodiment, by using two images 802 corresponding to visible light whose contours are easily detected in the second-order tone image, the centers of the two images 802 are calculated, thereby accurately detecting the position of the image 802 of the subject 800 corresponding to near-infrared light, etc.

[0178] [5. Fourth embodiment]

[0179] In the above-mentioned first embodiment, the position of the image 802 of the object 800 corresponding to near-infrared light or the like is detected by using a second-order tone image. On the other hand, in the following embodiment, when the position of the image 802 of the object 800 within a frame of image is known in advance, it is possible to obtain only the imaging signals from the corresponding pixels (imaging element 134) within each ROI from the beginning based on a plurality of predetermined areas pre-specified by the user within the frame of image. Thus, in this embodiment, the amount of imaging signals to be read out together by the readout unit 138 can be reduced, and the burden of subsequent processing can be reduced. Therefore, reference will be made to Fig.15 and Fig.16 A fourth embodiment of the present invention will be described. Fig.15 is a flowchart for explaining an example of an imaging method according to the present embodiment, and Fig.16 is an explanatory diagram for explaining an example of an image capturing method according to the present embodiment.

[0180] <5.1 Detailed Structure of Camera Module>

[0181] The detailed configuration of the camera module 100 according to the present embodiment is the same as that of the first embodiment described above, except that the synthesis section 140 is not provided with the binarization processing section 142 and the imaging region determination section 144. Therefore, their description is omitted here.

[0182] <5.2 Video Recording Method>

[0183] Next, we will refer to Fig.15 and Fig.16 The imaging method according to this embodiment is described below. Fig.15 As shown, the imaging method according to this embodiment includes multiple steps from step S401 to step S417. The details of each step included in the imaging method according to this embodiment will be described below. In this embodiment, it is assumed that the position of the image 802 of the subject 800 in a frame of image is known in advance.

[0184] (Step S401 to Step S405)

[0185] Since steps S401 to S405 according to this embodiment are different from those according to Figure 4 Steps S101 to S105 of the first embodiment shown are the same, and their description will be omitted here.

[0186] (Step S407)

[0187] Similar to the first embodiment, the control unit 150 controls the plurality of imaging elements 134 in synchronization with the illumination of the light emitting element 112 in step S405 so that the plurality of imaging elements 134 receive reflected light from the subject 800. Fig.16 As shown, the corresponding imaging elements 134 in each ROI 804 corresponding to the plurality of predetermined areas specified in advance by the user receive light in synchronization with the irradiation of each of the light emitting elements 112a to 112c, generate an image 802 of the subject obtained by the light reception as an imaging signal (a part of the signal information), and output the generated imaging signal to each storage unit 136. In addition, each storage unit 136 temporarily holds the imaging signal. Note that the imaging signal corresponds to the ROI 804 of each wavelength (for example, wavelength λ1 to λ7) of the light irradiated by the light emitting element 112 in step S405, and each ROI 804 is included in one frame of image (i.e., ROI exposure) to be described later. That is, in the present embodiment, since only the corresponding imaging signals in each ROI 804 corresponding to the plurality of predetermined areas specified in advance by the user are read out, the amount of imaging signals read out by the readout unit 138 can be reduced, and the burden of subsequent processing can be reduced.

[0188] (Step S409 and Step S411)

[0189] Since steps S409 and S411 according to this embodiment are different from those according to Figure 4 Steps S109 and S111 of the first embodiment shown are the same, and their description will be omitted here.

[0190] (Step S413)

[0191] The control unit 150 controls the synthesis processing unit 146 of the synthesis unit 140 to cut out each ROI included in one frame of image.

[0192] (Step S415 and Step S417)

[0193] Since steps S415 and S417 according to this embodiment are different from those according to Figure 4 Steps S119 and S121 of the first embodiment shown are the same, and their description will be omitted here.

[0194] As described above, in this embodiment, when the position of the image 802 of the subject 800 in the one frame image is known in advance, based on a plurality of predetermined regions in the one frame image that are specified in advance by the user, it is possible to initially acquire only the imaging signals from the corresponding pixels in each ROI 804. In this way, according to this embodiment, the amount of imaging signals read out by the readout unit 138 can be reduced, and the burden of subsequent processing can be reduced.

[0195] [6. Fifth embodiment]

[0196] In addition, in the present invention, the imaging method according to the fourth embodiment can also be applied to the imaging module 100 according to the second embodiment. Thus, according to the embodiment described below, the amount of imaging signals read out by the readout unit 138 can be reduced, and the burden of subsequent processing can be reduced. Moreover, it is not necessary to have an illumination unit 110 with a special structure, but a general lighting device can be used. Therefore, the reference Fig.17 and Fig.18 A fifth embodiment of the present invention will be described. Fig.17 is a flowchart for explaining an example of an imaging method according to the present embodiment, and Fig.18 is an explanatory diagram for explaining an example of an image capturing method according to the present embodiment.

[0197] <6.1 Detailed Structure of Camera Module>

[0198] The detailed configuration of the camera module 100 according to the present embodiment is the same as that of the above-described second embodiment, except that the synthesis section 140 is not provided with the binarization processing section 142 and the imaging region determination section 144. Therefore, their description is omitted here.

[0199] <6.2 Video Recording Method>

[0200] Next, we will refer to Fig.17 and Fig.18 The imaging method according to this embodiment is described below. Fig.17 As shown, the imaging method according to this embodiment includes multiple steps from step S501 to step S513. The details of each step included in the imaging method according to this embodiment will be described below. In this embodiment, similarly, it is assumed that the position of the image 802 of the subject 800 in a frame of image is known in advance.

[0201] (Step S501 and Step S503)

[0202] Since steps S501 and S503 according to this embodiment are different from those according to Figure 4 Steps S101 and S103 of the first embodiment shown are the same, and their description will be omitted here.

[0203] (Step S505)

[0204] The control unit 150 controls the plurality of imaging elements 134 so that the plurality of imaging elements 134 receive reflected light from the subject 800. In this embodiment, as in the fourth embodiment, Fig.18 As shown, the corresponding imaging elements 134 in each ROI 804 corresponding to a plurality of predetermined regions specified in advance by the user perform light reception, generate an image 802 of the subject obtained by light reception as an imaging signal (a part of signal information), and output the generated imaging signal to each storage unit 136. In addition, each storage unit 136 temporarily holds the imaging signal. Note that the imaging signal corresponds to the ROI 804 of each wavelength (for example, wavelength λ1 to λ7) of the light transmitted by the filters 162a to 162c, and each ROI 804 is included in one frame of image (i.e., ROI exposure) described later. That is, in the present embodiment, since only the corresponding imaging signals in each ROI 804 corresponding to a plurality of predetermined regions specified in advance by the user are read out, the amount of imaging signals read out together by the readout unit 138 can be reduced, and the burden of subsequent processing can be reduced.

[0205] (Step S507)

[0206] Since step S507 according to this embodiment is different from step S508 according to Fig. 9 The step S207 of the second embodiment shown is the same, and its description will be omitted here.

[0207] (Step S509)

[0208] Since step S509 according to this embodiment is the same as that in the embodiment except for reading out each ROI, Fig.15 The step S413 of the fourth embodiment shown is the same and its description will be omitted here.

[0209] (Step S511 and Step S513)

[0210] Since steps S511 and S513 according to this embodiment are different from those according to Figure 4 Steps S119 and S121 of the first embodiment shown are the same, and their description will be omitted here.

[0211] As described above, according to this embodiment, the amount of imaging signals read out by the readout section 138 can be reduced, and the burden of subsequent processing can be reduced. In addition, the illumination section 110 having a special structure is not required, and a general lighting device or the like can be used.

[0212] [7. Sixth embodiment]

[0213] The above-described technology according to the embodiment of the present invention is applicable to all electronic devices that use a camera in the image capture section, such as the following: a camera such as a digital camera or a video camera; a portable terminal device with a camera function; and a copy machine that uses a camera in the image reading section. In addition, the technology of the present invention is also applicable to automobiles, robots, airplanes, drones, various inspection instruments (for example, food inspection, etc.), medical equipment (endoscopes), etc. In the following, as the sixth embodiment of the present invention, reference will be made to Fig.19 An example of an electronic device 900 to which the technology according to the embodiment of the present invention is applied is described. Fig.19 is an explanatory diagram showing one example of an electronic device 900 according to the present embodiment.

[0214] like Fig.19 As shown, the electronic device 900 has a camera device 902, an optical lens (corresponding to Figure 2 The lens unit 132 in the embodiment of the present invention) 910, the shutter mechanism 912, the drive circuit unit (corresponding to Figure 2 The control unit 150 in the embodiment of the present invention) 914 and the signal processing circuit unit (corresponding to Figure 2 The optical lens 910 allows the image light (incident light) from the subject to be projected onto the plurality of imaging elements 134 (see FIG. 1 ) on the light receiving surface of the imaging device 902. Figure 2 As a result, the image is formed on the storage unit 136 (see Figure 2 ) accumulates signal charges in a certain period of time. The shutter mechanism 912 performs an on / off operation to control the light irradiation period and the light shielding period for the camera 902. The drive circuit unit 914 provides a drive signal for controlling the signal transmission operation of the camera 902 or the shutter operation of the shutter mechanism 912 to them. That is, the camera 902 performs signal transmission based on the drive signal (timing signal) provided from the drive circuit unit 914. The signal processing circuit unit 916 can perform various signal processing.

[0215] Note that the embodiments of the present invention are not limited to being applied to inspection devices that check for scratches, foreign matter, and whether the appearance of the product is a qualified product suitable for shipment based on an image of the appearance of the manufactured product in a production line installed at a production site or the like. For example, this embodiment can be applied to the appearance inspection of industrial products (the presence or absence of scratches, and the determination of whether the appearance of the manufactured product is suitable for shipment), etc. In addition, since this embodiment can use light of various wavelengths, this embodiment can be used, for example, for foreign matter mixed inspection of medicines and foods based on the light absorption characteristics unique to the material (the light absorption characteristics unique to the foreign matter can be utilized). In addition, in this embodiment, since light of various wavelengths can be used, for example, color recognition that is difficult to identify with visible light and the depth of the location of scratches or foreign matter can be detected.

[0216] [8. Application examples of mobile objects]

[0217] Furthermore, in an embodiment of the present invention, instead of moving the subject 800, the camera module 100 may be mounted on a mobile body so that the camera module 100 side moves. For example, in a case where the camera module 100 is mounted on a mobile body such as a drone, when the subject 800 is located directly below the light emitting element 112 of the camera module 100, light having a predetermined wavelength may be irradiated. In this case, according to the present embodiment, the distribution of a specific component on the surface of the earth or the like can be detected or the state of crops can be detected. That is, the camera module 100 according to the present embodiment may be implemented as a device to be mounted on any type of mobile body such as an automobile, an electric vehicle, a hybrid vehicle, a motorcycle, a bicycle, a personal motor vehicle, an airplane, an unmanned aerial vehicle, a ship, or a robot.

[0218] Fig. 20 : is a block diagram showing an example of a schematic configuration of a vehicle control system which is an example of a moving body control system to which the technology according to the present invention is applicable.

[0219] The vehicle control system 12000 includes a plurality of electronic control units connected to each other via a communication network 12001. Fig. 20 In the example shown, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an external information detection unit 12030, an internal information detection unit 12040, and an integrated control unit 12050. In addition, as a functional configuration of the integrated control unit 12050, a microcomputer 12051, a sound / image output unit 12052, and an in-vehicle network I / F (interface) 12053 are shown.

[0220] The drive system control unit 12010 controls the operation of the devices related to the drive system of the vehicle according to various programs. For example, the drive system control unit 12010 can function as a control device for the following various devices: a drive force generating device such as an internal combustion engine or a drive motor for generating the drive force of the vehicle; a drive force transmission mechanism for transmitting the drive force to the wheels; a steering mechanism for adjusting the steering angle of the vehicle; and a braking device for generating the braking force of the vehicle.

[0221] The body system control unit 12020 controls the operation of various devices installed on the vehicle body according to various programs. For example, the body system control unit 12020 may function as a control device for the following various devices: a keyless entry system; a smart key system; a power window device; or various lights such as a headlight, a reverse light, a brake light, a turn signal, or a fog light. In this case, a radio wave sent from a portable device that replaces the key or a signal from various switches may enter the body system control unit 12020. The body system control unit 12020 receives input of these radio waves or signals, and controls the door lock device, the power window device, and lights of the vehicle.

[0222] The vehicle exterior information detection unit 12030 detects information about the exterior of the vehicle on which the vehicle control system 12000 is installed. For example, the vehicle exterior information detection unit 12030 is connected to the camera unit 12031. The vehicle exterior information detection unit 12030 causes the camera unit 12031 to capture an image of the exterior of the vehicle and receives the captured image. Based on the received image, the vehicle exterior information detection unit 12030 can perform object detection processing or distance detection processing for objects such as people, cars, obstacles, signs, or text on the road.

[0223] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. The imaging unit 12031 can output the electrical signal as an image, or can output the electrical signal as distance measurement information. In addition, the light received by the imaging unit 12031 can be visible light, or can be non-visible light such as infrared rays.

[0224] The in-vehicle information detection unit 12040 detects information about the interior of the vehicle. For example, the in-vehicle information detection unit 12040 is connected to a driver state detection unit 12041 for detecting a driver state. For example, the driver state detection unit 12041 includes a camera for photographing the driver. Based on the detection information input from the driver state detection unit 12041, the in-vehicle information detection unit 12040 can calculate the driver's fatigue level or concentration level, or can determine whether the driver is dozing off.

[0225] Based on the information about the inside or outside of the vehicle obtained by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040, the microcomputer 12051 can calculate the control target value of the driving force generating device, the steering mechanism or the braking device, thereby outputting a control command to the drive system control unit 12010. For example, the microcomputer 12051 can perform coordinated control aimed at realizing the functions of the advanced driver assistance system (ADAS), which includes: collision avoidance or collision mitigation of the vehicle, following driving based on the vehicle-to-vehicle distance, vehicle constant speed driving, vehicle collision warning, vehicle lane departure warning, etc.

[0226] In addition, based on the information about the vehicle's surroundings obtained by the external information detection unit 12030 or the internal information detection unit 12040, the microcomputer 12051 can also perform coordinated control such as automatic driving aimed at enabling the vehicle to drive autonomously without relying on the driver's operation by controlling the driving force generating device, steering mechanism, or braking device, etc.

[0227] In addition, based on the information about the outside of the vehicle obtained by the vehicle exterior information detection unit 12030, the microcomputer 12051 can output a control command to the body system control unit 12020. For example, the microcomputer 12051 can control the headlights according to the position of the preceding vehicle or oncoming vehicle detected by the vehicle exterior information detection unit 12030, thereby performing coordinated control aimed at preventing glare, for example, by switching high beam to low beam.

[0228] The sound / image output unit 12052 transmits an output signal of at least one of sound and image to an output device capable of visually or auditorily notifying the vehicle occupants or the outside of the vehicle of the information. Fig. 20 In the example of FIG. 1 , as output devices, an audio speaker 12061, a display portion 12062, and an instrument panel 12063 are shown. For example, the display portion 12062 may include at least one of an onboard display and a head-up display.

[0229] Fig.21 12031 is a diagram showing an example of the installation position of the camera unit 12031.

[0230] exist Fig.21 In the figure, the vehicle 12100 has the camera units 12101, 12102, 12103, 12104, and 12105 as the camera unit 12031.

[0231] For example, the camera units 12101, 12102, 12103, 12104 and 12105 are arranged at positions of the vehicle 12100, such as the front nose, side mirrors, rear bumper, trunk door, and the upper part of the windshield in the vehicle compartment. The camera unit 12101 arranged at the front nose and the camera unit 12105 arranged at the upper part of the windshield in the vehicle compartment mainly obtain images in front of the vehicle 12100. The camera units 12102 and 12103 arranged at the side mirrors mainly obtain images on the side of the vehicle 12100. The camera unit 12104 arranged at the rear bumper or the trunk door mainly obtains images behind the vehicle 12100. The front images obtained by the camera units 12101 and 12105 are mainly used to detect vehicles, pedestrians, obstacles, traffic lights, traffic signs or lanes in front.

[0232] Notice, Fig.21 Examples of the imaging ranges of the imaging units 12101 to 12104 are shown. The imaging range 12111 indicates the imaging range of the imaging unit 12101 disposed at the front nose. The imaging ranges 12112 and 12113 respectively indicate the imaging ranges of the imaging units 12102 and 12103 disposed at the side mirrors. The imaging range 12114 indicates the imaging range of the imaging unit 12104 disposed at the rear bumper or the trunk door. For example, by superimposing the image data captured by the imaging units 12101 to 12104, a bird's-eye view image of the vehicle 12100 as seen from above can be obtained.

[0233] At least one of the imaging units 12101 to 12104 may have a function of obtaining distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera including a plurality of imaging elements, or may be an imaging element having pixels for phase difference detection.

[0234] For example, based on the distance information obtained by the camera units 12101 to 12104, the microcomputer 12051 can calculate the distance to each solid object within the camera range 12111 to 12114 and the change of the distance over time (relative speed relative to the vehicle 12100), thereby extracting the following solid object as the front vehicle: the solid object is particularly the closest solid object on the driving route of the vehicle 12100, and the solid object is traveling at a predetermined speed (for example, 0 km / h or more) in the same direction as the vehicle 12100. In addition, the microcomputer 12051 can set the vehicle-to-vehicle distance that should be ensured in advance with the front of the vehicle, and perform automatic braking control (including follow-up stop control), automatic acceleration control (including follow-up start control), etc. Therefore, coordinated control such as automatic driving that enables the vehicle to travel autonomously without relying on the driver's operation can be performed.

[0235] For example, based on the distance information obtained from the camera units 12101 to 12104, the microcomputer 12051 can classify the three-dimensional object data related to the three-dimensional object into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, telephone poles and other three-dimensional objects, extract the classified three-dimensional object data, and use the extracted data to automatically avoid obstacles. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that can be visually identified by the driver of the vehicle 12100 and obstacles that are difficult for the driver of the vehicle 12100 to visually identify. In addition, the microcomputer 12051 determines the collision risk used to indicate the degree of danger of collision with each obstacle. When the collision risk is equal to or higher than the set value and thus indicates that there is a possibility of collision, the microcomputer 12051 can output a warning to the driver via the audio speaker 12061 or the display unit 12062, or can implement forced deceleration and evasive steering via the drive system control unit 12010, thereby enabling the microcomputer 12051 to provide assisted driving for avoiding collisions.

[0236] At least one of the camera units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 may identify a pedestrian by determining whether there is a pedestrian in the image captured by the camera units 12101 to 12104. For example, such identification of a pedestrian is performed by the following steps: a step of extracting feature points from the image captured by the camera units 12101 to 12104 as infrared cameras; and a step of performing pattern matching processing on a series of feature points representing the contour of an object to determine whether it is a pedestrian. When the microcomputer 12051 determines that there is a pedestrian in the image captured by the camera units 12101 to 12104 and thus identifies the pedestrian, the sound / image output unit 12052 controls the display unit 12062 to display a rectangular contour line for indicating emphasis in a superimposed manner on the identified pedestrian. The sound / image output unit 12052 may also control the display unit 12062 to display an icon or the like for representing a pedestrian at a desired position.

[0237] An example of a vehicle control system to which the technology according to the present invention is applicable has been described above. The technology according to the present invention can be applied to, for example, the camera unit 12031 in the above-mentioned configuration.

[0238] [9. Conclusion]

[0239] As described above, according to various embodiments of the present invention, the camera module 100 does not need many optical elements such as diffraction gratings and reflective mirrors, so it is possible to avoid complication of the structure and increase in the manufacturing cost of the camera module 100. In addition, according to various embodiments of the present invention, since a composite image is generated using one frame of image, an image obtained by synthesizing a plurality of spectral images into one image can be generated within the imaging time of one frame. That is, according to this embodiment, a composite image of spectral images can be obtained with a simple structure and at a high speed.

[0240] Note that in the above-mentioned embodiments and variations of the present invention, it has been described that a plurality of cut ROIs are superimposed to generate a composite image, but the embodiments and variations of the present invention are not limited thereto. For example, in each embodiment, a frame of image itself may be output, or a ROI cut out from a frame of image may be output. For example, in this case, since images corresponding to each wavelength of light can be acquired and analyzed individually, the presence and distribution of components for each corresponding wavelength can be easily identified.

[0241] [10.Supplement]

[0242] The embodiment of the present invention described above may include, for example, a program for causing a computer to function as the camera system 10 according to the present embodiment, and a non-transitory tangible medium having the program recorded thereon. In addition, the program may be distributed via a communication line such as the Internet (including wireless communication).

[0243] In addition, the various steps in the above-mentioned camera method according to the embodiment of the present invention do not necessarily have to be processed in the order described. For example, the various steps can be processed in a properly reordered manner. In addition, the various steps may not be processed in chronological order, but may be processed partially in parallel or individually. Furthermore, the processing method of each step does not necessarily have to be processed in the described method, and may also be processed by other methods, for example, through other functional units.

[0244] Although the preferred embodiments of the present invention have been described in detail with reference to the accompanying drawings, the technical scope of the present invention is not limited to these examples. Obviously, a person skilled in the art of the present invention may propose various modifications or changes within the scope of the technical concept described in the accompanying claims. Of course, these modifications or changes should be understood to belong to the technical scope of the present invention.

[0245] In addition, the effects described in this specification are only illustrative or exemplary and are not limiting. That is, in addition to the above effects or in place of the above effects, the technology according to the present invention can exert other effects that are obvious to those skilled in the art based on the description of this specification.

[0246] Please note that this technology can also adopt the following technical solutions.

[0247] (1) An imaging device comprising:

[0248] An imaging unit that generates one frame of image by sequentially receiving respective reflected lights reflected by a moving object when the object is intermittently and sequentially irradiated with respective irradiation lights having different wavelengths depending on the position of the moving object, temporarily and sequentially retaining respective signal information based on the reflected lights of the respective wavelengths, and reading out the retained respective signal information together; and

[0249] The synthesis unit generates a synthesized image by cutting out a subject image corresponding to the reflected light of each wavelength from the one frame image and superimposing a plurality of the cut-out subject images.

[0250] (2) The imaging device according to (1) above, wherein

[0251] The imaging unit has a plurality of pixels, and

[0252] Each of the pixels comprises:

[0253] an imaging element that receives the reflected light to generate the signal information; and

[0254] A storage unit temporarily stores the signal information from the imaging element.

[0255] (3) The imaging device according to (2) above, wherein

[0256] The imaging unit operates in a global shutter method that collectively reads out the signal information held in the respective storage units.

[0257] (4) The imaging device according to (2) or (3), wherein

[0258] The pixels include InGaAs imaging elements for detecting near-infrared light.

[0259] (5) The imaging device according to any one of (2) to (4), wherein

[0260] The synthesis unit cuts out each subject image corresponding to the reflected light of each wavelength by cutting out a plurality of predetermined areas designated in advance from the one frame image.

[0261] (6) The imaging device according to any one of (2) to (4), wherein

[0262] The synthesis unit includes an imaging region determination unit that determines each subject image corresponding to the reflected light of each wavelength within the one frame image.

[0263] (7) The imaging device according to (6) above, wherein

[0264] The synthesis unit further includes a binarization processing unit, which converts the one frame image into a binary tone to generate a binary tone image.

[0265] The imaging region determination unit determines each subject image corresponding to the reflected light of each wavelength based on the two-tone image.

[0266] (8) The imaging device according to (6) above, wherein

[0267] The imaging unit sequentially receives: each reference light reflected by the object by intermittently and sequentially irradiating the object moving at a constant speed in a predetermined direction before and after the irradiation of each irradiation light,

[0268] The imaging unit generates the one frame image including the subject image corresponding to the reference light by temporarily and sequentially retaining the signal information based on the respective reference lights and reading out the retained respective signal information together, and

[0269] The imaging region determination unit determines, based on the two object images corresponding to the reference lights, an object image corresponding to the reflected light of each wavelength located between the two object images corresponding to the reference lights.

[0270] (9) The imaging device according to any one of (2) to (8), wherein

[0271] The synthesis unit includes a synthesis processing unit.

[0272] The synthesis processing unit calculates the color parameters of each pixel in the subject image corresponding to the reflected light of each wavelength based on the color information preset corresponding to each wavelength and the signal information of each pixel in the subject image corresponding to the reflected light of each wavelength.

[0273] The synthesis processing unit calculates an arithmetic mean value of the color parameters in the plurality of subject images for each of the pixels, and

[0274] The synthesis processing unit generates a color image as the synthesized image based on the calculated arithmetic mean value.

[0275] (10) The imaging device according to any one of (1) to (9), wherein

[0276] The imaging unit has a plurality of filters.

[0277] The plurality of filters are disposed to face the subject and are sequentially arranged along a moving direction of the subject, and are capable of transmitting light having different wavelengths.

[0278] (11) The imaging device according to (10) above, wherein

[0279] The plurality of filters are on-chip color filters or plasmonic filters.

[0280] (12) The imaging device according to any one of (1) to (9), further comprising:

[0281] An irradiation unit intermittently and sequentially irradiates the subject with the irradiation lights having different wavelengths according to the position of the moving subject.

[0282] (13) The imaging device according to (12) above, wherein

[0283] The irradiation unit includes a plurality of light emitting elements, and the plurality of light emitting elements emit light of different wavelengths.

[0284] (14) The imaging device according to (13) above, wherein

[0285] The plurality of light emitting elements include a plurality of light emitting diodes that emit near infrared light.

[0286] (15) The imaging device according to (13) or (14), wherein

[0287] The plurality of light emitting elements include a reference light emitting element that emits reference light having a predetermined wavelength other than near infrared light.

[0288] (16) The imaging device according to (15) above, wherein

[0289] The reference light emitting element emits visible light as the reference light.

[0290] (17) The imaging device according to any one of (12) to (16), further comprising:

[0291] A control unit controls the imaging unit so that the imaging unit receives the reflected light in synchronization with the irradiation of the irradiation unit.

[0292] (18) Imaging devices, including:

[0293] An imaging unit generates one frame of image by sequentially receiving respective reflected lights reflected by a moving object when the object is intermittently and sequentially irradiated with respective irradiation lights having different wavelengths in accordance with the position of the moving object, temporarily and sequentially retaining respective signal information based on the reflected lights of respective wavelengths, and collectively reading out a portion of the signal information corresponding to a plurality of predetermined areas designated in advance among the respective retained signal information; and

[0294] The synthesis unit generates a synthesized image by cutting out a subject image corresponding to the reflected light of each wavelength from the one frame image and superimposing a plurality of the cut-out subject images.

[0295] (19) Camera system, including:

[0296] A moving device that moves the subject;

[0297] an irradiation device for intermittently and sequentially irradiating the subject with irradiation lights having different wavelengths according to the position of the moving subject;

[0298] An imaging device that generates one frame of image by sequentially receiving respective reflected lights obtained by reflecting the irradiation from the object, temporarily sequentially retaining respective signal information based on the reflected lights of respective wavelengths, and reading out the retained respective signal information together; and

[0299] A synthesis device generates a synthesized image by cutting out a subject image corresponding to the reflected light of each wavelength from the one frame image and superimposing a plurality of the cut-out subject images.

[0300] (20) A method of photographing, comprising:

[0301] Generate one frame of image by sequentially receiving respective reflected lights reflected by the subject when the subject is intermittently and sequentially irradiated with respective irradiation lights having different wavelengths depending on the position of the moving subject, temporarily and sequentially retaining respective signal information based on the reflected lights of respective wavelengths, and reading out the retained respective signal information together; and

[0302] The composite image is generated by cutting out the subject image corresponding to the reflected light of each wavelength from the one frame image and superimposing the cut-out subject images.

[0303] [Explanation of Reference Numerals]

[0304] 10: Camera system

[0305] 100: Camera module

[0306] 110: Irradiation Department

[0307] 112a, 112b, 112c, 112d, 112f: light emitting elements

[0308] 120: Camera devices

[0309] 130: Camera Department

[0310] 132: Lens

[0311] 134: Camera Components

[0312] 136: Storage

[0313] 138: Reading unit

[0314] 140: Synthesis Department

[0315] 142: Binarization processing unit

[0316] 144: Imaging area determination unit

[0317] 146: Synthesis Processing Department

[0318] 150: Control Department

[0319] 160: Filter unit

[0320] 162a, 162b, 162c: Filters

[0321] 200: Control server

[0322] 300: Belt conveyor

[0323] 410: Pixel array unit

[0324] 432: Vertical drive circuit unit

[0325] 434: Column signal processing circuit unit

[0326] 436: Horizontal drive circuit unit

[0327] 438: Output circuit

[0328] 440: Control circuit department

[0329] 442: Pixel drive wiring

[0330] 444: Vertical signal line

[0331] 446: Horizontal signal line

[0332] 448: Input / output terminals

[0333] 480: Peripheral Circuit Department

[0334] 500: Semiconductor substrate

[0335] 800: Subject

[0336] 802: Image

[0337] 804: ROI (Region of Interest)

[0338] 804a, 804b, 804c: pixel data group

[0339] 806: Pseudo-color images

[0340] 900: Electronic equipment

[0341] 902: Camera

[0342] 910: Optical lens

[0343] 912: Shutter mechanism

[0344] 914: Driving circuit unit

[0345] 916: Signal processing circuit unit

Claims

1. Camera devices, including: An imaging unit that generates one frame of image by sequentially receiving respective reflected lights reflected by a moving object when the object is intermittently and sequentially irradiated with respective irradiation lights having different wavelengths depending on the position of the moving object, temporarily and sequentially retaining respective signal information based on the reflected lights of the respective wavelengths, and reading out the retained respective signal information together; and The synthesis unit generates a synthesized image by cutting out a subject image corresponding to the reflected light of each wavelength from the one frame image and superimposing a plurality of the cut-out subject images.

2. The imaging device according to claim 1, wherein: The imaging unit has a plurality of pixels, and Each of the pixels comprises: an imaging element that receives the reflected light to generate the signal information; and A storage unit temporarily stores the signal information from the imaging element.

3. The imaging device according to claim 2, wherein: The imaging unit operates in a global shutter method that collectively reads out the signal information held in the respective storage units.

4. The imaging device according to claim 2, wherein: The pixels include InGaAs imaging elements for detecting near-infrared light.

5. The imaging device according to claim 2, wherein: The synthesis unit cuts out each subject image corresponding to the reflected light of each wavelength by cutting out a plurality of predetermined areas designated in advance from the one frame image.

6. The imaging device according to claim 2, wherein: The synthesis unit includes an imaging region determination unit that determines each subject image corresponding to the reflected light of each wavelength within the one frame image.

7. The imaging device according to claim 6, wherein: The synthesis unit further includes a binarization processing unit, which converts the one frame image into a binary tone to generate a binary tone image. The imaging region determination unit determines each subject image corresponding to the reflected light of each wavelength based on the two-tone image.

8. The imaging device according to claim 6, wherein: The imaging unit sequentially receives: each reference light reflected by the object by intermittently and sequentially irradiating the object moving at a constant speed in a predetermined direction before and after the irradiation of each irradiation light, The imaging unit generates the one frame image including the subject image corresponding to the reference light by temporarily and sequentially retaining the signal information based on the respective reference lights and reading out the retained respective signal information together, and The imaging region determination unit determines, based on the two object images corresponding to the reference lights, an object image corresponding to the reflected light of each wavelength located between the two object images corresponding to the reference lights.

9. The imaging device according to claim 2, wherein: The synthesis unit includes a synthesis processing unit. The synthesis processing unit calculates the color parameters of each pixel in the subject image corresponding to the reflected light of each wavelength based on the color information preset corresponding to each wavelength and the signal information of each pixel in the subject image corresponding to the reflected light of each wavelength. The synthesis processing unit calculates an arithmetic mean value of the color parameters in the plurality of subject images for each of the pixels, and The synthesis processing unit generates a color image as the synthesized image based on the calculated arithmetic mean value.

10. The imaging device according to any one of claims 1 to 9, wherein: The imaging unit has a plurality of filters. The plurality of filters are disposed to face the subject and are sequentially arranged along a moving direction of the subject, and are capable of transmitting light having different wavelengths.

11. The imaging device according to claim 10, wherein: The plurality of filters are on-chip color filters or plasmonic filters.

12. The imaging device according to any one of claims 1 to 9, further comprising: An irradiation unit intermittently and sequentially irradiates the subject with the irradiation lights having different wavelengths according to the position of the moving subject.

13. The imaging device according to claim 12, wherein: The irradiation unit includes a plurality of light emitting elements, and the plurality of light emitting elements emit light of different wavelengths.

14. The imaging device according to claim 13, wherein: The plurality of light emitting elements include a plurality of light emitting diodes that emit near infrared light.

15. The imaging device according to claim 13, wherein: The plurality of light emitting elements include a reference light emitting element that emits reference light having a predetermined wavelength other than near infrared light.

16. The imaging device according to claim 15, wherein: The reference light emitting element emits visible light as the reference light.

17. The imaging device according to claim 12, further comprising: A control unit controls the imaging unit so that the imaging unit receives the reflected light in synchronization with the irradiation of the irradiation unit.

18. Camera devices, including: An imaging unit generates one frame of image by sequentially receiving respective reflected lights reflected by a moving object when the object is intermittently and sequentially irradiated with respective irradiation lights having different wavelengths according to the position of the moving object, temporarily and sequentially retaining respective signal information based on the reflected lights of respective wavelengths, and collectively reading out a portion of the signal information corresponding to a plurality of predetermined areas designated in advance among the respective retained signal information; as well as The synthesis unit generates a synthesized image by cutting out a subject image corresponding to the reflected light of each wavelength from the one frame image and superimposing a plurality of the cut-out subject images.

19. Camera system, including: A moving device that moves the subject; an irradiation device for intermittently and sequentially irradiating the subject with irradiation lights having different wavelengths according to the position of the moving subject; An imaging device that generates one frame of image by sequentially receiving respective reflected lights obtained by reflecting the irradiation from the object, temporarily sequentially retaining respective signal information based on the reflected lights of respective wavelengths, and reading out the retained respective signal information together; and A synthesis device generates a synthesized image by cutting out a subject image corresponding to the reflected light of each wavelength from the one frame image and superimposing a plurality of the cut-out subject images.

20. Video recording methods, including: Generate one frame of image by sequentially receiving respective reflected lights reflected by the subject when the subject is intermittently and sequentially irradiated with respective irradiation lights having different wavelengths depending on the position of the moving subject, temporarily and sequentially retaining respective signal information based on the reflected lights of respective wavelengths, and reading out the retained respective signal information together; and The composite image is generated by cutting out the subject image corresponding to the reflected light of each wavelength from the one frame image and superimposing the cut-out subject images.

Images