Imaging device, imaging method, image processing device, and image processing method

Abstract

An imaging device according to the present technology is provided with an imaging unit including a plurality of linear sensors having different sensitivity wavelength bands. In addition, an image processing device according to the present technology is provided with an image processing unit that performs image processing with respect to captured images for each linear sensor, obtained by means of the imaging unit that includes the plurality of linear sensors having different sensitivity wavelength bands.

Description

Imaging device, imaging method, image processing device, and image processing method

[0001] The present technology relates to an imaging device, an imaging method, an image processing device, and an image processing method, and in particular to a technology relating to imaging using a linear sensor.

[0002] For example, as disclosed in Patent Document 1 below, there is a technology for inspecting an object using a linear sensor. The linear sensor here refers to a sensor that scans the object to obtain an image with a number of horizontal lines greater than the number of horizontal lines in the pixel array of the sensor. Scanning imaging means that imaging is performed while moving at least one of the object and the sensor.

[0003] Special table 2019-502112 publication

[0004] Here, the technology described in Patent Document 1 is configured to obtain captured images of multiple wavelengths using a single linear sensor. Specifically, in Patent Document 1, R (red) light, G (green) light, and B (blue) light are irradiated onto a subject in a time-division manner as illumination light, and the linear sensor captures images for each light irradiation period of the respective colors, thereby obtaining captured images of the respective colors of R, G, and B.

[0005] However, in the technique of Patent Document 1, the images captured at a plurality of wavelengths are acquired in a time-division manner, which means that it takes a long time to acquire the images for inspection.

[0006] This technology has been developed in consideration of the above circumstances, and aims to shorten the time required to complete image processing when images of multiple wavelengths are obtained as images captured by a linear sensor and image processing is performed for purposes such as inspecting a subject.

[0007] An imaging device according to the present technology includes an imaging unit having a plurality of linear sensors with different wavelength sensitivity bands. The imaging unit enables the plurality of linear sensors with different wavelength sensitivity bands to simultaneously obtain captured images in parallel.

[0008] In addition, the image processing device according to the present technology includes an image processing unit that performs image processing on captured images of each linear sensor obtained by an imaging unit having a plurality of linear sensors with different sensitive wavelength bands. With this image processing device, it is possible to perform image processing on captured images of each wavelength simultaneously acquired in parallel by the plurality of linear sensors with different sensitive wavelength bands.

[0009] 1 is an explanatory diagram illustrating an overview of an inspection system as an embodiment; FIG. 1 is an explanatory diagram illustrating an example of a moving body used in an embodiment; FIG. 2 is a diagram illustrating a front view of an optical system in an imaging device as an embodiment; FIG. 3 is a block diagram illustrating an example of the configuration of an imaging device as an embodiment; FIG. 4 is an explanatory diagram illustrating an example of the configuration of a linear sensor used in an embodiment; FIG. 5 is a block diagram illustrating an example of the configuration of an image processing device as an embodiment; FIG. 6 is a functional block diagram for explaining functions of an image processing unit as an embodiment; FIG. 7 is a diagram illustrating examples of captured images of each linear sensor; FIG. 8 is a diagram illustrating an example of a composite image; FIG. 9 is an explanatory diagram of an example of a specific subject; FIG. 10 is a functional block diagram illustrating functions related to an image display method as a first method; FIG. 11 is a flowchart illustrating an example of a processing procedure for realizing the image display method as the first method; FIG. 12 is a functional block diagram illustrating functions related to the image display method as a second method; FIG. 13 is a diagram illustrating an example of a display of an area display object; FIG. 14 is a flowchart illustrating an example of a processing procedure for realizing the image display method as the second method; FIG. 15 is an explanatory diagram of an application example of an inspection system of an embodiment; FIG. 16 is an explanatory diagram of another application example of the inspection system of an embodiment; FIG. 17 is a diagram for explaining an example of the configuration of an imaging unit as a modified example.

[0010] Hereinafter, with reference to the accompanying drawings, an embodiment of a sensor device according to the present technology will be described in the following order: <1. Overview of inspection system> <2. Device configuration example> [2-1. Configuration example of imaging device] [2-2. Configuration example of image processing device] <3. Image processing method as an embodiment> <4. Image display method as an embodiment> [4-1. First method] [4-2. Second method] <5. Application example> <6. Modification example> <7. Summary of embodiment> <8. This technology>

[0011] 1 is a diagram illustrating an overview of an inspection system according to an embodiment. As shown in the figure, the inspection system according to an embodiment includes a camera device 1, an information processing device 2, and a mobile object 3.

[0012] The camera device 1 is an embodiment of an imaging device according to the present technology, and is configured as a camera device that captures images using a linear sensor. Here, in this specification, a linear sensor refers to a sensor that scans an object to obtain an image with a number of horizontal lines greater than the number of horizontal lines in the pixel array of the sensor. Scanning imaging means capturing an image while moving at least one of the object or the sensor. The pixel array of a linear sensor may have more than one horizontal line. In particular, a linear sensor can be defined as one whose pixel array has an aspect ratio (vertical:horizontal) of 1:4 or less.

[0013] In this embodiment, in order to achieve scan imaging, the camera device 1 is attached to a moving body 3. The moving body 3 is capable of moving parallel to an object Ob (object to be inspected) that is the subject of scan imaging, and scan imaging is achieved by the camera device 1 performing imaging while attached to such a moving body 3.

[0014] The information processing device 2 is an embodiment of an image processing device according to the present technology, and is configured as a computer device such as a PC (personal computer), a tablet terminal, a smartphone, etc. The information processing device 2 acquires an image captured by a linear sensor in the camera device 1, and performs image processing on the acquired image. Details of the image processing will be described later.

[0015] In this example, the target object Ob is assumed to be a vertically extending object such as a wall or bridge pier. Accordingly, in this example, a drone configured as shown in FIG. 2 is employed as the mobile body 3. As shown in FIG. 2, the mobile body 3 as a drone in this case is configured with casters 4 that contact the target object Ob, a propeller 5 for generating wind force to press the casters 4 against the target object Ob, and a propeller 6 for generating wind force to push the entire mobile body 3 upward. In this example, multiple propellers 5 (e.g., four) are provided, and only one propeller 6 is provided. This configuration realizes a mobile body 3 that can ascend and descend parallel to the vertically extending target object Ob. The skid 7 in the figure is provided to protect the propeller 6 and its drive motor and other electrical circuit components.

[0016] In this example, the camera device 1 is attached to the mobile body 3 as a drone as described above in an orientation that allows it to capture an image of the object Ob, thereby making it possible to scan and capture an image of the object Ob as the mobile body 3 moves.

[0017] The figure shows an optical system 1a and an illumination unit 1b provided in the camera device 1. The optical system 1a collectively represents optical components such as various lenses and apertures for guiding light from a subject to a linear sensor.

[0018] The illumination unit 1b emits illumination light to illuminate the imaging range of the linear sensor. In this example, the illumination unit 1b is provided in the camera device 1, but the illumination unit 1b can also be provided separately from the camera device 1.

[0019] Here, the camera device 1 of this embodiment includes an imaging unit having multiple linear sensors with different sensitivity wavelength bands as an imaging unit that obtains an image using linear sensors. Specifically, the camera device 1 in this example includes an imaging unit 1c (not shown in FIG. 2 ) that includes a first linear sensor 10 whose sensitivity wavelength band is the visible light wavelength band and a second linear sensor 11 whose sensitivity wavelength band is the infrared light wavelength band. In other words, both visible light images and infrared light images are obtained as images captured by the linear sensors. The visible light image makes it possible to detect scratches and stains visible to the eye on the object Ob. Meanwhile, the infrared light image makes it possible to detect cracks and lifted (peeled) portions on the object Ob. Cracks and lifted portions have a different temperature from other portions, and are therefore captured in the infrared light image due to such temperature differences.

[0020] Furthermore, in the camera device 1 of this embodiment, multiple linear sensors capture images via a common optical system 1a. Fig. 3 shows a front view of the optical system 1a in the camera device 1. In the figure, the relative positions of the first linear sensor 10 and the second linear sensor 11 are indicated by dashed lines. As can be seen from this figure, in this example, the first linear sensor 10 and the second linear sensor 11 capture images via the common optical system 1a.

[0021] As described above, by adopting a configuration in which multiple linear sensors capture images through a common optical system 1a, it is possible to align the optical characteristics of the multiple linear sensors, which makes it possible to simplify the composition process when combining images captured by the linear sensors, as will be described later.

[0022] 4 is a block diagram showing an example of the configuration of the camera device 1. As shown in the figure, the camera device 1 includes the above-described optical system 1a, illumination unit 1b, and imaging unit 1c having first linear sensor 10 and second linear sensor 11, as well as a position detection unit 13, a communication unit 14, and a camera control unit 15.

[0023] In the imaging unit 1c, the first linear sensor 10 and the second linear sensor 11 have an optical filter for selectively receiving light in the visible light band and an optical filter for selectively receiving infrared light, respectively.

[0024] Specifically, in the first linear sensor 10, as schematically shown in FIG. 5A , an optical filter layer 10b is formed on a pixel formation layer 10a to selectively receive light in the visible light band. The optical filter layer 10b is, for example, composed of an optical filter that cuts off light other than the visible light band. By forming such an optical filter layer 10b, the first linear sensor 10 obtains a monochrome image in the visible light band as a captured image. Also, in the second linear sensor 11, as schematically shown in FIG. 5B , an optical filter layer 11b is formed on a pixel formation layer 11a to selectively receive light in the infrared light band. The optical filter layer 11b is, for example, composed of an optical filter that cuts off light other than the infrared light band. This allows the second linear sensor 11 to obtain an infrared light image as a captured image. Here, the pixel formation layer 10a and the pixel formation layer 11a each represent a layer in which a plurality of pixels having light receiving elements are arranged.

[0025] As described above, in a linear sensor, the pixel array may have multiple horizontal lines instead of a single line. When the pixel array has multiple horizontal lines, the linear sensor can perform denoising by averaging the pixel values ​​of multiple pixels arranged in the vertical direction. For example, it is possible to perform denoising by averaging the pixel values ​​for each block, where 3 x 3 = 9 pixels in the pixel array are treated as one block.

[0026] 4 , the imaging unit 1c outputs images captured by the first linear sensor 10 and images captured by the second linear sensor 11 to the camera control unit 15. The imaging unit 1c sequentially outputs the images captured by the first linear sensor 10 and the images captured by the second linear sensor 11 in predetermined frame units. The frame unit here is determined to satisfy the condition that the number of horizontal lines in the captured image is greater than the number of horizontal lines in the pixel array of each of the first linear sensor 10 and the second linear sensor 11. For example, as an illustrative example, the number of horizontal lines in the pixel array of each of the first linear sensor 10 and the second linear sensor 11 may be 4, while the number of horizontal lines in the captured image in frame units may be 1080 or 2160.

[0027] Hereinafter, with regard to images captured by a linear sensor, images captured in frame units will be referred to as "frame images."

[0028] The illumination unit 1b has a light source 12 for illumination light. The illumination unit 1b in this embodiment irradiates light in a wavelength band that includes the sensitive wavelength bands of each linear sensor as illumination light during imaging. Specifically, in this example, a first linear sensor 10 and a second linear sensor 11 are provided as linear sensors, and therefore the illumination unit 1b irradiates light in a wavelength band that covers from the visible light wavelength band to the infrared light wavelength band as illumination light. For this reason, the illumination unit 1b in this example uses a xenon light source as the light source 12. Specifically, the illumination unit 1b in this example uses a xenon lamp as the light source 12. By using a xenon light source as the light source 12, it is possible to irradiate light that covers from the visible light wavelength band to the infrared light wavelength band as illumination light.

[0029] The position detection unit 13 detects the position of the camera device 1. In this example, since the camera device 1 also moves in the vertical direction, three-dimensional position information is detected as the position information. For example, a Global Navigation Satellite System (GNSS) sensor may be used to detect the position of the camera device 1. Alternatively, the position of the camera device 1 may be detected using a technique such as Simultaneous Localization and Mapping (SLAM) using detection information from an Inertial Measurement Unit (IMU). There are various possible specific methods for detecting the position of the camera device 1, and the present invention is not limited to a specific method.

[0030] The camera control unit 15 is configured with a microcomputer having, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory), and the CPU executes various processes according to programs stored in the ROM or programs loaded into the RAM, thereby performing overall control of the camera device 1. For example, the camera control unit 15 controls the execution of the imaging operation by the imaging unit 1c (such as issuing an instruction to perform an operation or an instruction to stop an operation) and controls the turning on and off of the light source 12 in the illumination unit 1b.

[0031] Furthermore, the camera control unit 15 performs a process of associating the position information detected by the position detection unit 13 with the captured images of the first linear sensor 10 and the second linear sensor 11 input from the imaging unit 1c. Specifically, the camera control unit 15 performs a process of associating the position information detected by the position detection unit 13 during the exposure period (exposure period of the first linear sensor 10 and the second linear sensor 11) for obtaining each frame image of the first linear sensor 10 and the frame image of the second linear sensor 11 as "capture position information." For confirmation, this association of the imaging position information is performed for each frame image of the captured images of the first linear sensor 10 and the second linear sensor 11. For example, the association of the imaging position information is performed by assigning the imaging position information as metadata to the frame image.

[0032] As shown in the figure, a communication unit 14 is connected to the camera control unit 15. The communication unit 14 is a communication device provided for performing data communication at least with the information processing device 2, and performs wired or wireless data communication with an external device. The camera control unit 15 performs processing to transmit, via the communication unit 14, the captured images of the first linear sensor 10 and the second linear sensor 11, specifically, frame images associated with imaging position information in this example, to the information processing device 2.

[0033] Here, the captured images and imaging position information of each of the first linear sensor 10 and the second linear sensor 11 may be transmitted to the information processing device 2 in approximately real time, for example, each time a new frame image is obtained. Alternatively, the captured images and imaging position information may be transmitted to the information processing device 2 in a lump in response to an instruction from the information processing device 2 after a series of imaging operations is completed, for example, after scanning the entire predetermined target range of the object Ob. Note that, in the latter case, when the information processing device 2 acquires the captured images and imaging position information in a lump in the latter case after a series of imaging operations is completed, the transmission of the captured images and imaging position information may be performed using means other than the communication unit 14. For example, if the camera device 1 is provided with a drive capable of writing data to a removable recording medium such as a memory card, the captured images and imaging position information may be acquired by the information processing device 2 via the removable recording medium.

[0034] 6 is a block diagram showing an example of the configuration of an information processing device 2. As shown in the figure, the information processing device 2 includes a CPU 21. The CPU 21 executes various processes in accordance with a program stored in a ROM 22 or a program loaded from the storage unit 19 to a RAM 23. The RAM 23 also stores data necessary for the CPU 21 to execute various processes as appropriate. The CPU 21, the ROM 22, and the RAM 23 are connected to one another via a bus 33.

[0035] Furthermore, in the information processing device 2, an image processing unit 24 is connected to the bus 33. The image processing unit 24 performs image processing on the captured images of each linear sensor obtained by the imaging unit 1c. That is, in this example, image processing is performed on the corresponding captured images of the first linear sensor 10 and the second linear sensor 11. Specific examples of the image processing performed by the image processing unit 24 will be described later.

[0036] An input / output interface (I / F) 25 is also connected to the bus 33. An input unit 26 including controls and operation devices is connected to the input / output interface 25. For example, the input unit 26 may be various controls and operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, a remote controller, etc. An operation is detected by the input unit 26, and a signal corresponding to the detected operation is interpreted by the CPU 21.

[0037] A display unit 27, such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, and an audio output unit 28, such as a speaker, are connected integrally or separately to the input / output interface 25. The display unit 27 is used to display various types of information, and is configured, for example, by a display device provided in the housing of the information processing device 2, or a separate display device connected to the information processing device 2, or the like.

[0038] The display unit 27 displays images for various image processing, moving images to be processed, etc. on the display screen based on instructions from the CPU 21. The display unit 27 also displays various operation menus, icons, messages, etc., i.e., a GUI (Graphical User Interface), based on instructions from the CPU 21.

[0039] The input / output interface 25 may be connected to a storage unit 29 configured with an SSD (Solid State Drive) or an HDD (Hard Disk Drive), or a communication unit 30 configured with a modem or the like.

[0040] The communication unit 30 is capable of performing wired or wireless inter-device communication and network communication, which is communication with an external device via a predetermined communication network such as the Internet. In particular, in this embodiment, the communication unit 30 is configured to be able to perform data communication with the communication unit 14 of the camera device 1.

[0041] A drive 31 is also connected to the input / output interface 25 as required, and a removable recording medium 32 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately mounted thereon.

[0042] The drive 31 can read data files such as programs used in various processes from the removable recording medium 32. The read data files are stored in the storage unit 29, and images and sounds contained in the data files are output on the display unit 27 and the audio output unit 28. Furthermore, the computer programs and the like read from the removable recording medium 32 are installed in the storage unit 29 as needed.

[0043] 3. Image Processing Method as an Embodiment A specific method of image processing performed by the information processing device 2 on images captured by multiple linear sensors will be described. Fig. 7 is a functional block diagram for explaining the functions of the image processing unit 24 of the information processing device 2. As shown in the figure, the image processing unit 24 has the functions of a synthesis processing unit F1 and an image analysis processing unit F2.

[0044] The synthesis processing unit F1 performs synthesis processing to synthesize images captured by a plurality of linear sensors. Specifically, the synthesis processing unit F1 in this example performs the synthesis processing to synthesize an image captured by the first linear sensor 10 and an image captured by the second linear sensor 11. This synthesis processing is performed for each frame image of the images captured by the first linear sensor 10 and the second linear sensor 11.

[0045] Figures 8 and 9 are explanatory diagrams of the synthesis processing by the synthesis processing unit F1. In Figure 8, Figure 8A shows an example of an image (frame image) captured by the first linear sensor 10, and Figure 8B shows an example of an image (frame image) captured by the second linear sensor 11, and Figure 9 shows an example of a composite image of these captured images obtained by the synthesis processing.

[0046] 8A , the image captured by the first linear sensor 10 captures features that are visible to a person when visually inspecting the object Ob, such as cracks (represented by thin solid lines in the figure) and stains (represented by black patterns in the figure) formed on the object Ob. On the other hand, the image captured by the second linear sensor 11 captures features that are invisible or difficult to visually inspect when a person visually inspects the object Ob, such as deep cracks (represented by thick solid lines in the figure) and peeling areas (represented by matte finish in the figure).

[0047] In the compositing process by the compositing processor F1, the images captured by these multiple linear sensors are composited so that the positions of the subject are consistent (see the composite image in Figure 9). "Consistent positions" here means that pixels capturing the same position on the subject are composited.

[0048] Such a composite image visualizes objects captured in each wavelength band in a single image, and can therefore be suitably used for inspecting a subject, such as checking for abnormalities in the subject.

[0049] In this example, the compositing process by the compositing processing unit F1 is performed as a process using an AI (Artificial Intelligence) model having a neural network such as a CNN (Convolutional Neural Network). The AI ​​model for realizing the compositing process that ensures consistency in the positions of the subjects as described above can be generated by, for example, performing machine learning (deep learning) using the images captured by the first linear sensor 10 and the images captured by the second linear sensor 11 as learning input data and a correct composite image of these captured images as training data.

[0050] In this example, in which such synthesis processing is performed using AI processing, the image processing unit 24 may be provided with a signal processing unit suitable for convolution operations, such as a DSP (Digital Signal Processor) or FPGA (Field Programmable Gate Array), as a configuration for realizing the synthesis processing.

[0051] 7, the image analysis processing unit F2 performs image analysis processing to determine the presence or absence of a specific subject on the composite image generated by the composition processing unit F1. In this example, the presence or absence of a specific subject is determined by determining the presence or absence of a subject that indicates an abnormality in the object Ob.

[0052] 10 is an explanatory diagram of an example of a specific subject. In this example, the specific subjects indicating an abnormality in the object Ob are a stained portion, a deep cracked portion, and a peeled portion as shown in the figure. As can be understood from the above explanation, the stained portion is mainly captured in the image captured by the first linear sensor 10, and the deep cracked portion and the peeled portion are mainly captured in the image captured by the second linear sensor 11.

[0053] In this example, the image analysis processing of the image analysis processing unit F2 that determines the presence or absence of specific subjects is also performed as AI processing using an AI model. An AI model for implementing this image analysis processing can be generated, for example, as an AI model that extracts features from an input image, calculates the similarity between the features and the features of specific subjects, and determines the presence or absence of specific subjects based on the similarity. Such an AI model can be implemented, for example, using R-CNN (Region-based CNN), Faster R-CNN, or SSD (Single Shot Multibox Detector). In this case, the AI ​​model can be trained using machine learning (deep learning) techniques, for example, using a composite image of an image captured by the first linear sensor 10 and an image captured by the second linear sensor 11 as training input data, and using information about the presence area of ​​the specific subject in the composite image as training data.

[0054] In this case, the image analysis processing by the image analysis processing unit F2 not only determines whether or not a specific subject is present in the input image (composite image), but also outputs information indicating the area in which the specific subject exists as ROI (Region of Interest) information if it is determined that the specific subject is present. In this example, the ROI information is generated as information on a rectangular area that includes the specific subject.

[0055] Here, in this example where image analysis processing is performed using AI processing as described above, it is conceivable that the image processing unit 24 is provided with a signal processing unit suitable for convolution operations, such as a DSP or FPGA, as a configuration for realizing the image analysis processing.

[0056] Although the above example shows the synthesis process by the synthesis processing unit F1 being realized by AI processing, the synthesis process can also be realized as rule-based processing. For example, a method can be used in which corresponding points are detected in the images captured by each linear sensor, as in the corresponding point detection performed in distance measurement processing using the stereo method, and a synthetic image is generated so that the corresponding points overlap. When the synthesis process is realized by rule-based processing, the synthesis process can also be realized by software processing by the CPU 21.

[0057] Furthermore, the image analysis process (determination of the presence or absence of a specific subject) of the image analysis processing unit F2 may also be realized as rule-based processing such as template matching processing, rather than AI processing. When the image analysis process is performed as rule-based processing in this way, the image analysis process may also be realized by software processing of the CPU 21.

[0058] 4. Image display methods according to embodiments 4-1. First method The information processing device 2 according to this embodiment has a function of performing the image processing described above on images captured by a plurality of linear sensors, as well as a function of displaying the images on the display unit 27. Specifically, the information processing device 2 according to this embodiment has a function of displaying a composite image of the images captured by the plurality of linear sensors on the display unit 27. Below, a first method and a second method will be described as examples of methods for displaying such a composite image.

[0059] The first method is a method of displaying a composite image based on a user's operation input. For example, a case can be envisioned in which a plurality of composite images generated by scanning a predetermined area of ​​the object Ob, such as the entire object Ob, are saved, and a composite image selected from the saved composite images based on the user's operation input is presented to the user.

[0060] 11 is a functional block diagram for explaining functions relating to the first method that the CPU 21 of the information processing device 2 has. As shown in the figure, the CPU 21 has functions as a recording processing unit F11 and an image display processing unit F12 as functions relating to the first method.

[0061] The recording processing unit F11 performs a process of recording (storing) the captured images of each linear sensor transmitted from the camera device 1 and the composite image generated by the composition processing unit F1 in a predetermined storage device. Specifically, the recording processing unit F11 in this example performs a process of recording the captured images of each linear sensor transmitted frame by frame from the camera device 1 in the storage unit 19. The recording processing unit F11 also performs a process of recording in the storage unit 19 a composite image (i.e., a composite image of multiple frames) obtained by the composition processing unit F1 performing a composition process on the captured images of each linear sensor transmitted frame by frame from the camera device 1. Here, as described above, in this example, imaging position information is associated with the captured images of each linear sensor. The recording processing unit F11 in this example also records the imaging position information associated with the captured images of each linear sensor in the storage unit 19.

[0062] The image display processing unit F12 performs processing to display the composite image generated by the compositing processing unit F1 on the display unit 27. Specifically, the image display processing unit F12 selects a composite image from among the multiple composite images recorded by the recording processing unit F11 (i.e., the multiple composite images generated by the compositing processing) based on imaging position information in response to a user's operation to specify an imaging position, and performs processing to display the selected composite image on the display unit 27. More specifically, the image display processing unit F12 selects, from the multiple composite images recorded by the recording processing unit F11, a composite image whose imaging position matches (or is closest to) the imaging position specified by the user's operation, and displays the selected composite image on the display unit 27. For clarity, the imaging position of the composite image can be identified by imaging position information of the original captured images.

[0063] Here, the operation of specifying the imaging position by the user may be performed, for example, as an operation of specifying the position of the object Ob on a map, etc. For example, if the object Ob is the wall of a building or a tunnel, the operation may be performed as an operation of specifying the position of the building or tunnel on a three-dimensional map, etc.

[0064] When a composite image of a plurality of wavelengths is generated for each position on the object Ob by the image display method as the first method described above, it is possible to present a composite image of a desired position to the user.

[0065] An example of a specific processing procedure to be executed by the CPU 21 to realize the image display method as the first method described above will be described with reference to the flowchart in Fig. 12. The processing shown in Fig. 12 is executed by the CPU 21 based on a program stored in a predetermined storage device such as the ROM 22.

[0066] First, in step S101, the CPU 21 waits for a position designation operation, that is, waits for a user to designate an image capture position.

[0067] If a position designation operation has been performed, the CPU 21 proceeds to step S102 to search for an image corresponding to the designated position. That is, the CPU 21 selects, from among the multiple composite images recorded (stored) in the storage unit 19, a composite image whose image capture position coincides with (or is closest to) the position designated by the designation operation in step S101.

[0068] In step S103 following step S102, the CPU 21 performs a display process for the searched image, that is, a process for displaying on the display unit 27 the composite image selected by the search process in step S102.

[0069] After executing the display process in step S103, the CPU 21 ends the series of processes shown in FIG.

[0070] It is also possible to display the original image, i.e., the image captured by each linear sensor, together with the composite image. Alternatively, it is also possible to display an image specified by the user for the composite image and the original image. Furthermore, when a specific subject is detected by the image analysis processing unit F2 in the composite image selected based on the position specification operation, a display object (region display object d, described later) indicating the area in which the specific subject exists can be superimposed on the composite image.

[0071] [4-2. Second Method] The second method is a method that assumes that the captured images of each linear sensor are transmitted in approximately real time from the camera device 1. Specifically, the second method is a method in which, when a composite image is displayed on the display unit 27 in approximately real time based on the captured images of each linear sensor transmitted in approximately real time from the camera device 1, if the image analysis processing unit F2 detects a specific subject, the area in which the specific subject exists is displayed on the composite image.

[0072] 13 is a functional block diagram for explaining the function related to the second technique of the CPU 21 of the information processing device 2. As shown in the figure, the CPU 21 in this case has a function as an image display processing unit F12A as a function related to the second technique. The image display processing unit F12A performs a process of superimposing and displaying an area display object d, which is a display object indicating the existence area of ​​a specific subject, on a composite image.

[0073] When the second method is adopted, the synthesis processing unit F1 in the image processing unit 24 sequentially performs synthesis processing on the frame images of the linear sensors sequentially transmitted from the camera device 1. The image analysis processing unit F2 also performs the above-mentioned image analysis processing on the synthesized images sequentially input from the synthesis processing unit F1.

[0074] The image display processing unit F12A sequentially displays the composite images sequentially obtained by the composition processing unit F1 on the display unit 27. Specifically, each time a new composite image is obtained, the composite image displayed on the display unit 27 is switched to the new composite image. Then, while performing the display processing of such composite images, if a specific subject is detected by the image analysis processing unit F2, the image display processing unit F12A performs processing to superimpose an area display object d, which is a display object indicating the area where the specific subject exists, on the composite image.

[0075] 14 shows a display example of a region display object d. In this example, since the ROI information is generated as information on a rectangular region that includes a specific subject, a frame-shaped display object is displayed as the region display object d as shown in the figure.

[0076] The image display method as the second method described above allows the user to be presented with an image of a specific object's presence area, such as an area where an abnormality has occurred, along with the image of the presence area, thereby improving the efficiency of the user's work of inspecting the object.

[0077] A specific example of a processing procedure to be executed by the CPU 21 to realize the image display method as the second method described above will be described with reference to the flowchart of Fig. 15. The processing shown in Fig. 15 is also executed by the CPU 21 based on a program stored in a predetermined storage device such as the ROM 22. The CPU 21 repeatedly executes the processing shown in Fig. 15 for each frame of the image captured by each linear sensor. In addition, in parallel with the processing shown in this figure, the CPU 21 in this case executes a process of sequentially displaying composite images obtained sequentially by the composition processing unit F1 on the display unit 27.

[0078] As shown in the figure, in this case, the CPU 21 performs a process of waiting until a specific subject is detected in step S201, that is, waiting until a specific subject is detected by the image analysis processing unit F2.

[0079] If it is determined in step S201 that any specific subject has been detected, the CPU 21 proceeds to step S202, where it acquires ROI information of the detected specific subject. Then, in the following step S203, the CPU 21 performs a superimposed display process of the region display object d. That is, as the superimposed display process of the region display object d on the composite image, the CPU 21 performs a process of superimposing and displaying the region display object d at a position on the composite image according to the ROI information acquired in step S202.

[0080] After executing the superimposed display process in step S203, the CPU 21 ends the series of processes shown in FIG.

[0081] 5. Application Examples The inspection system according to the embodiment can be applied to a variety of uses. Below, several examples of cases in which the inspection system can be applied will be described.

[0082] For example, as shown in Fig. 16, a case can be exemplified in which a train is used as the moving body 3. Specifically, as shown in <1> in the figure, it is conceivable to apply this to a case in which the camera device 1 is installed on the top of the train and overhead wires are inspected. Alternatively, as shown in <2> in the figure, it is conceivable to apply this to a case in which the camera device 1 is installed on the bottom of the train and tracks are inspected, or as shown in <3>, it is conceivable to apply this to wheel inspection by installing the camera device 1 in a position where it can capture an image of the train wheels.

[0083] 17, a case may be considered in which an automobile is used as the moving body 3. For example, as illustrated in the figure, an application example may be considered in which a camera device 1 is provided on the bottom of an automobile to inspect roads.

[0084] It should be noted that application examples of the inspection system are not limited to those described above, and various other applications are conceivable, and the inspection system is not limited to a specific use.

[0085] 6. Modifications Note that the embodiment is not limited to the specific examples described above, and various modification configurations are possible. For example, although the above example shows that the first linear sensor 10 and the second linear sensor 11 are provided separately, the first linear sensor 10 and the second linear sensor 11 may also be provided integrally.

[0086] 18A, a configuration may be adopted in which a first pixel formation region 10c, which is a pixel formation region (pixel array portion) in the first linear sensor 10, and a second pixel formation region 11c, which is a pixel formation region (pixel array portion) in the second linear sensor 11, are formed on the same semiconductor substrate Bs. The example of Fig. 18A illustrates a configuration in which the first pixel formation region 10c and the second pixel formation region 11c are continuous pixel formation regions, in other words, a configuration in which the gap between the first pixel formation region 10c and the second pixel formation region 11c is 0 pixels.

[0087] 18B , the optical filter layer 10b described above is provided as the optical filter layer on the first pixel formation region 10c, and the optical filter layer 11b described above is provided as the optical filter layer on the second pixel formation region 11c, thereby making it possible to set the sensitive wavelength band of the first pixel formation region 10c to the visible light wavelength band and the sensitive wavelength band of the second pixel formation region 11c to the infrared light wavelength band.

[0088] As described above, by forming the pixel arrays of multiple linear sensors on the same semiconductor substrate, the imaging unit 1cA can be made smaller. In addition, the gap between the pixel arrays of each linear sensor can be narrowed, which reduces the amount of deviation in the imaging position of each linear sensor.

[0089] Furthermore, in the above example, a lamp light source is used as the light source for illumination light for imaging, but light sources other than lamps, such as LEDs (Light Emitting Diodes) and semiconductor lasers, can also be used as the light source.

[0090] Although the above description exemplifies a case where a monochrome image is obtained as an image captured by a linear sensor in the visible light band, it is also possible to obtain color images of R, G, B, etc. For example, as in the above-mentioned Patent Document 1, a method is conceivable in which R, G, and B light are irradiated in a time-division manner to obtain R, G, and B images in a time-division manner. Alternatively, it is conceivable to form an optical filter in a Bayer array pattern such as RGGB as the optical filter in the first linear sensor 10, and perform two-dimensional interpolation processing such as demosaicing to obtain R, G, and B images.

[0091] In addition, in the above example, image processing such as synthesis processing and image analysis processing for images captured by each linear sensor is performed in an information processing device 2 separate from the camera device 1 equipped with the linear sensor, but it is also possible that the image processing is performed in the camera device 1.

[0092] In addition, in the above example, the first linear sensor 10, whose sensitive wavelength band is the visible light wavelength band, and the second linear sensor 11, whose sensitive wavelength band is the infrared light wavelength band, are used as the multiple linear sensors, but the sensitive wavelength bands of each linear sensor are not limited to this and various examples are possible. Furthermore, the number of linear sensors is not limited to two, and a configuration including three or more linear sensors is also possible.

[0093] 7. Summary of the Embodiment As described above, the imaging device (camera device 1) according to the embodiment includes an imaging unit (1c, 1cA) having a plurality of linear sensors with different wavelength sensitivity bands. The imaging unit described above enables the plurality of linear sensors with different wavelength sensitivity bands to simultaneously acquire captured images. Therefore, when capturing images of multiple wavelengths using the linear sensors and performing image processing for inspecting a subject, it is possible to simultaneously acquire images of each wavelength in parallel rather than in a time-division manner, thereby shortening the time required to complete image processing.

[0094] In addition, in the imaging device according to the embodiment, multiple linear sensors capture images via a common optical system (see 1a). This allows the optical characteristics of the multiple linear sensors to be uniform. Therefore, when combining images captured by the individual linear sensors, the combining process can be simplified.

[0095] Furthermore, the imaging device according to the embodiment includes an illumination unit (1b) that irradiates light in a wavelength band that includes the sensitive wavelength bands of each linear sensor as illumination light during imaging. Using illumination light can facilitate capturing a subject in a captured image. Using light in a wavelength band that includes the sensitive wavelength bands of each linear sensor as illumination light enables simultaneous imaging of each linear sensor. Instead of using separate light sources for each sensitive wavelength band of each linear sensor, a single light source that emits light in a wavelength band that includes the sensitive wavelength bands of each linear sensor can be used, which allows for a smaller illumination unit (i.e., a smaller imaging device).

[0096] Furthermore, in the imaging device according to the embodiment, the imaging unit (1c, 1cA) includes a first linear sensor (10) sensitive to the visible light wavelength band and a second linear sensor (11) sensitive to the infrared light wavelength band as a plurality of linear sensors. By using an infrared light image as the image captured by the linear sensor, it is possible to capture objects that cannot be captured by a visible light image. Therefore, when inspecting a subject using an image captured by the linear sensor, the inspection accuracy can be improved.

[0097] In addition, in an image capturing device according to an embodiment, the image capturing unit includes a plurality of linear sensors, including a first linear sensor having a wavelength band sensitive to visible light and a second linear sensor having a wavelength band sensitive to infrared light, and the illumination unit includes a xenon light source (12) for the illumination light. The xenon light source emits light in a wavelength band covering the visible light to infrared light wavelength bands. Therefore, when the first linear sensor for the visible light wavelength band and the second linear sensor for the infrared light wavelength band are used as the linear sensors, there is no need to use a light source for each wavelength band, which allows for the miniaturization of the illumination unit (miniaturization of the image capturing device).

[0098] Furthermore, in the imaging device according to the embodiment, in the imaging unit (same as 1cA), the pixel arrays of the multiple linear sensors are formed on the same semiconductor substrate. This allows for a smaller imaging unit. Furthermore, since the gaps between the pixel arrays of the linear sensors can be narrowed, the amount of deviation in the imaging positions of the linear sensors can be reduced.

[0099] In addition, an imaging method according to an embodiment is an imaging method in which an imaging device captures images using an imaging unit having multiple linear sensors with different sensitive wavelength bands. This imaging method can also achieve the same effects and advantages as the imaging device according to the above embodiment.

[0100] An image processing device (information processing device 2) according to an embodiment includes an image processing unit (24) that performs image processing on images captured by each linear sensor obtained by an imaging unit having multiple linear sensors with different wavelength sensitivity bands. This image processing device makes it possible to perform image processing on images captured at each wavelength simultaneously acquired by multiple linear sensors with different wavelength sensitivity bands. Therefore, when performing image processing for purposes such as inspecting a subject by obtaining images captured by the linear sensors with multiple wavelengths, the time required to complete the image processing can be shortened.

[0101] In addition, in the image processing device according to the embodiment, the image processing unit performs a synthesis process to synthesize images captured by a plurality of linear sensors. This allows a synthesis image to be generated by synthesizing images captured at a plurality of wavelengths. Such a synthesis image visualizes an object captured in each wavelength band in a single image, and therefore can be suitably used for inspecting an object, such as checking for abnormalities in the object.

[0102] Furthermore, in the image processing device according to the embodiment, the imaging unit includes a first linear sensor having a wavelength band sensitive to visible light and a second linear sensor having a wavelength band sensitive to infrared light, and the image processing unit performs a synthesis process to synthesize an image captured by the first linear sensor and an image captured by the second linear sensor. By using an infrared light image as the image captured by the linear sensor, it is possible to capture an object that cannot be captured by a visible light image. Therefore, a synthesized image suitable for inspecting a subject, such as checking for abnormalities in the subject, can be obtained.

[0103] Furthermore, in the image processing device according to the embodiment, the image processing unit performs image analysis processing to determine the presence or absence of a specific subject on the composite image generated by the composition processing. By performing the image analysis processing to determine the presence or absence of a specific subject as described above, it becomes possible to determine the presence or absence of an abnormality. Since it is possible to determine the presence or absence of an abnormality on the composite image of images captured at multiple wavelengths, it is possible to improve the accuracy of subject inspection.

[0104] In addition, in the image processing device according to the embodiment, captured images are associated with imaging position information indicating the imaging position, and an image display processing unit (F12) is provided that processes a composite image generated by the compositing process on a display unit. The image display processing unit selects a composite image from among multiple composite images generated by the compositing process based on the imaging position information in response to an operation to specify the imaging position, and then displays the selected composite image on a display unit. This allows a user to view a composite image at a desired position when multiple wavelength composite images are generated for each position of the subject. By associating captured images with imaging position information, the user can display the desired composite image simply by specifying the imaging position, eliminating the need for the user to review the image contents of multiple composite images to search for the desired composite image. This reduces the workload for the user when viewing a desired composite image.

[0105] Furthermore, the image processing device according to the embodiment includes an image display processing unit (F12A) that performs processing to display the composite image generated by the composition processing on the display unit, and the image display processing unit performs processing to superimpose an area display object, which is a display object indicating the area where a specific subject exists, on the composite image. This allows the area where a specific subject exists, such as an area where an abnormality has occurred, to be presented to the user along with an image of the area. This improves the efficiency of the user's work of inspecting the subject.

[0106] In addition, an image processing method as an embodiment is an image processing method in which an image processing device performs image processing on captured images obtained by an imaging unit having multiple linear sensors with different sensitive wavelength bands. This image processing method can also achieve the same effects and advantages as the image processing device as the above embodiment.

[0107] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0108] <8. Present Technology> The present technology may also have the following configurations. (1) An imaging device including an imaging unit having a plurality of linear sensors with different sensitive wavelength bands. (2) The imaging device described in (1), in which the plurality of linear sensors perform imaging via a common optical system. (3) The imaging device described in (1) or (2), in which an illumination unit irradiates light in a wavelength band that includes the sensitive wavelength bands of the linear sensors as illumination light during imaging. (4) The imaging device described in any of (1) to (3), in which the imaging unit includes, as the plurality of linear sensors, a first linear sensor whose sensitive wavelength band is a visible light wavelength band and a second linear sensor whose sensitive wavelength band is an infrared light wavelength band. (5) The imaging device described in (3), in which the imaging unit includes, as the plurality of linear sensors, a first linear sensor whose sensitive wavelength band is a visible light wavelength band and a second linear sensor whose sensitive wavelength band is an infrared light wavelength band, and the illumination unit includes a xenon light source as a light source of the illumination light. (6) The imaging device according to any one of (1) to (5), wherein in the imaging section, pixel array sections of the plurality of linear sensors are formed on the same semiconductor substrate. (7) An imaging method, wherein the imaging device performs imaging using an imaging section having a plurality of linear sensors with different sensitivity wavelength bands. (8) An image processing device comprising an image processing section that performs image processing on images captured by each of the linear sensors obtained by an imaging section having a plurality of linear sensors with different sensitivity wavelength bands. (9) The image processing device according to (8), wherein the image processing section performs a synthesis process to synthesize images captured by the plurality of linear sensors. (10) The image processing device according to (9), wherein the imaging section has, as the plurality of linear sensors, a first linear sensor whose sensitivity wavelength band is a visible light wavelength band and a second linear sensor whose sensitivity wavelength band is an infrared light wavelength band, and the image processing section performs the synthesis process to synthesize an image captured by the first linear sensor and an image captured by the second linear sensor. (11) The image processing device according to (9) or (10), wherein the image processing unit performs an image analysis process to determine whether or not a specific subject is present in a composite image generated by the composite process.(12) The image processing device according to any of (9) to (11), wherein the captured images are associated with imaging position information indicating imaging positions, and the image display processing unit performs processing to display a composite image generated by the compositing processing on a display unit, the image display processing unit selecting a composite image from among a plurality of composite images generated by the compositing processing based on the imaging position information in response to an operation to specify an imaging position, and displaying the selected composite image on the display unit. (13) The image processing device according to (11), wherein the image display processing unit performs processing to display the composite image generated by the compositing processing on a display unit, and the image display processing unit performs processing to superimpose an area display object that is a display object indicating an area where the specific subject exists on the composite image. (14) An image processing method, wherein the image processing device performs image processing on captured images obtained by an imaging unit having a plurality of linear sensors with different sensitive wavelength bands, for each of the linear sensors.

[0109] REFERENCE SIGNS LIST 1 camera device 2 information processing device 3 moving body Ob object 4 caster 5, 6 propeller 7 skid 1a optical system 1b lighting unit 1c, 1cA imaging unit 10 first linear sensor 11 second linear sensor 12 light source 13 position detection unit 14 communication unit 15 camera control unit 10a, 11a pixel formation layer 10b, 11b optical filter layer 21 CPU 24 image processing unit 29 storage unit 30 communication unit 31 drive 32 removable recording medium 33 bus F1 synthesis processing unit F2 image analysis processing unit F11 recording processing unit F12, F12A image display processing unit d area display object Bs semiconductor substrate 10c first pixel formation area 11c second pixel formation area

Claims

1. An imaging device equipped with an imaging section having multiple linear sensors with different sensitive wavelength bands.

2. The imaging device according to claim 1, wherein the plurality of linear sensors capture images via a common optical system.

3. The imaging device according to claim 1, further comprising an illumination unit that irradiates light of a wavelength band that includes the sensitive wavelength band of each of the linear sensors as illumination light during imaging.

4. The imaging device according to claim 1, wherein the imaging unit has as the plurality of linear sensors a first linear sensor whose sensitive wavelength band is the visible light wavelength band and a second linear sensor whose sensitive wavelength band is the infrared light wavelength band.

5. The imaging device according to claim 3, wherein the imaging unit has as the plurality of linear sensors a first linear sensor whose sensitive wavelength band is the visible light wavelength band and a second linear sensor whose sensitive wavelength band is the infrared light wavelength band, and the illumination unit has a xenon light source as a light source of the illumination light.

6. The imaging device according to claim 1, wherein in said imaging section, pixel array sections of said plurality of linear sensors are formed on a single semiconductor substrate.

7. An imaging method in which an imaging device captures images using an imaging section having multiple linear sensors with different sensitive wavelength bands.

8. An image processing device comprising an image processing unit that performs image processing on the captured images obtained by an imaging unit having a plurality of linear sensors with different wavelength bands of sensitivity.

9. The image processing device according to claim 8, wherein the image processing unit performs a synthesis process for synthesizing the images captured by the multiple linear sensors.

10. The image processing device according to claim 9, wherein the imaging unit has as the plurality of linear sensors a first linear sensor whose sensitive wavelength band is the visible light wavelength band and a second linear sensor whose sensitive wavelength band is the infrared light wavelength band, and the image processing unit performs as the synthesis process a process of synthesizing an image captured by the first linear sensor and an image captured by the second linear sensor.

11. The image processing device according to claim 9, wherein the image processing unit performs image analysis processing to determine the presence or absence of a specific subject on the composite image generated by the composition processing.

12. An image processing device as described in claim 9, further comprising an image display processing unit which performs processing for displaying a composite image generated by the compositing process on a display unit, the image display processing unit selecting a composite image from among a plurality of composite images generated by the compositing process based on the image capture position information in response to an operation for specifying an image capture position, and displaying the selected composite image on the display unit.

13. An image processing device as described in claim 11, further comprising an image display processing unit that performs processing to display a composite image generated by the synthesis processing on a display unit, wherein the image display processing unit performs processing to superimpose an area display object, which is a display object indicating the area in which the specific subject exists, on the composite image.

14. An image processing method in which an image processing device performs image processing on images captured by each linear sensor obtained by an imaging unit having multiple linear sensors with different sensitive wavelength bands.

Images