Imaging device

Abstract

To provide a technique for imaging both a visual image and an infrared image, and imaging a high-quality image by improving color reproducibility in visible light imaging.SOLUTION: An imaging system is configured to enable stereo imaging for both a visible image and an infrared image, and improve color reproducibility in visible light imaging. The imaging system includes: two imaging sensors 12; and two DBPFs which are optical filters having a visible light band and a second wavelength band with transparency and arranged for the two imaging sensors. The imaging system has at least four types of filter units which are different in spectral transmission characteristics in accordance with a wavelength in the visible light band and similar in transmittance in the second wavelength band, and two color filters arranged for the two imaging sensors, respectively. The imaging system measures a distance to an object based on two visible or infrared image signals.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to an imaging device and an imaging system.

Background Art

[0002] In recent years, with the combination of image recognition technology and biometric authentication, the development of surveillance cameras that detect criminals, shoplifters, terrorists, etc. through face recognition, and in-vehicle cameras used for automatic driving of automobiles has been underway.

[0003] For example, in a surveillance camera, there is a known system that uses a stereo camera to detect distance and detect the intrusion of suspicious persons (see Patent Document 1). Patent Document 1 describes a "distance measurement image recognition device using a stereo image" as a method for measuring the shape and distance of an object, and a technique called the stereo method for obtaining the distance from a stereo image is known. In this stereo method, first, two left and right images called stereo images are input, and corresponding points of the left and right images (where the target object at a certain position in the left image appears in the right image) are obtained by calculating feature amounts in the images. Details of how to obtain corresponding points are described, for example, in Patent Document 2 as an "image matching method". Then, when the corresponding points of the two left and right images are obtained, the distance to the object surface can be calculated based on the principle of triangulation, so the distance to the object and the shape of the object surface can be known.

[0004] Also, Patent Document 1 proposes a moving object recognition device that can detect a moving object with high accuracy and high speed and measure its shape and distance by using the known correspondence relationship of stereo images.

[0005] Surveillance cameras and cameras for autonomous driving require shooting regardless of location or time, such as outdoors and indoors, day and night, but in some situations, sufficient lighting may not be available. In such cases, infrared photography using infrared illumination invisible to the human eye can be considered. Similarly, for cameras for autonomous driving, infrared photography using infrared light as illumination to light distant objects can be considered, taking into account the impact of headlights on oncoming vehicles at night. In any case, it is conceivable to shoot in visible light without illumination during the daytime when there is likely to be sufficient visible light, and to use infrared illumination, which is difficult for the human eye to see, for infrared photography when illumination is required at night.

[0006] Considering these circumstances, it is preferable for surveillance cameras and cameras for autonomous driving to be able to capture both visible light and infrared light simultaneously.

[0007] In imaging devices such as surveillance cameras that continuously capture images day and night, infrared light is detected and captured during nighttime hours. Photodiodes (light-receiving elements), which are the light-receiving parts of imaging sensors such as CCD sensors and CMOS sensors, can receive light up to the near-infrared wavelength band of about 1300 nm. Therefore, in principle, imaging devices using these imaging sensors can capture images up to the infrared band.

[0008] Since the wavelength range of light to which humans have the highest visual sensitivity is 400nm to 700nm, when an image sensor detects near-infrared light, the image will appear redder to the human eye. For this reason, when shooting during the day or in bright indoor locations, it is desirable to install an infrared cut filter in front of the image sensor to block infrared light and remove light with a wavelength of 700nm or more, in order to match the sensitivity of the image sensor to human visual sensitivity. On the other hand, when shooting at night or in dark places, it is necessary to shoot without an infrared cut filter.

[0009] Such imaging devices include those that allow manual attachment and removal of infrared cut filters, and those that automatically insert and remove infrared cut filters. Furthermore, imaging devices that eliminate the need for the insertion and removal of infrared cut filters have been disclosed. For example, an optical filter has been proposed that has transmission characteristics in the visible light band, cutoff characteristics in a first wavelength band adjacent to the longer wavelength side of the visible light band, and transmission characteristics in a second wavelength band which is a part of the first wavelength band (see Patent Document 3). With this filter, light can be transmitted in both the visible light band and the second wavelength band located away from the visible light band on the longer wavelength side of the visible light band, i.e., on the infrared side.

[0010] Hereafter, an optical filter that transmits light in the visible light band and a second wavelength band outside the infrared range, while blocking light in other wavelength bands, will be referred to as a DBPF (Double Band Pass Filter).

[0011] Furthermore, in recent years, various authentication technologies have been developed for biometric authentication, including fingerprint, facial, iris, vein, signature, voiceprint, and gait recognition. However, when used in conjunction with image recognition technologies captured by the aforementioned surveillance cameras and cameras for autonomous driving, facial recognition and iris recognition are the most common examples. [Prior art documents] [Patent Documents]

[0012] [Patent Document 1] Japanese Patent Application Publication No. 3-81878 [Patent Document 2] Japanese Patent Application Publication No. 62-107386 [Patent Document 3] Patent No. 5009395 [Overview of the project] [Problems that the invention aims to solve]

[0013] In the DBPF described in Patent Document 3, light in a second wavelength band included in the infrared (near-infrared) wavelength band (a relatively narrow wavelength band included in the infrared wavelength band) is not constantly blocked and light is transmitted. In other words, unlike when an infrared cut filter is used that cuts out wavelengths longer than the visible light band, when imaging in the visible light band, the infrared light transmitted through the second wavelength band will have a considerable influence.

[0014] For imaging in the visible light band, color filters are used in image sensors that capture color images. A color filter has areas (filter sections) for red, green, and blue arranged in a predetermined pattern corresponding to each pixel of the image sensor. Basically, it has a peak in light transmittance in the wavelength band of each color and blocks the transmission of light in the wavelength bands of other colors.

[0015] However, in wavelengths longer than the visible light band, light transmittance differs depending on the color region and wavelength, but basically, light is transmitted. Therefore, if infrared light is transmitted in the second wavelength band outside the infrared range, as in the DBPF described above, this infrared light passes through the color filter and reaches the photodiode (light-receiving element) of the image sensor, increasing the amount of electrons generated by the photoelectric effect in the photodiode.

[0016] Furthermore, when performing both color photography using visible light and photography with infrared illumination, for example, a color filter in which the regions of red, green, and blue are arranged in a predetermined pattern would have an infrared region (infrared region) that has a peak in light transmittance in the second wavelength band mentioned above. That is, the arrangement (pattern) of the color filter would consist of four regions: red (R), green (G), blue (B), and infrared (IR). In this case, the infrared region blocks light in the visible light band and mainly transmits light in the second wavelength band. Therefore, it is conceivable to remove the infrared light component from the image signals of the red, green, and blue colors by utilizing the infrared light image signal output from the image sensor that receives light that has passed through the infrared region of the color filter. However, even with such signal processing, it was difficult to achieve color reproduction that was nearly equivalent to that when using an infrared cut filter for color photography. In addition, when calculating distance by stereo imaging, a difference in the signal levels of the left and right sides caused errors in the parallax calculation.

[0017] Furthermore, when using facial recognition with camera images, there are two main types: one where users are required to align their faces with a designated camera, such as in access control systems for offices and buildings, or for boarding procedures and immigration control at airports; and another where users are unconsciously photographed by multiple cameras in public facilities, airports, and transportation systems for purposes such as tracking criminals, preventing terrorism, and quickly identifying suspicious individuals. In the former type, the limited shooting conditions allow for accurate recognition even with recent technology, but in the latter type, the varying lighting conditions, face orientation, and angle due to environmental changes greatly affect the recognition rate.

[0018] To implement suspicious person detection using surveillance cameras, it would be essential to enable 24-hour continuous recording using both visible and infrared light, regardless of location or time of recording. Furthermore, if the captured images could be as clear and high-resolution as possible with minimal noise, the ability to detect suspicious individuals early and analyze the situation when a crime occurs would be significantly enhanced.

[0019] Furthermore, by using a configuration that allows for stereo distance measurement with two cameras, even in cameras for autonomous driving, when infrared and visible light imaging are used in combination as described above, it is possible to obtain clear images with less noise, thereby improving the accuracy of image recognition.

[0020] Therefore, it is desirable to have a system that can simultaneously capture both infrared and visible images, maintain image quality (noise, resolution, color reproduction, etc.) at or above the same level as a normal visible image without infrared, and further enable stereo imaging with a two-camera configuration.

[0021] This invention provides a technology that enables the capture of both visible and infrared images, and improves color reproduction during visible light imaging, thereby enabling the acquisition of high-quality images. [Means for solving the problem]

[0022] The imaging device or imaging system according to the present invention is characterized by comprising: an image sensor; a filter unit having the characteristic of transmitting at least visible light wavelengths and infrared wavelengths, and filtering signals from the image sensor based on this characteristic; a signal processing unit processing the signals filtered by the filter unit and outputting visible light signals and infrared signals; a motion region extraction unit generating information about motion objects in an image captured by the image sensor from the infrared signals output from the signal processing unit; and a signal output control unit transmitting to the outside first data including at least one of the visible light signals or infrared signals output from the signal processing, and second data based on the information about motion objects generated by the motion region extraction unit.

[0023] Furthermore, the imaging device or imaging system according to the present invention includes two imaging elements, at least two filter units having the property of transmitting both the visible light wavelength region and the infrared wavelength region, and filtering the signals from the imaging elements based on this property, two signal processing units that process the signals filtered by the filter units to output visible light signals and infrared signals, a distance calculation unit that calculates the distance to a subject shown in a visible image by a visible image signal and / or an infrared image by an infrared image signal using the two visible image signals and / or two infrared image signals output from the signal processing unit, a moving object region extraction unit that generates information regarding a moving object in an image captured by the imaging element from the infrared signal output from the signal processing unit, and a signal output control unit that transmits to the outside first data including at least one of the visible light signal or the infrared signal output from the signal processing, second data based on the information regarding the moving object generated by the moving object region extraction unit, and third data based on the distance information calculated by the distance calculation unit. This is another feature of the present invention.

Advantages of the Invention

[0024] According to the present invention, it is possible to obtain high-quality images. More specifically, for example, according to one aspect of the present invention, since it is possible to simultaneously capture both high-quality infrared images and visible images with a camera composed of one imaging sensor and one optical film, it is possible to improve visibility even in environmental changes such as insufficient lighting or at night. Also, according to another aspect of the present invention, the distance to an object can be measured more accurately with both infrared images and visible images, and the information can be provided to an external system together with the visible image or the infrared image. Also, according to another aspect of the present invention, by using an infrared image instead of a visible image, the moving object region in the image can be extracted more quickly, and the information can be provided to an external system together with the visible image or the infrared image.

Brief Description of the Drawings

[0025] [Figure 1] It is a schematic diagram showing an imaging system according to Embodiment 1 of the present invention. [Figure 2]This is a schematic diagram showing the configuration of the imaging sensor section of the imaging system according to Embodiment 1 of the present invention. [Figure 3] This graph shows the DBPF and color filter transmittance spectra of the imaging sensor and color filter of the imaging system according to Embodiment 1 of the present invention. [Figure 4] This is a schematic diagram showing one example of the configuration of a color filter in an imaging system according to Embodiment 1 of the present invention. [Figure 5] This is a block diagram illustrating the signal processing unit of an imaging system according to Embodiment 1 of the present invention. [Figure 6] This is a flowchart illustrating the communication flow between the camera and controller of the imaging system according to Embodiment 1 of the present invention. [Figure 7] This figure shows one configuration of various image information handled by the imaging system according to Embodiment 1 of the present invention. [Figure 8] This is a schematic diagram showing an imaging system according to Embodiment 2 of the present invention. [Figure 9] This figure shows one configuration of various analysis metadata information handled by the imaging system according to Embodiment 2 of the present invention. [Figure 10] This is a flowchart illustrating the communication flow between the camera and the controller of the imaging system according to Embodiment 2 of the present invention. [Figure 11] This figure shows an example of an image screen of an image handled by the imaging system according to Embodiment 2 of the present invention. [Figure 12] This figure shows an example of the configuration of analysis metadata information handled by the imaging system according to Embodiment 2 of the present invention. [Figure 13] This is a schematic diagram showing an imaging system according to Embodiment 3 of the present invention. [Figure 14] This is a schematic diagram showing another configuration example of the imaging system according to Embodiment 3 of the present invention. [Figure 15] This is a schematic diagram showing another configuration example of the imaging system according to Embodiment 3 of the present invention. [Figure 16] This is a schematic diagram showing another configuration example of the imaging system according to Embodiment 3 of the present invention. [Figure 17] This figure shows one configuration of various image information handled by the imaging system according to Embodiment 3 of the present invention. [Figure 18] This figure shows one configuration of various image information handled by the imaging system according to Embodiment 3 of the present invention. [Figure 19] This figure shows one configuration of analysis metadata information handled by the imaging system according to Embodiment 3 of the present invention. [Figure 20] This figure shows an example of the configuration of analysis metadata information handled by the imaging system according to Embodiment 3 of the present invention. [Figure 21] This graph shows the DBPF and color filter transmittance spectra of the imaging sensor and color filter of the imaging system according to Embodiment 1 of the present invention. [Figure 22] This diagram shows the processing flow of the imaging device in the imaging system according to Embodiment 3 of the present invention. [Figure 23] This is an example of operating the imaging system according to Embodiment 3 of the present invention. [Figure 24] This is an example of operating the imaging system according to Embodiment 3 of the present invention. [Modes for carrying out the invention]

[0026] (Embodiment 1) Embodiments of the present invention will be described below with reference to the drawings.

[0027] Figure 1 shows an example configuration of an imaging system according to Embodiment 1 of the present invention. The imaging system is broadly composed of one or more imaging devices 100 ((a) to (n)) and one or more controller devices 200. The imaging devices 100 and the controller devices 200 are connected via a network 303. In this embodiment, the network 303 is described assuming a wired LAN (Local Area Network), but it may also be a general-purpose network such as wireless LAN (WiFi), USB (Universal Serial Bus), or IEEE1394.

[0028] Network 303 uses the standard IP (Internet Protocol) as the network protocol, and TCP (Transmission Control Protocol) and UDP (User Datagram Protocol) as higher-level transport protocols. For transferring images captured by the imaging device 100, even higher-level application protocols such as RTP (Real-time Transport Protocol) / RTCP (RTP Control Protocol) or HTTP (Hyper Text Transfer Protocol) are used, and RTSP (Real-Time Streaming Protocol) is used for transfer control. Either IPv4 or IPv6 may be used for IP. Furthermore, communication between higher-level applications is possible using web services utilizing technologies such as HTTP and RTP as described above. Although not shown in the diagram, internet connectivity is also possible via hubs and routers.

[0029] The controller device 200 is capable of controlling multiple imaging devices 100 and can also exchange information with other controller devices 200.

[0030] The imaging system of this embodiment can be used in applications and services such as surveillance and access control.

[0031] The imaging device 100, a feature of this embodiment, consists of a lens 11, an imaging sensor unit 12, a signal processing unit 13, a signal output control unit 14, a communication control unit 15, an IF unit 16, an anomaly detection unit 17, an illuminance monitoring unit 18, a GPS 19, a clock 20, a memory 21, a maintenance IF unit 22, a control unit 23, and an infrared LED 24.

[0032] Lens 11 is an optical lens for imaging that forms an image of visible light 301 and infrared light (invisible light) 302 from a subject at a predetermined focal length on the image sensor unit 12, and is composed of multiple lenses.

[0033] The imaging sensor unit 12 is the part that outputs multiple pixel signals corresponding to predetermined wavelength components by spectrally analyzing the visible light and infrared light imaged by the lens 11 using various filters and converting them into photoelectric signals.

[0034] The signal processing unit 13 processes the output signal from the imaging sensor unit 12 and applies image processing such as internal processing, removal of the effects of infrared light that has passed through the second wavelength band during color shooting, gamma correction, white balance, and RGB matrix correction to the image signal, and outputs the output signal for the visible image and the output signal for the infrared image.

[0035] The visible image signal and infrared image signal output from the signal output control unit 14 and the signal processing unit 13, which "captured the target object at the same time," are transmitted via the IF unit 16 to a predetermined controller device 200 connected to the network, according to instructions from the communication control unit 15 or the control unit 23.

[0036] The communication control unit 15 controls the image signal output from the signal output control unit 14 via the IF unit 16 and transmits and receives control signals to and from the controller device 200 via the IF unit. It is also the part that executes the aforementioned network protocols, application protocols, and web services.

[0037] The IF section 16 is a communication IF section that connects the imaging device 100 and the network 303.

[0038] The abnormality detection unit 17 is a part that constantly or periodically monitors whether there are any abnormalities in the hardware or software of the imaging device 100 and detects any abnormalities. For example, this could be the case if the imaging device 100 is removed from its designated installation location, if it becomes unable to capture images, if it becomes unable to communicate with the network, or if there is unauthorized access.

[0039] The illuminance monitoring unit 18 is a part that continuously or periodically monitors the brightness of the imaging range of the imaging device 100 using an illuminance sensor or the like. If it detects that the illuminance is insufficient, it notifies the control unit 23 and illuminates the infrared LED 24.

[0040] The GPS 19 is the part that obtains the current position of the imaging device 100 itself from position information received from satellites. The obtained position information can also be notified to the controller device 200 via the IF unit 16.

[0041] Clock 20 is the part that manages the current time information and sets / cancels the timer. The time information is automatically adjusted using general-purpose technologies such as NTP (Network Time Protocol) or standard radio waves.

[0042] Memory 21 includes a storage device (ROM (Read-Only Memory) area, FROM (Flash ROM) area) for storing programs, various setting information, and property information, and a storage device (RAM (Random Access Memory) area) for loading and temporarily storing these programs and data, and for storing work data. Here, in addition to the built-in memory, external memory (USB memory or NAS (Network-Attached Storage)) or portable media (microflash, SD card, magnetic tape, etc.) may also be used as recording devices.

[0043] The maintenance IF unit 22 is the interface (IF) section through which maintenance personnel of the imaging device 100 communicate for program update processing and diagnosis in the event of a malfunction. Furthermore, when the abnormality detection unit 17 detects an abnormality, it can automatically notify a remote maintenance site of the detected abnormality.

[0044] The control unit 23 is the part that comprehensively controls the operation of each of the above-mentioned components.

[0045] Meanwhile, the controller device 200 consists of a user interface unit 201, a display unit 202, a clock 203, a memory 204, a recording and playback unit 205, a communication control unit 206, an interface unit 207, a camera management unit 208, a motion area extraction unit 209, a face area detection unit 210, a face feature point detection unit 211, a face matching unit 212, a face database 213, and a control unit 214.

[0046] The user interface section 201 is the part where the user operates the controller device 200 using a remote control, touch panel, keyboard, mouse, buttons, etc.

[0047] The display unit 202 is the part that displays the operation screen of the controller device 200, visible images and infrared images received via the network 303, facial recognition results, warning screens, etc., on an external or built-in monitor.

[0048] Clock 203 is the part that manages the current time information and sets / cancels the timer. The time information is automatically adjusted using general-purpose technologies such as NTP and standard radio waves.

[0049] Memory 204 consists of a storage device (ROM area, FROM area) for storing programs, various setting information, and property information, and a storage device (RAM area) for loading and temporarily storing these programs and data, and for storing work data. Here, in addition to the built-in memory, external memory (USB memory or NAS) or portable media (microflash, SD card, DVD, Blu-ray® Disc, magnetic tape, etc.) may also be used as recording devices.

[0050] The recording and playback unit 205 is responsible for recording and playing back visible images, infrared images, and associated metadata received via the network 303 and the IF unit 207 in the memory 204. Encryption, decryption, compression, and decompression of the data to be recorded are performed as needed.

[0051] The communication control unit 206 is the part that sends and receives control signals between the network 303 and the imaging device 100 via the IF unit 207. It is also the part that executes the aforementioned network protocols, application protocols, and web services.

[0052] The IF section 207 is a communication IF section that connects the controller device 200 and the network 303.

[0053] The camera management unit 208 is the part that manages one or more imaging devices 100 managed by the control device 200 via the network 303. It is the part that creates, maintains, updates, and deletes information about the imaging devices 100 under management (for example, IP address, installation location, manufacturer name, model name, installation date / operating hours, functional specifications, maintenance contact information, etc.).

[0054] The motion region extraction unit 209 is responsible for extracting moving objects such as people, animals, and other objects present in visible or infrared images received via the IF unit 207 or recorded by the recording / playback unit 205, and acquiring their positional information. Methods for extracting moving objects in an image include creating difference images (for example, the difference between the first and second images, and the difference between the second and third images) from multiple consecutive images (for example, three images) and comparing them to extract moving objects, or using background subtraction while generating a background image from the captured image to extract moving objects.

[0055] The face region detection unit 210 detects areas containing human faces either directly from visible or infrared images received via the IF unit 207 or recorded by the recording / playback unit 205, or from moving regions extracted by the moving region extraction unit 209. Detection methods include techniques such as the high-speed face detection algorithm using Viola & Johns' integral image and cascade classifier.

[0056] The facial feature point detection unit 211 is responsible for detecting feature points such as eyes, nose, and corners of the mouth within the facial region detected by the facial region detection unit 210. This enables image position correction so that facial features can be accurately extracted.

[0057] The face matching unit 212 selects the most suitable features for identifying an individual from the feature points detected by the face feature point detection unit 211 and performs matching using the face database 213. Here, the features used to distinguish faces can include methods that use the entire grayscale information within the face region (e.g., the eigenfactor method using principal component analysis), methods that use the interval and directional components of local grayscale changes within the face region as features (e.g., Elastic Bunch Graph Matching), or methods that combine both of these methods. Furthermore, methods such as the nearest neighbor method and linear discriminant analysis can be applied as matching methods.

[0058] The face database 213 stores data containing pre-registered face images for matching by the face matching unit 212 in an internal or external storage medium. Images artificially generated from these registered images, such as changes in lighting or face orientation, can also be registered. For example, in an access control system, face images of users or employees who are permitted to enter a specific area are registered. It is also possible to register additional images that have been confirmed to be of the person as a result of face recognition at a specific location. Here, this face database 213 may be an external database accessible via the network 303, rather than being on the controller device 200. For example, in surveillance camera systems at airports, a face database of suspects or terrorists provided by the police or law enforcement agencies is used. The database may also be shared among multiple controller devices.

[0059] The control unit 214 is the part that comprehensively controls the operation of each of the above-mentioned components. Furthermore, if the face matching unit 212 finds that the person is not a pre-registered user (e.g., a suspicious person) or matches a suspect, it can automatically generate a report based on a predetermined format, notify the administrator and police, and send the report to them.

[0060] Figure 2 shows an example of the configuration of the imaging sensor unit 12 of the camera unit 100.

[0061] The image sensor unit 12 consists of a sensor body 2, a color filter 3, a cover glass 4, and a DBPF 5.

[0062] Sensor body 2 is a CCD (Charge Coupled Device) image sensor, and it is the part where a photodiode is arranged as a light-receiving element for each pixel. A CMOS (Complementary Metal·Oxide Semiconductor) image sensor may be used instead of a CCD image sensor.

[0063] The color filter 3 is provided on the sensor body 2 and is a portion in which the red (R), green (G), blue (B), and infrared (IR) regions are arranged in a predetermined sequence for each pixel of the sensor body 2. Figure 4 shows variations of the color filter used in this embodiment.

[0064] The cover glass 4 covers and protects the sensor body 2 and the color filter 3.

[0065] DBPF5 is an optical filter formed on the cover glass 4. DBPF5 is an optical filter that has transmission characteristics in the visible light band, cutoff characteristics in a first wavelength band adjacent to the longer wavelength side of the visible light band, and transmission characteristics in a second wavelength band which is a part of the first wavelength band. The placement position of DBPF5 is not limited and may be provided on the lens 11, for example.

[0066] Figure 3 shows the transmittance spectra of the R, G, B, and IR filters of the color filter 3, with transmittance on the vertical axis and wavelength on the horizontal axis. The wavelength range in the graph includes the visible light band and a portion of the near-infrared band, for example, the wavelength range of 300 nm to 1100 nm.

[0067] As shown by the double line in the graph, the filter section of R exhibits nearly maximum transmittance at a wavelength of 600 nm, and maintains nearly maximum transmittance even beyond 1000 nm on the longer wavelength side.

[0068] As shown in the graph (widely spaced dashed line G), the filter section of G has a peak with maximum transmittance at a wavelength of approximately 540 nm, and a minimum transmittance at a longer wavelength of approximately 620 nm. Furthermore, the transmittance of the filter section of G tends to increase at longer wavelengths than the minimum transmittance, reaching its maximum at approximately 850 nm. Beyond that wavelength, the transmittance remains at its maximum even beyond 1000 nm.

[0069] As shown in graph B (the closely spaced dashed line), the filter section of B has a peak where the transmittance is maximum at a wavelength of approximately 460 nm, and a minimum at a longer wavelength of approximately 630 nm. Furthermore, the transmittance increases at longer wavelengths, reaching its maximum at approximately 860 nm, and remains at its maximum even beyond 1000 nm.

[0070] The IR filter section blocks short-wavelength light from around 780nm and long-wavelength light from around 1020nm, with maximum transmittance in the 820nm to 920nm range.

[0071] The transmittance spectra of the R, G, B, and IR filter sections are not limited to those shown in Figure 3, etc., but the color filter 3 currently in general use is expected to show a transmittance spectrum close to this. Note that a value of 1 on the horizontal axis, which represents transmittance, does not mean that 100% of light is transmitted, but rather represents, for example, the maximum transmittance of the color filter 3.

[0072] In this embodiment, the DBPF5 used has high transmittance in two bands, as shown by DBPF(solid line) in the graph: the visible light band represented by DBPF(VR) and the infrared band (second wavelength band) represented by DBPF(IR), which is located slightly further away from the visible light band at longer wavelengths. Furthermore, DBPF(VR), which is the visible light band with high transmittance, has a wavelength band of approximately 370nm to 700nm. Also, DBPF(IR), which is the second wavelength band with high transmittance in the infrared region, has a wavelength band of approximately 830nm to 970nm.

[0073] In this embodiment, the relationship between the transmittance spectra of each filter section of the color filter 3 and the transmittance spectrum of DBPF5 is defined as follows. Specifically, DBPF(IR), which is the second wavelength band that transmits infrared light in the transmittance spectrum of DBPF5, is included in wavelength band A shown in Figure 2, where the transmittance of each filter section is approximately the same, with the R filter section, G filter section, and B filter section all having approximately the maximum transmittance, and is also included in wavelength band B, where the IR filter section transmits light with the maximum transmittance.

[0074] Here, wavelength band A, where the transmittance of each filter section (R, G, and B) is the same, is defined as the region where the difference in transmittance between each filter section is 10% or less. Note that, at wavelengths shorter than this wavelength band A, the transmittance of the G and B filters becomes lower than that of the R filter section, which has approximately the highest transmittance. In DBPF5, the region where there is a difference in the transmittance of each filter section (R, G, and B) corresponds to the region between DBPF(VR), which has high transmittance in the visible light band, and DBPF(IR), which has high transmittance in the second wavelength band of the infrared light band, where the transmittance of DBPF5 is minimal and approximately blocks light. In other words, on the infrared side, light transmission is cut off in the region where the difference in transmittance between the R, G, and B filter sections is large, and light is transmitted in wavelength band A, where the transmittance of each filter section is maximum and the transmittances are the same, at longer wavelengths.

[0075] From the above, it can be concluded that in this embodiment, the DBPF5 used in place of the infrared light cut filter has a region that transmits light not only in the visible light band but also in the second wavelength band on the infrared side. Therefore, when taking color photographs with visible light, the light that has passed through the second wavelength band will be affected. However, as mentioned above, the second wavelength band does not transmit light in the portion where the transmittance differs in the R, G, and B filter sections, and only transmits light in the wavelength band where the transmittance of each filter section is maximized and the transmittance is the same.

[0076] Furthermore, in the second wavelength band of the DBPF5, the IR filter section transmits light in the portion where its transmittance is maximum. Therefore, assuming that four very close pixels irradiated with the same light each have R, G, B, and IR filter sections, light will pass through the R, G, B, and IR filter sections similarly in the second wavelength band. As far as the infrared light, the same amount of light will reach the photodiode of the image sensor body through each filter section, including IR. In other words, the amount of light passing through the second wavelength band in the infrared band among the light transmitted through each of the R, G, and B filters is the same as the amount of light passing through the IR filter section. Assuming the above, the difference between the output signal of the pixel assumed above from the sensor body 2 that receives light transmitted through the R, G, and B filters, and the output signal of the pixel assumed above from the sensor body 2 that receives light that has passed through the IR filter, becomes the output signal of the visible light portion of each R, G, and B filter, after the infrared light that has passed through each R, G, and B filter has been cut out.

[0077] In reality, with color filter 3, one of the R, G, B, or IR filter sections is placed at each pixel of the sensor body 2, and it is highly likely that the amount of light of each color irradiated to each pixel will be different. Therefore, for example, it is possible to determine the brightness of each color at each pixel using a well-known interpolation method, and then use the difference between the interpolated R, G, and B brightness of each pixel and the similarly interpolated IR brightness as the R, G, and B brightness, respectively. Note that the image processing method for removing the infrared light component from the brightness of each R, G, and B color is not limited to this, and any method that can ultimately cut out the influence of light that has passed through the second wavelength band from the brightness of each R, G, and B color may be used. In any method, DBPF5 cuts out the portion where the transmittance of the R, G, and B filter sections differs by more than 10% in the infrared region, that is, the portion where the transmittance differs by a predetermined percentage, making it easy to remove the influence of infrared light at each pixel.

[0078] As described above, by using the image sensor unit 12, an imaging device 100 capable of both color and infrared light imaging can be realized. Generally, it is conceivable to perform normal imaging in color and infrared imaging at night using infrared light illumination, which is difficult for humans to perceive, without using visible light illumination. For example, in various surveillance cameras, when nighttime imaging is performed in locations where nighttime illumination is not required or is preferable, it is conceivable to perform nighttime imaging using infrared light illumination. It can also be used for applications such as daytime and nighttime imaging for observing wild animals.

[0079] When using infrared light for nighttime photography, infrared illumination is necessary because, like visible light, the amount of light is insufficient at night, even though it is infrared light.

[0080] The transmittance spectra (A) and (B) of the DBPF5 shown in Figure 21 were determined by considering the transmittance spectra of the R, G, B, and IR filter sections, as well as the emission spectrum of the infrared light used for illumination, such as the infrared LED used for illumination.

[0081] Figure 21 shows the transmittance spectra R, G, B, and IR of the filter sections for each color, similar to Figure 2, as well as the transmittance spectrum DBPF of DBPF5, and the emission spectrum IR-light of the LED lighting.

[0082] The second wavelength band shown in Figure 21(A), represented by DBPF(IR), which is the infrared light-transmitting portion of the DBPF, is contained within wavelength band A shown in Figure 2, where the R filter section, G filter section, and B filter section all have approximately the maximum transmittance and the transmittance of each filter section is approximately the same. It is also contained within wavelength band B, where the IR filter section transmits light with maximum transmittance.

[0083] In addition, the wavelength band of DBPF(IR) is such that almost the entire wavelength band that forms the peak of the emission spectrum of infrared illumination, which is included in both wavelength band A and wavelength band B mentioned above, is included in the wavelength band of DBPF(IR). Note that when infrared imaging is performed under infrared illumination rather than natural light at night, the second wavelength band indicated by DBPF(IR) does not need to be wider than the peak width of the optical spectrum of the infrared illumination. If the spectrum of the infrared illumination is included in both wavelength band A and wavelength band B mentioned above, the peak portion of the transmittance of DBPF5 indicated by DBPF(IR) may be set as the second wavelength band with a peak width approximately the same as the peak width of the peak of the emission spectrum of the infrared illumination, for example, with a peak of around 860.

[0084] In other words, in Figure 21(A), the peak in the emission spectrum of infrared illumination, shown as IR-light, is located on the shorter wavelength side of the aforementioned wavelength bands A and B, and the second wavelength band of DBPF, shown as DBPF(IR), overlaps with the peak in the emission spectrum of IR-light in the shorter wavelength portion of wavelength bands A and B.

[0085] Furthermore, the graph shown in Figure 21(B) is similar to (A) in that it adds the emission spectrum of infrared illumination to the graph shown in Figure 2, and aligns the second wavelength band shown by DBPF(IR), which is the part of the DBPF5 transmittance spectrum with high transmittance in the infrared region, with the peak of the emission spectrum shown by IR-light from the aforementioned infrared illumination.

[0086] In Figure 21(B), the infrared illumination used has a longer wavelength peak in its emission spectrum than in (A). This peak is included in the aforementioned wavelength bands A and B, and is located on the longer wavelength side of wavelength bands A and B. Correspondingly, the second wavelength band indicated by DBPF(IR) in DBPF5 is set to overlap with the peak indicated by IR-light for infrared illumination within the aforementioned wavelength bands A and B.

[0087] The second wavelength band of DBPF5 may be any of those shown in Figure 2 or Figure 21, as long as the second wavelength band is included in both wavelength band A and wavelength band B described above. Furthermore, if the wavelength band that corresponds to the peak of the emission spectrum of the infrared illumination used in nighttime infrared imaging is determined, it is preferable to include that wavelength band in both wavelength band A and wavelength band B described above, and to align the second wavelength band of DBPF5 with the peak of the emission spectrum of the infrared illumination.

[0088] In such an image sensor, the second wavelength band through which light is transmitted outside the infrared region of the DBPF5 is included in wavelength band A where the transmittance of each filter section is maximum outside the infrared region of the R, G, B, and IR filters, and where the transmittance of each filter section is the same, as well as wavelength band B where the transmittance of the IR filter section is maximum. In other words, on the wavelength side longer than the visible light band, the transmittance of the R filter section is maximum, while the transmittance of the G and B filters is not maximum. As a result, light in the portion where the transmittance of the R, G, and B filters is not the same is cut off by the DBPF5.

[0089] In other words, since the R, G, B, and IR filter sections are designed to transmit light in the second wavelength band outside the infrared region, the transmittance outside the infrared region is the same for all of these filter sections. If the same amount of light in the second wavelength band is irradiated, the amount of transmitted light will be the same for each of the R, G, B, and IR filter sections. As a result, as described above, it is possible to correct the color based on the output signal from the pixels corresponding to each of the R, G, and B filter sections, and easily obtain an image that suppresses the influence of infrared light passing through the second wavelength band of color during color photography.

[0090] Furthermore, by aligning the second wavelength band with the peaks of the emission spectra of infrared illumination contained in the aforementioned wavelength bands A and B, the infrared illumination light can be used efficiently, and the width of the second wavelength band can be narrowed, thereby reducing the influence of infrared light passing through the second wavelength band during color imaging.

[0091] Figure 5 shows a block diagram of the signal processing in the signal processing unit 12 described above.

[0092] The processing overview for the output signal from the image sensor unit 12 equipped with the color filter shown in Figure 4 will be explained below.

[0093] The output signals of each pixel (R, G, B, IR) are sent to the respective internal processing blocks 21r, 21g, 21b, and 21ir. In each internal processing block 21r, 21g, 21b, and 21ir, the R, G, B, and IR signals are converted using interpolation (interpolation) methods known to the public, so that the image data of each frame of the color filter 3 described above consists of image data 20r where all pixels are represented by red (R), image data 20g where all pixels are represented by green (G), image data 20b where all pixels are represented by blue (B), and image data 20ir where all pixels are represented by infrared (IR), respectively.

[0094] Next, in the infrared light rejection signal creation blocks 22r, 22g, 22b, and 22ir, signals to be subtracted from the R, G, and B signals are generated from the IR signal in order to remove the influence of infrared light received from the second wavelength band mentioned above. The signals created for each of R, G, and B in these infrared light rejection signal creation blocks 22r, 22g, and 22b are then subtracted from the R, G, and B signals. In this case, as mentioned above, for the same pixel, it is basically sufficient to remove the IR signal from each of the R, G, and B signals, making the processing easier. In reality, since the sensitivity differs for each pixel of each color due to the characteristics of the filter section of each pixel, signals to be subtracted from each of the R, G, and B signals are created from the IR signal for each R, G, and B image.

[0095] Next, in the image processing block 23, the R, G, and B signals undergo a well-known RGB matrix processing, which corrects the color by transforming the R, G, and B signals using a matrix; a well-known white balance processing, which ensures that the output values ​​of the R, G, and B signals are the same in the white areas of the image; and a well-known gamma correction, which is a correction for image output to a display or the like. Then, in the luminance matrix block 24, the luminance Y signal is generated by multiplying the R, G, and B color signals by coefficients. Furthermore, the RY and BY color difference signals are calculated by dividing the luminance Y signal by the blue B signal and the red R signal, and the Y, RY, and BY signals are output.

[0096] Furthermore, IR signals are basically output as a black and white gradient image.

[0097] Figure 6 shows the communication flow for exchanging visible images, infrared images, and control commands between the imaging device 100 and the controller device 200 shown in Figure 1. Here, the communication flow may use a proprietary communication protocol, but it may also use a protocol such as the one developed by ONVIF (Open Network Video Interface Forum) as a standard communication protocol for surveillance cameras.

[0098] First, the imaging device 100 is installed in a designated location and connected to the network 303. When the power is turned on, the imaging device 100 starts up, and the control unit 23 of the imaging device 100 performs initial setup processing. This mainly involves hardware startup and initial software parameter setting processing, such as loading a program stored in memory 21 and acquiring the current location from GPS 19. Alternatively, the device may be configured to start up when connected to the network 303 using a PoE (Power Over Ethernet) hub (step 601).

[0099] Once the necessary initial setup process is complete, the control unit 23 of the imaging device 100 sets the IP address to be used by the communication control unit 15 and the IF unit 16. The IP address is set using a general-purpose network protocol, such as by directly connecting a PC or pad terminal to the maintenance IF unit 22 and setting a static IP address, or by automatically setting an IP address using DHCP (Dynamic Host Configuration Protocol) (step 602).

[0100] Once the IP address configuration is complete, the control unit 23 of the imaging device 100 instructs the communication control unit 15 to notify the controller device 200 of its presence. Protocols such as UPnP (Universal Plug and Play) or WS-Discovery (Web Services Dynamic Discovery) may be used to automatically discover devices on the network. The notification may also include the manufacturer's name, model number, installation location, and date / time (step 603). The installation location may be pre-configured information or information obtained from the GPS 19. Information determining whether the location is indoors or outdoors using the GPS 19 or the illuminance monitoring unit 18 may also be included.

[0101] Upon receiving the aforementioned notification, the control unit 214 of the controller device 200 can recognize the presence of the imaging device 100 by obtaining its IP address. The control unit 214 notifies the administrator via the display unit 202 that a new imaging device 100 has been connected and waits for instructions from the administrator regarding whether or not it will manage the imaging device 100. If instructions are received from the administrator via the user interface unit 201, or if the controller device 200 checks the number of imaging devices 100 currently under its management and confirms that the maximum number has not been reached, it automatically instructs the communication control unit 206 to send a request to the imaging device 100 to acquire information on its installed functions (step 605).

[0102] Upon receiving the request to acquire the function information, the control unit 23 of the imaging device 100 acquires the function information stored in the memory 21 and instructs the communication control unit 15 to transmit the function information to the controller device 200. The function information includes, for example, device management information (support status and parameter values ​​for networks, systems, and security), imaging device performance information (parameter values ​​related to image quality such as backlight compensation, brightness, contrast, white balance, focus adjustment, and wide dynamic range, and parameter values ​​related to media profiles such as resolution, frame rate, and codec type), PTZ (pan-tilt-zoom) function information (coordinate system definition, movable parameters, preset positions, etc.), and analysis function information (supported analysis functions, types of authentication, analysis result format, etc.) (step 606).

[0103] Here, Figure 7 shows one configuration of information regarding the imaging device 100 of this embodiment. The imaging device 100 of this embodiment has "visible image" and "infrared image" as output image types 701, and four types of output modes 702 transmitted to the controller device 200: "output visible image only," "output infrared image only," "automatically switch between outputting one or the other depending on illuminance and time," and "output both visible image and infrared image simultaneously." In addition, as visible image access destination information 703, there is URI / URL information that the controller device 200 accesses to obtain information about the visible image and the actual visible image from the imaging device 100, and similarly, as infrared image access destination information 704, there is URI / URL information that the controller device 200 accesses to obtain information about the infrared image and the actual infrared image from the imaging device 100. Furthermore, as visible image information 705, it includes the codec, transfer rate, and resolution of the outputtable visible image, and similarly, as infrared image information 706, it includes the codec, transfer rate, and resolution of the outputtable infrared image. This configuration is just one example, and other information may be included.

[0104] The control unit 214 of the controller device 200, upon receiving the functional information of the imaging device 100, notifies or automatically confirms the contents of the functional information to the administrator via the display unit 202. If it is decided that the controller device 200 will manage the device, it adds it to the camera management unit 208 as a managed device. The camera management unit 208 stores and manages all or part of the functional information in the memory 204. The control unit 214 also checks the analysis and authentication functions supported by the controller device 200 itself and determines whether or not to use the images from the imaging device 100. Alternatively, it may also check the analysis function information supported by the imaging device 100 and determine the authentication method and analysis method to be executed when using the imaging device 100 (step 607).

[0105] When the control unit 214 of the controller device 200 decides to use the imaging device 100, the control unit 214 instructs the communication control unit 206 to set any parameters that need to be changed or set from the functional information acquired in step 606, and sends a device setting request to the imaging device 100. For example, in this embodiment, the output mode 702 is set to "simultaneous output of both visible and infrared images" (step 608). Here, for example, the output mode 702 may be determined based on the installation location of the imaging device 100.

[0106] Upon receiving the aforementioned equipment setting request, the control unit 23 of the imaging device 100 checks whether the received setting is executable and returns the result of the execution to the controller device 200 (step 609).

[0107] Next, the control unit 214 of the controller device 200 instructs the communication control unit 206 to send a request to obtain access destination information in order to acquire the protocols and parameters necessary for actually acquiring visible images and infrared images (step 610).

[0108] Upon receiving the access destination information acquisition request, the control unit 23 of the imaging device 100 instructs the communication control unit 15 to return access destination information for the media, including the access destination information 703 for the visible image and the access destination information 704 for the infrared image (for example, media type, port number, transfer protocol, payload number, etc.) (step 611).

[0109] Upon receiving the access destination information, the control unit 214 of the controller device 200 subsequently sends a request to the imaging device 100 to obtain session information (DESCRIBE) necessary for receiving the image (step 612).

[0110] Upon receiving the session information acquisition request, the control unit 23 of the imaging device 100 instructs the communication control unit 15 to generate session information described using SDP (Session Description Protocol) and transmit it to the controller device 200 (step 613).

[0111] Upon receiving the session information, the control unit 214 of the controller device 200 instructs the communication control unit 206 to establish an RTSP session with the imaging device 100. Here, RTSP sessions are usually established separately for the transfer of visible images and for the transfer of infrared images (step 614).

[0112] After establishing the RTSP session, the controller device 200 prepares to receive these images and to perform facial recognition (step 615), the imaging device 100 prepares to transmit visible and infrared images (step 616), and transmits the results (step 617).

[0113] Once the control unit 214 of the controller device 200 confirms that all preparations are complete, it instructs the communication control unit 206 to send a streaming start request (PLAY) to the imaging device 100 (step 618).

[0114] Upon receiving the streaming start request, the control unit 23 of the imaging device 100 instructs the signal output control unit 14 to output the image as requested by the controller device 200 in step 608, and instructs the communication control unit 15 to transmit the image output by the signal output control unit 14 to the imaging device 100 using RTP over the session established in steps 612 / 613 (step 620).

[0115] Furthermore, the control unit 214 of the controller device 200 starts receiving images (step 621).

[0116] Subsequently, the RTP transfer of the visible and infrared images captured by the imaging device 100 is performed (steps 621, 622). Here, the communication control unit 15 of the imaging device 100 may use the marker bits of the RTP header to identify frame breaks in order to reduce the processing load on the controller device.

[0117] Furthermore, the communication control unit 15 of the imaging device 100 sends an RTCP transmission report to the controller device 200 each time a predetermined number of frames are transmitted. To indicate that visible and infrared images were captured simultaneously, the same timestamp, frame number, packet count, etc., are stored in the report (step 623).

[0118] The control unit 214 of the controller device 200, which receives visible and infrared images from the imaging device 100, stores these images in the memory 204 via the recording and playback unit 205, while performing face authentication using the motion region extraction unit 209, face region detection unit 210, face feature point detection unit 211, and face matching unit 212. Then, it controls the interruption or stopping of streaming as needed (step 624).

[0119] The above describes the basic communication flow between the controller device 200 and the imaging device 100.

[0120] Here, the above communication flow uses RTP communication, but HTTP communication or other proprietary communication methods can also be used. In addition, instead of transferring visible and infrared images in separate streams, they can be superimposed on the same stream (for example, a common header (including timestamp and sequence number) + first visible image + first infrared image + ...). Also, since transferring both images simultaneously will increase the communication bandwidth usage, infrared images can be transferred one frame at a time, and visible images every 30 frames. In this case as well, for frames captured at the same time, the same timestamp and frame number should be used for both the infrared and visible images.

[0121] In step 623, the imaging device 100 sends a transmission report to the controller device 200, but similarly, the controller device 200 may send a reception report to the imaging device 100 that includes information about packet loss and transmission delays.

[0122] Furthermore, although step 623 states that a transmission report with the same timestamp and frame number is sent to indicate that the visible and infrared images were captured simultaneously, there are other methods, such as setting the same value for the timestamp and sequence number in the RTP header to be transmitted, or setting the same timestamp and frame number in the extended header of the RTP header.

[0123] The control unit 214 of the controller device 200 instructs the recording and playback unit 205 to store the received visible and infrared images in the memory 204. Using the motion region extraction unit 209, face region detection unit 210, face feature point detection unit 211, and face matching unit 212, it detects people included in the images and performs face authentication to determine whether or not they are suspicious. At this time, it is possible to improve authentication accuracy by performing face authentication on both images, taking advantage of the fact that visible and infrared images of the same object can be acquired at the same time, and that the same timestamp and sequence number are added to both images, making synchronization of the two images easy. Alternatively, face authentication is usually performed on only one of the images (for example, only the infrared image), and when it is necessary to compare and confirm both images (for example, to grasp additional information such as background or color, or to use the other image for face authentication), the other image with the same sequence number can be used.

[0124] Furthermore, in this embodiment, the output mode 702 was set to "simultaneous output of both visible and infrared images" in step 608. However, it is also possible to change this setting depending on the time of day and the surrounding environment, such as setting "simultaneous output of both visible and infrared images" during the day and "infrared image only" at night. Alternatively, when performing face recognition while receiving either one of the images, if the face matching unit 212 finds a person who appears suspicious and wants to obtain further information, it is possible to automatically switch to receiving both images midway through the process.

[0125] If the control unit 214 of the controller device 200 determines, based on the results of the face matching unit 212, that there is a suspicious person or a suspected suspicious person in the image, it will notify the administrator via the display unit 202, or notify and share information with other controller devices 200 via the IF unit 207, making it possible to track the suspicious person among multiple imaging devices 100.

[0126] (Embodiment 2) Next, the configuration of the imaging system according to Embodiment 2 of the present invention will be described.

[0127] Figure 8 shows an example configuration of an imaging system according to Embodiment 2 of the present invention. Note that the imaging device 100 of Embodiment 1 and the imaging devices 800 and 810 of this embodiment can be installed together in multiple locations on the same imaging system, and the controller device 200 can manage these imaging devices.

[0128] The imaging device 800 of this embodiment is configured by adding a motion region extraction unit 801, which has the same function as the motion region extraction unit 209 of the controller device 200, to the imaging device 100 of Embodiment 1. The other components are configured the same as those of the imaging device 100.

[0129] The control unit 23 of the imaging device 800 inputs only the infrared image from the visible image and infrared image output by the signal processing unit 13 to the motion region extraction unit 801. Reasons for using only the infrared image include its ability to detect objects that cannot be detected in the visible image, and its greater contrast between humans and the background compared to the visible image, making it effective for human detection.

[0130] The motion region extraction unit 801 extracts motion regions within the image using the input infrared image and outputs their number and position information. These results are output to the control unit 25 or the signal output control unit 14. They may also be stored in the memory 22.

[0131] As described above, the imaging device 800 can constantly monitor the moving object region within the image using the infrared image from among the visible and infrared images captured at the same time, and provide the controller device 200 with highly accurate extracted information about the moving object region along with the visible or infrared image. Since the controller device 200 can acquire information about the moving object region along with the image, the image processing burden can be reduced.

[0132] Here, the control unit 23 of the imaging device 800 may output an image from the signal output control unit 14 via the IF unit 16 only when a motion region is extracted by the motion region extraction unit 801, and if a motion region cannot be extracted, it may not output an image from the signal output control unit 14, or it may reduce the frame rate of the image output from the signal output control unit 14.

[0133] Furthermore, the control unit 23 of the imaging device 800 may instruct the signal output control unit 14 to combine the motion region extracted by the motion region extraction unit 801 with the visible image and / or infrared image output by the signal processing unit 13 to generate / process an image in which the motion region is enclosed by a rectangle.

[0134] Similarly, the imaging device 810 of this embodiment is configured by mounting a face region detection unit 802, which has the same function as the face region detection unit 210 of the controller device 200, onto the imaging device 800. The other components are configured the same as those of the imaging device 100.

[0135] The control unit 23 of the imaging device 810 inputs only the infrared image from the visible image and infrared image output by the signal processing unit 13 to the motion region extraction unit 801. The motion region extraction unit 801 uses the input infrared image to extract motion regions within the image and outputs their number and position information to the control unit 23 or the signal output control unit 14, while simultaneously inputting this information to the face region detection unit 802. The face region detection unit 802 detects the regions containing human faces from the input motion regions and outputs this information to the control unit 23 or the signal output control unit 14.

[0136] As described above, the imaging device 810 can constantly monitor the moving object region within the image using the infrared image from among the visible and infrared images captured at the same time, extract the moving object region with high accuracy, and further detect the region where a human face exists from that moving object region. It can then provide the controller device 200 with information on the moving object region and the face region along with the visible or infrared image. Since the controller device 200 can acquire this information along with the image, the image processing burden can be reduced.

[0137] Here, the control unit 23 of the imaging device 810 may output an image from the signal output control unit 14 via the IF unit 16 only when the face region detection unit 802 detects a human face region, in order to reduce the amount of communication bandwidth used on the network. If a human face region cannot be detected even after extracting the motion region, the signal output control unit 14 may not output an image, or it may reduce the frame rate of the image output from the signal output control unit 14. Similarly, in order to detect only objects, the signal output control unit 14 may output an image via the IF unit 16 only when it detects a motion region in which a human face region could not be detected.

[0138] Furthermore, the control unit 23 of the imaging device 810 may instruct the signal output control unit 14 to combine the face region extracted by the face region detection unit 802 (and the motion region extracted by the motion region extraction unit 801) with the visible image and / or infrared image output by the signal processing unit 13 to generate / process an image in which the face region is enclosed by a rectangle.

[0139] The communication flow between the imaging devices 800 and 810 and the controller device 200 is almost the same as that described in Figure 6 of Embodiment 1, but the differences are described below.

[0140] First, in step 606 of Figure 6, the functional information that imaging devices 800 and 810 provide to the controller device 200 includes, for example, the information shown in Figure 7, as well as the information shown in Figure 9, as the aforementioned analysis function information. Specifically, the functional information includes content indicating that imaging device 800 is equipped with a "motion region extraction function," and content indicating that imaging device 801 is equipped with both a "motion region extraction function" and a "face region detection function."

[0141] In this embodiment, this information is used as analysis metadata, and the functional information (analysis function information) includes the type of analysis metadata 901, the output mode of the analysis metadata 902, the access destination information for the motion region metadata 903, the access destination information for the face region metadata 904, the access destination information for the motion region / face region metadata 905, the motion region metadata information 906, and the face region metadata information 907, as shown in Figure 9.

[0142] Upon receiving the aforementioned functional information, the control unit 214 of the controller device 200 checks the analysis and authentication functions supported by the controller device 200 itself in step 607 and determines whether or not to use the analysis metadata output by the imaging device 800. For example, it is possible to choose to use only the analysis metadata "location information of the moving body region" for both imaging devices 800 and 810, or not use the analysis metadata of imaging device 800 at all and use only the "location information of the face region" from the analysis metadata of imaging device 810.

[0143] Figure 10 shows the communication flow for transmitting visible images, infrared images, and analysis metadata between the imaging devices 800 and 810 and the controller device 200. In this explanation, it is assumed that imaging device 800 transmits "location information of the moving body region," and imaging device 810 transmits analysis metadata for both "location information of the moving body region" and "location information of the face region." Furthermore, it is assumed that imaging devices 800 and 810 establish a session for transferring analysis metadata in addition to visible and infrared images in step 614 of Figure 6.

[0144] The imaging devices 800 and 810 begin transferring visible and infrared images (steps 1001 and 1002), and after transferring a predetermined number of frames (steps 1003 and 1004), they transmit the analysis metadata extracted by the motion region extraction unit 801 and the face region detection unit 802 (step 1005). Here, the analysis metadata may be sent at the time when a motion region or face region is detected.

[0145] Upon receiving the analysis metadata 1200, the control unit 214 of the controller device 200 checks whether the analysis metadata 1200 contains information about the motion region or information about the face region (step 1006). If it does not contain information about the motion region, it uses its own motion region extraction unit 209 to perform a motion region extraction process (step 1007).

[0146] On the other hand, if the data contains either motion region information or face region information, it is checked whether or not face region information is included (step 1008). If face region information is not included (i.e., only motion region information is included), the device uses the received motion region information and its own face region detection unit 210 to perform face region detection processing (step 1008).

[0147] On the other hand, if facial region information is included, the system extracts facial feature points using the received facial region information and its own facial feature point detection unit 211 (step 1010), and performs matching using the facial matching unit 212 (step 10100).

[0148] Figure 11 shows the image handling capabilities of imaging devices 800 and 810. Image 1100 is an example of a visible image captured by imaging devices 800 and 810. Image 1101 is obtained by removing the background from image 1100 and extracting only the moving object region. In this example, three regions (A), (B), and (C) (areas enclosed by dotted rectangles) have been extracted. Image 1102 is obtained by further extracting the face region from image 1101. In this example, two regions (a) and (b) (areas enclosed by solid rectangles) have been extracted.

[0149] Figure 12 shows an example of the configuration of the analysis metadata 1200 sent by the imaging devices 800 and 810 in step 1005.

[0150] The parsing metadata 1200 is broadly composed of a communication header 1201 and a payload 1210. The communication header 1201 is similar to, for example, an RTP header or an HTTP header.

[0151] The payload 1210 stores analysis metadata. For example, it consists of the frame number 1211 of the infrared image used for extracting motion regions or face regions, the frame number 1212 of the visible image, the maximum number of motion regions that imaging devices 800 and 810 can extract 1213, the actual number of motion regions extracted by the motion region extraction unit 801 1214 (n in this case), the coordinate information 1 to n (1215 to 1216) of the extracted motion regions, the maximum number of face regions that imaging device 810 can extract 1217, the actual number of face regions extracted by the face region detection unit 802 1218 (m ≤ n in this case), and the coordinate information 1 to m (1219 to 1220) of the extracted motion regions.

[0152] As described above, the imaging devices 800 and 810 of this embodiment can provide the controller device 200 with information regarding moving object regions and human regions, which have been accurately extracted using infrared images, in addition to visible images and infrared images, at the same time as the necessary image output.

[0153] On the other hand, the controller device 200 can eliminate conventional procedures by immediately utilizing the received information on the moving object area and the human area, thereby shortening the execution time of facial recognition compared to conventional methods. This is effective in reducing the processing load on the controller device 200 when managing many imaging devices with a single controller device 200.

[0154] In this embodiment, we have described an example in which the imaging devices 800 and 810 transmit at least one image (either a visible image or an infrared image) and analysis parameters to the controller device 200. However, in order to reduce the amount of data on the network, it is also possible to transmit only the analysis parameters and an image of only the portion indicated by the analysis parameters (motion region, face region).

[0155] Furthermore, the control unit 23 of the imaging devices 800 and 810 may, upon first detecting a moving object region in an image using the moving object region extraction unit 801, retain the frame number of the corresponding image, track the moving object from images subsequently captured until the object of that moving object region is no longer present, and add the frame number as supplementary information to the coordinate information in the analysis metadata 1200 shown in Figure 12. This allows the controller device 200 to easily determine the frame number containing the moving object region and calculate the time by referring to the analysis metadata 1200.

[0156] (Embodiment 3) Next, the configuration of the imaging system according to Embodiment 3 of the present invention will be described.

[0157] The imaging devices of Embodiments 1 and 2 described above used one set of lenses 11, an imaging sensor unit 12, and a signal processing unit 13 to capture visible and infrared images. The imaging device of this embodiment has a configuration in which two sets of lenses 11, imaging sensor units 12, and signal processing units 13 are arranged on the left and right sides, making it possible to capture stereo images (distance images) consisting of two images, one on the left and one on the right, for both visible light and infrared light.

[0158] Figure 13 shows an example of the configuration of the imaging system of this embodiment. This imaging system consists of one or more imaging devices 1300 and a controller device 1310.

[0159] As described above, the imaging device 1300 includes two sets of lenses 11, an imaging sensor unit 12, and a signal processing unit 13, and newly includes a correction parameter calculation unit 1301 and a distance calculation unit 1302. The two lenses 11(a) and (b) are arranged left and right so that their respective optical axes are parallel to each other. The other components are basically the same as those of the imaging devices 100, 800, and 810 of Embodiments 1 and 2.

[0160] The correction parameter calculation unit 1301 sets parameters such as clipping levels and signal level correction values ​​(for example, correction values ​​that are added to, subtracted from, multiplied from, or divided by signals such as visible image signals, infrared image signals, infrared signals, and each color signal) to approximate the signal intensity (signal level) of each visible image output from the two signal processing units 13(a) and (b), so that the signal levels of the two visible image signals (two infrared image signals) become similar. The correction amount in the image signal correction processing unit 203 is set by looking at the output from the two signal processing units 13(a) and (b) and adjusting it accordingly to match the levels of the image signals. The process of matching the levels of the left and right image signals can be performed for both infrared image signals and visible image signals.

[0161] In other words, the correction parameter calculation unit 1301 determines the correction amount based on the signal levels of the image signals output from the two signal processing units 13(a) and (b), so that the signal levels of the image signals output from the two signal processing units 13(a) and (b) are similar. This suppresses errors that may occur in measuring distance, such as when different parts of a subject are recognized as the same part (corresponding point) due to differences in brightness levels between two image data.

[0162] The distance calculation unit 1302 calculates the distance to an object using two visible image signals or infrared image signals input from two signal processing units 13(a) and (b), respectively. In this process, it determines the same subject (corresponding points) from the two images and detects the parallax, which is the difference in the position of these identical subjects in the images, to determine the distance in the same way as in the conventional method. That is, the corresponding points for measuring the difference are determined by image recognition, and the distance is calculated based on the parallax, which is the difference in the position of the corresponding points in the images. Then, a stereo image (distance image) is generated based on the distance information corresponding to each pixel and output to the signal output control unit 14.

[0163] The signal output control unit 14 can provide the controller device 1310 with a stereo image (distance image) generated by the distance calculation unit 1302, in addition to the two visible and infrared images captured by the left and right cameras.

[0164] From the above, the imaging device 1300 can simultaneously acquire visible light and infrared images of a subject, and calculate the distance from both images. In this case, since the positions of the visible and infrared images are aligned, it is possible to prevent the measured distance from changing between the two images.

[0165] Here, the distance calculation unit 1302 may calculate the distances using two visible images and two infrared images, then generate two stereo images (distance images) and output them as is. Alternatively, it may compare the two generated stereo images and output one of the distance images if the difference in their distance information is within a predetermined threshold range, or output both distance images if it exceeds the threshold, or output a distance image that has been set to be output in advance (for example, prioritizing the distance image calculated from the infrared images, or the distance image showing a closer distance value), or output the area exceeding the threshold separately as analysis metadata. Figure 22 shows an example of the processing overview of the distance calculation unit 1302.

[0166] Furthermore, the control unit 23 of the imaging device 1300, in accordance with instructions from the controller device 200, uses the signal output control unit 14 to control which data is output via the IF unit 16 from the visible image, infrared image output from the two signal processing units 13(a) and (b), and the stereo image (distance image) output from the distance calculation unit 1302. For example, if the imaging device 1300 is installed in a location where privacy protection is necessary (e.g., a toilet or changing room), only the stereo image will be output, and if it is installed in a location where high security is required, all images will be output.

[0167] On the other hand, the controller device 1310 is equipped with a different motion region extraction unit 1311, face region detection unit 1312, face feature point detection unit 1313, face matching unit 1314, and 3D face DB 1315, instead of the motion region extraction unit 209, face region detection unit 210, face feature point detection unit 211, face matching unit 212, and face DB 213 of the controller device 200 in Embodiments 1 and 2, in order to perform analysis and authentication processing using stereo images (distance images) in addition to visible images and infrared images, or using only stereo images. This allows, for example, in face recognition, to acquire 3D data regarding the contours of the face and use it to perform face region detection and face feature point detection accurately and easily.

[0168] Furthermore, the controller device 1310 acquires a stereo image (distance image) from the imaging device 1300, and by referring to the distance of the motion region extracted by the motion region extraction unit 209, it can determine whether to perform face recognition if it is within a predetermined distance, and not to perform it otherwise.

[0169] Figure 23 shows an example in which the controller device 1310 displays the images of imaging devices 1300(a),(b), and(c) installed in different locations. Using visible or infrared images and depth images received from imaging device 1300(a) installed in the entrance of an office or building, the controller device 1310 performs 3D-based facial recognition and displays the results. This makes it easier to identify visitors and suspicious individuals, helping to alleviate congestion at reception. Furthermore, using depth images received from imaging device 1300(b) installed in stores in public facilities or commercial facilities, the controller device 1310 displays the number of people looking at the shelves and information that does not identify individuals (e.g., gender, height, face orientation, body orientation, posture, etc.). This helps to determine the purchasing demographics of shoppers and their level of interest based on their gaze and posture, which is useful for strengthening marketing and sales capabilities related to products and displays. Using visible images, infrared images, and depth images received from the imaging device 1300(c) installed outdoors in amusement parks, parks, etc., the controller device 1310 extracts people from the images, performs 3D-based facial recognition, and if it is confirmed that the person is a pre-registered person, it displays that part in the depth image, and displays only the people that cannot be confirmed in the visible image or infrared image. Alternatively, it displays information that does not allow for the identification of a person (for example, gender, height, face direction, whether there are children, posture, etc.). This helps to ensure the safety of visitors and to detect suspicious individuals early.

[0170] Next, Figure 14 shows another configuration example of the imaging system of this embodiment. This imaging system consists of one or more imaging devices 1400 and a controller device 1410. Although not shown, the imaging device 100 of Embodiment 1, the imaging devices 800 and 810 of Embodiment 2, and the above-mentioned imaging device 1300 may coexist on the network 303. The controller device 1410 is capable of managing any of the imaging devices.

[0171] The imaging device 1400 is equipped with two motion region extraction units 1401(a) and (b) on top of the imaging device 1300. These motion region extraction units 1401 may be the same as the motion region extraction unit 1311 of the controller device 1310. The other components are configured the same as those of the imaging device 1300.

[0172] The motion region extraction unit 1401(a)(b) is the part that extracts motion regions from each image using infrared images output from the two signal processing units 13(a)(b). The extracted motion region information can be output to the signal output control unit 14 or the control unit 23 and provided to the controller device 1410, as in the second embodiment described above. Furthermore, the motion region extraction unit 1401(a)(b) can extract motion regions using the stereo image (distance image) output from the distance calculation unit 1302 and extract motion regions with greater accuracy by comparing the results with the method described above. Alternatively, motion regions may be extracted first using the stereo image (distance image), and then the portion of the extracted motion region may be examined in more detail using the infrared image.

[0173] Here, the control unit 23 can refer to the information of the two motion regions output from the motion region extraction unit 1401(a)(b), compare the number and location of the extracted regions, and transmit the comparison result as analysis metadata. The controller device 1410 can use the analysis metadata to select whether to use the left or right visible image or infrared image for face recognition. For example, if the number of motion regions extracted by the motion region extraction unit 1401(a) (or motion region extraction unit 1401(b)) is greater than the number of motion regions extracted by the motion region extraction unit 1401(a) (or motion region extraction unit 1401(b)), the control unit 23 of the imaging device 1400 will send the information of the motion regions extracted by the motion region extraction unit 1401(a) (or motion region extraction unit 1401(b)) and the visible image or infrared image output from the signal processing unit 13(a) (or signal processing unit 13(b)).

[0174] On the other hand, the controller device 1410 is equipped with the face region detection unit 210, face feature points 211, face matching unit 212, face DB 213 of the controller device 200 described in Embodiment 1, and the comprehensive determination unit 1411, on top of the controller device 1310.

[0175] As a result, the imaging device 1400 can simultaneously perform face recognition processing using the face region detection unit 210, face feature points 211, face matching unit 212, and face DB 213 described in Embodiment 1 (using visible images and infrared images), and face recognition processing using the aforementioned face region detection unit 1312, face feature point detection unit 1313, face matching unit 1314, and 3D face DB 1315 (using visible images, infrared images, and stereo images). The comprehensive determination unit 1411 is the part that makes the final determination of the person authentication result based on the results of performing both face recognition processes. By performing two different face recognition methods, it is possible to perform face recognition with higher accuracy.

[0176] Similarly, Figure 15 shows another configuration example of the imaging system of this embodiment. This imaging system consists of one or more imaging devices 1500 and a controller device 1510. Although not shown, the imaging device 100 of Embodiment 1, the imaging devices 800 and 810 of Embodiment 2, and the above-mentioned imaging devices 1300 and 1400 may be mixed on the network 303. The controller device 1510 is capable of managing any of the imaging devices.

[0177] The imaging device 1500 is equipped with two face region detection units 1502(a) and (b) on top of the imaging device 1400. These face region detection units 1502 may be the same as the face region detection unit 1312 of the controller device 1310. The other components are configured the same as those of the imaging device 1400.

[0178] The face region detection unit 1501(a)(b) is the part that extracts the human face region using the information of the moving regions output from the two moving region extraction units 1401(a)(b). This extracted face region information can be output to the signal output control unit 14 or the control unit 23 and provided to the controller device 1510, as in the second embodiment described above.

[0179] Here, the control unit 23 can refer to the information of the two face regions output from the face region detection unit 1501(a)(b), compare the number, position, and orientation of the extracted faces, and transmit the comparison result as analysis metadata. The controller device 1510 can use the analysis metadata to select an image more suitable for face recognition, thereby performing face recognition with higher accuracy.

[0180] On the other hand, the controller device 1510 is equipped with an authentication method selection unit 1511, an iris detection unit 1512, an iris matching unit 1513, and an iris database 1514, in addition to the controller device 200 or controller device 1410.

[0181] The authentication method selection unit 1511 is responsible for selecting whether to perform facial recognition or iris recognition using visible images, infrared images, stereo images (distance images), and analysis parameter information received from the imaging device 1500. For example, if the object enters a predetermined distance range, iris recognition is performed, and otherwise, facial recognition is performed. Alternatively, facial recognition is performed normally, and iris recognition is also performed if the conditions for iris recognition are met.

[0182] The iris detection unit 1512 uses the infrared image received from the imaging device 1500 and analysis parameters including the face region extracted from the image to detect the position of the iris of a person's eye. Furthermore, it detects the boundary between the iris and the white of the eye, as well as the boundary between the iris and the surrounding area, to identify the iris area and generate a pupil code. Any known method may be applied for these steps.

[0183] The iris matching unit 1513 performs matching using the iris database 1514, similar to facial recognition, based on the information detected by the iris detection unit 1512.

[0184] This allows the controller device 1510 to select the optimal biometric authentication method using visible images, infrared images, stereo images (distance images), and analysis parameter information received from the imaging device 1500, enabling more accurate user authentication.

[0185] Figure 24 shows an example in which the controller device 1410 and controller device 1510 process the images captured by the imaging device 1500 installed in airports, building entrances, etc. In this example, the controller device 1410 acquires visible images and face region and distance information as analysis parameters from the imaging device 1500, performing 2D facial recognition which has a relatively low image processing load for face regions that are far away, and 3D facial recognition which has a high image processing load for face regions that are close by. The controller device 1510 also acquires visible images, infrared images, and face region and distance information as analysis parameters from the imaging device 1500, performing facial recognition using visible images for face regions that are far away, and iris recognition using infrared images for face regions that are close by. Furthermore, to alleviate congestion caused by waiting in line, it is also possible to determine people who are close to a predetermined position in the captured image and perform facial recognition in that order.

[0186] Figure 16 shows another configuration example of the imaging system of this embodiment. This example shows the imaging system mounted on a mobile device such as a smartphone or tablet.

[0187] Figures 17 and 18 show examples of the configuration of functional information and analysis parameters for the imaging devices 1400 and 1500 used in this embodiment.

[0188] The stereo image (distance image) generated by the imaging device can be provided to the controller device using the same transfer method as visible images and infrared images, as shown in Figure 17.

[0189] Alternatively, as shown in Figure 18, the information can be provided by adding it to the position information of the moving body region or the face region. In that case, only the distance information related to the coordinate regions corresponding to the moving body region or face region is extracted and added.

[0190] Figure 20 shows an example configuration in which distance information is sent as part of the analysis parameters. Distance information is stored in the payload 1210. For example, distance information for extracted moving body regions (number n) is stored immediately after the coordinate information of the moving body regions (2001, 2002), and distance information for face regions (number m) is stored immediately after the coordinate information of the face regions (2003, 2004). In addition to this configuration, a configuration in which coordinate information and distance information are stored alternately is also acceptable.

[0191] Figure 19 shows another configuration example of the imaging system of this embodiment. The imaging device 1900 of this imaging system is equipped with one motion region extraction unit 1901 and is configured to extract the motion region using either the left or right signal processing unit 13(a) or (b) infrared image. Alternatively, it is configured to extract the motion region using either the left or right infrared image and the stereo image output from the distance calculation unit 1302. [Explanation of symbols]

[0192] 2...Sensor body, 3...Color filter, 5...DBPF (optical filter), 11...Lens (optical system), 12...Imaging sensor unit, 13...Signal processing unit, 14...Signal output control unit, 15...Communication control unit, 16...IF unit, 23...Control unit, 100, 800, 810, 1300, 1400, 1500...Imaging device, 200, 1310, 1410, 1510...Controller device, 801, 1401...Motion area extraction unit, 802, 1501...Face area detection unit, 1301...Correction parameter calculation unit, 1302...Distance calculation unit, 1600...Mobile terminal.

Claims

[Claim 1] A first filter unit and a second filter unit having the characteristic of transmitting at least visible light wavelengths and infrared wavelengths, and filtering light based on the said characteristic, A first image sensor that receives light from a subject filtered by the first filter unit and outputs a signal, and a second image sensor that receives light from the subject filtered by the second filter unit and outputs a signal, A first signal processing unit that processes the signal from the first image sensor and outputs a visible light signal and an infrared signal, and a second signal processing unit that processes the signal from the second image sensor and outputs a visible light signal and an infrared signal, Control unit and Equipped with, The control unit controls the transmission to the outside of a first data set based on one of the visible light signal output from the first signal processing unit and the infrared signal acquired at the same timing as the visible light signal, and a second data set based on the visible light signal output from the second signal processing unit and the infrared signal acquired at the same timing as the visible light signal, which corresponds to the first data, by adding or multiplexing a third data set related to the distance calculated based on the first data and the second data. Imaging device.

Images