Imaging apparatus

Abstract

To provide a technique for imaging both a visual image and an infrared image, and imaging a high-quality image by improving color reproducibility in visible light imaging.SOLUTION: An imaging system is configured to enable stereo imaging for both a visible image and an infrared image, and improve color reproducibility in visible light imaging. The imaging system includes: two imaging sensors 12; and two DBPFs which are optical filters having a visible light band and a second wavelength band with transparency and arranged for the two imaging sensors. The imaging system has at least four types of filter units which are different in spectral transmission characteristics in accordance with a wavelength in the visible light band and similar in transmittance in the second wavelength band, and two color filters arranged for the two imaging sensors, respectively. The imaging system measures a distance to an object based on two visible or infrared image signals.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present invention relates to an imaging device and an imaging system. [Background technology]

[0002] In recent years, development has progressed in the combination of image recognition technology and biometric authentication to develop surveillance cameras that use facial recognition to detect criminals, shoplifters, terrorists, etc., as well as in-vehicle cameras for use in self-driving cars.

[0003] For example, a system is known for detecting the distance using a stereo camera in a surveillance camera to detect the intrusion of a suspicious person (see Patent Document 1). Patent Document 1 describes an "image recognition device for distance measurement using stereo images" as a method for measuring the shape and distance of an object, and a technology called a stereo method for finding distance from stereo images is known. In this stereo method, first, two images, left and right, called stereo images, are input, and corresponding points of the left and right images (where an object at a certain position in the left image is located in the right image) are found by calculating the feature amount in the image. Details of how to find the corresponding points are described in, for example, Patent Document 2 as an "image matching method." Then, when the corresponding points of the two left and right images are found, the distance to the object surface can be calculated using the principle of triangulation, so that the distance to the object and the shape of the object surface can be known.

[0004] Furthermore, Patent Document 1 proposes a moving object recognition device that can detect a moving object with high accuracy and high speed and measure its shape and distance by utilizing a known correspondence relationship between stereo images.

[0005] Surveillance cameras and cameras for autonomous driving require shooting regardless of location or time, such as outdoors and indoors, day and night, but in some situations, sufficient lighting may not be available. In this case, infrared shooting can be performed using infrared lighting that is invisible to humans. In addition, for cameras for autonomous driving, infrared shooting can be performed using infrared light as lighting to illuminate distant objects, taking into account the effect of headlights on oncoming vehicles at night. In any case, visible light shooting can be performed without lighting during the day when the amount of visible light is likely to be sufficient, and infrared shooting can be performed using infrared lighting that is difficult for the human eye when lighting is required at night.

[0006] Considering such circumstances, it is preferable for surveillance cameras and cameras for autonomous driving to be able to capture images using both visible light and infrared light.

[0007] In imaging devices such as security cameras that take pictures continuously day and night, infrared light is detected and taken at night. Photodiodes (light receiving elements), which are the light receiving parts of imaging sensors such as CCD sensors and CMOS sensors, can receive light up to the near-infrared wavelength band of about 1300 nm, so imaging devices using these imaging sensors can, in principle, take pictures up to the infrared band.

[0008] Since the wavelength band of light with high human visibility is 400nm to 700nm, when an image sensor detects near-infrared light, the image appears reddish to the human eye. For this reason, when taking pictures during the day or in bright indoor locations, it is desirable to provide an infrared cut filter that blocks infrared light in front of the image sensor and remove light with a wavelength of 700nm or more in order to match the sensitivity of the image sensor to the human visibility. On the other hand, when taking pictures at night or in dark locations, it is necessary to take pictures without providing an infrared cut filter.

[0009] As such an imaging device, imaging devices in which an infrared cut filter is manually attached / detached and imaging devices in which an infrared cut filter is automatically inserted / removed are conventionally known. Furthermore, imaging devices that do not require the above-mentioned insertion / removal of an infrared cut filter have been disclosed. For example, an optical filter has been proposed that has a transmission characteristic in the visible light band, a blocking characteristic in a first wavelength band adjacent to the long wavelength side of the visible light band, and a transmission characteristic in a second wavelength band that is a part of the first wavelength band (see Patent Document 3). This filter allows light to pass through both the visible light band and the second wavelength band on the long wavelength side of the visible light band, i.e., on the infrared side and away from the visible light band.

[0010] Hereinafter, an optical filter that transmits light in the visible light band and a second wavelength band on the infrared side as described above and blocks light in other wavelength bands will be referred to as a DBPF (double band pass filter).

[0011] Furthermore, in recent years, various biometric authentication technologies have been developed, including those based on fingerprints, faces, irises, veins, signatures, voiceprints, and gait. The main biometric authentication methods used together with the image recognition technology using images captured by surveillance cameras and autonomous driving cameras are face recognition and iris recognition. [Prior art documents] [Patent documents]

[0012] [Patent Document 1] Japanese Patent Application Publication No. 3-81878 [Patent Document 2] Japanese Patent Application Publication No. 62-107386 [Patent Document 3] Patent No. 5009395 Summary of the Invention [Problem to be solved by the invention]

[0013] In the DBPF of Patent Document 3, light in a second wavelength band (a relatively narrow wavelength band included in the infrared wavelength band) included in the infrared (near-infrared) wavelength band is not always blocked and is allowed to pass through. In other words, unlike the case where an infrared cut filter that cuts wavelengths longer than the visible light band is used, the infrared light that has passed through the second wavelength band is more or less affected in photographing in the visible light band.

[0014] Color filters are used in image sensors that capture color images in the visible light band. The color filters have red, green, and blue regions (filter sections) arranged in a specific pattern corresponding to each pixel of the image sensor, and basically have a peak in light transmittance in the wavelength band of each color and block the transmission of light in the wavelength bands of other colors.

[0015] However, on the longer wavelength side than the visible light band, the light transmittance differs depending on the region and wavelength of each color, but basically the light is transmitted. Therefore, if infrared light is transmitted in the second wavelength band on the infrared side as in the above-mentioned DBPF, this infrared light passes through the color filter and reaches the photodiode (light receiving element) of the image sensor, increasing the number of electrons generated by the photoelectric effect in the photodiode.

[0016] In addition, when both color photography with visible light and photography with infrared light illumination are performed, for example, a color filter in which red, green, and blue color regions are arranged in a predetermined pattern is provided with an infrared light region (infrared region) having a peak in light transmittance in the above-mentioned second wavelength band. That is, the arrangement (pattern) of the color filter is made up of four regions: red R, green G, blue B, and infrared IR. In this case, the infrared light region blocks light in the visible light band and mainly transmits light in the second wavelength band, so it is possible to remove infrared light components from the image signals of each color of red, green, and blue by using an infrared light image signal output from an image sensor that receives light that has passed through the infrared light region of the color filter. However, even with such signal processing, it is difficult to achieve color reproduction that is approximately the same as that in color photography when an infrared cut filter is used. In addition, when calculating distance by stereoscopic conversion, if there is a difference in the signal levels between the left and right, it becomes a factor that causes errors in parallax calculation.

[0017] Furthermore, when face recognition is used using images captured by a camera, there are two main types: one in which a user presents their face to a specified camera in an entrance / exit management system for an office or building, boarding procedures at an airport, immigration control, etc., and the other in which recognition is performed for an unspecified number of users who have unconsciously been photographed by multiple cameras in places such as public facilities, airports, and transportation facilities for the purpose of tracking criminals, preventing terrorism, and early detection of suspicious persons, etc. In the former type, since the photographing conditions are limited, accurate recognition is possible even with recent technology, but in the latter type, the photographing conditions such as lighting conditions, face direction, angle, etc. vary greatly due to environmental changes, so the recognition rate is greatly affected.

[0018] In order to use surveillance cameras to detect suspicious people, it would be necessary to use both visible and infrared light and make it possible to take continuous images 24 hours a day, regardless of the location or time of shooting, and if the images captured could be clear and of high resolution with as little noise as possible, it would be possible to dramatically improve the ability to detect suspicious people early and analyze the situation when a crime occurs.

[0019] In addition, by configuring two cameras to enable distance measurement using the stereo method, even in cameras for autonomous driving, when using both infrared and visible light photography as described above, clear images with little noise can be obtained, thereby improving the accuracy of image recognition.

[0020] For these reasons, it is desirable to have a system that can capture both infrared and visible images simultaneously, with image quality in terms of noise, resolution, color reproducibility, etc., at the same level or higher as that of normal visible images that do not include infrared images, and furthermore, to enable stereoscopic shooting using a two-camera configuration.

[0021] The present invention provides a technology that is capable of capturing both visible and infrared images and that improves color reproducibility during visible light capture to capture high-quality images. [Means for solving the problem]

[0022] The imaging device or imaging system of the present invention is characterized in that it comprises an imaging element, a filter unit having a characteristic of transmitting at least visible light wavelength region and infrared wavelength region and filtering a signal from the imaging element based on said characteristic, a signal processing unit that processes the signal filtered by the filter unit and outputs a visible light signal and an infrared signal, a moving object region extraction unit that generates information about a moving object in an image captured by the imaging element from the infrared signal output from the signal processing unit, and a signal output control unit that transmits to the outside first data including at least one of the visible light signal and the infrared signal output from the signal processing, and second data based on the information about the moving object generated by the moving object region extraction unit.

[0023] Furthermore, the imaging device or imaging system according to the present invention is further characterized in that it comprises two imaging elements, two filter units each having a characteristic of transmitting at least visible light wavelength region and infrared wavelength region and filtering signals from the imaging elements based on said characteristic, two signal processing units each processing the signals filtered by the filter units to output a visible light signal and an infrared signal, a distance calculation unit each calculating a distance to a subject appearing in a visible image based on a visible image signal and / or an infrared image based on an infrared image signal using the two visible image signals and / or two infrared image signals output from the signal processing units, a moving object region extraction unit generating information about a moving object in an image captured by the imaging elements from the infrared signal output from the signal processing unit, and a signal output control unit each transmitting to the outside first data including at least one of the visible light signal or the infrared signal output from the signal processing, second data based on the information about the moving object generated by the moving object region extraction unit, and third data based on the distance information calculated by the distance calculation unit. Effect of the Invention

[0024] According to the present invention, it is possible to obtain a high-quality image. More specifically, according to one aspect of the present invention, for example, a camera consisting of one image sensor and one optical film can simultaneously capture both high-quality infrared images and visible images, so that visibility can be improved even in a change in the environment such as insufficient lighting or at night. According to another aspect of the present invention, the distance to an object can be measured more accurately using both infrared images and visible images, and the information can be provided to an external system together with a visible image or an infrared image. According to another aspect of the present invention, by using an infrared image instead of a visible image, a moving object area in an image can be extracted more quickly, and the information can be provided to an external system together with a visible image or an infrared image. [Brief description of the drawings]

[0025] [Figure 1] 1 is a schematic diagram showing an imaging system according to a first embodiment of the present invention. [Diagram 2]1 is a schematic diagram showing a configuration of an image pickup sensor unit of an image pickup system according to a first embodiment of the present invention. [Diagram 3] 4 is a graph showing the DBPF and transmittance spectra of color filters of the image sensor of the imaging system according to the first embodiment of the present invention. [Figure 4] 2 is a schematic diagram showing an example of a configuration of a color filter of the imaging system according to the first embodiment of the present invention. FIG. [Diagram 5] 2 is a block diagram illustrating a signal processing unit of the imaging system according to the first embodiment of the present invention. FIG. [Figure 6] FIG. 4 is a flowchart illustrating a communication flow between the camera and the controller of the imaging system according to the first embodiment of the present invention. [Figure 7] 1 is a diagram showing one configuration of various types of image information handled by the imaging system according to the first embodiment of the present invention. [Figure 8] FIG. 11 is a schematic diagram showing an imaging system according to a second embodiment of the present invention. [Figure 9] 11 is a diagram showing one configuration of various pieces of analysis metadata information handled by the imaging system according to the second embodiment of the present invention. FIG. [Figure 10] FIG. 11 is a flowchart illustrating a communication flow between a camera and a controller of an imaging system according to a second embodiment of the present invention. [Figure 11] FIG. 11 is a diagram showing an example of an image screen of an image handled by the imaging system according to the second embodiment of the present invention. [Figure 12] 11 is a diagram showing an example of the configuration of analysis metadata information handled by the imaging system according to the second embodiment of the present invention. FIG. [Figure 13] FIG. 11 is a schematic diagram showing an imaging system according to a third embodiment of the present invention. [Figure 14] FIG. 11 is a schematic diagram showing another configuration example of the imaging system according to the third embodiment of the present invention. [Figure 15] FIG. 11 is a schematic diagram showing another configuration example of the imaging system according to the third embodiment of the present invention. [Figure 16] FIG. 11 is a schematic diagram showing another configuration example of the imaging system according to the third embodiment of the present invention. [Figure 17] FIG. 11 is a diagram showing one configuration of various types of image information handled by the imaging system according to the third embodiment of the present invention. [Figure 18] FIG. 11 is a diagram showing one configuration of various types of image information handled by the imaging system according to the third embodiment of the present invention. [Figure 19] FIG. 11 is a diagram showing one configuration of analysis metadata information handled by the imaging system according to the third embodiment of the present invention. [Figure 20] 13 is a diagram showing an example of the configuration of analysis metadata information handled by the imaging system according to the third embodiment of the present invention. FIG. [Figure 21] 4 is a graph showing the DBPF and transmittance spectra of color filters of the image sensor of the imaging system according to the first embodiment of the present invention. [Figure 22] FIG. 11 is a diagram showing a processing flow of an imaging device of an imaging system according to a third embodiment of the present invention. [Figure 23] 13 is an example of operation of the imaging system according to the third embodiment of the present invention. [Figure 24] 13 is an example of operation of the imaging system according to the third embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] (Embodiment 1) Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0027] 1 shows an example of the configuration of an imaging system according to the first embodiment of the present invention. The imaging system is broadly composed of one or more imaging devices 100 ((a) to (n)) and one or more controller devices 200. The imaging devices 100 and the controller device 200 are connected via a network 303. In this embodiment, the network 303 will be described on the assumption that it is a wired LAN (Local Area Network), but it may also be a general-purpose network such as a wireless LAN (WiFi), USB (Universal Serial Bus), or IEEE1394.

[0028] In the network 303, standard IP (Internet Protocol) is used as the network protocol, and TCP (Transmission Control Protocol) and UDP (User Datagram Protocol) are used as upper transport protocols. For the transfer of images captured by the imaging device 100, higher-level application protocols, such as RTP (Real-time Transport Protocol) / RTCP (RTP Control Protocol) and HTTP (Hyper Text Transfer Protocol), are used, and RTSP (Real-Time Streaming Protocol) is used for transfer control. Either IPv4 or IPv6 may be used for IP. In addition, communication between higher-level applications is also possible using Web services using the above-mentioned HTTP and RTP techniques. Although not shown, Internet connection is also possible via a hub or router.

[0029] The controller device 200 is capable of controlling a plurality of image capture devices 100, and is also capable of exchanging information with other controller devices 200.

[0030] The imaging system of this embodiment can be used in applications and services such as surveillance and entrance / exit management.

[0031] The imaging device 100, which is a feature of this embodiment, is composed of a lens 11, an imaging sensor unit 12, a signal processing unit 13, a signal output control unit 14, a communication control unit 15, an IF unit 16, an abnormality detection unit 17, an illuminance monitoring unit 18, a GPS 19, a clock 20, a memory 21, a maintenance IF unit 22, a control unit 23, and an infrared LED 24.

[0032] The lens 11 is an optical lens for photographing that focuses visible light 301 and infrared light (invisible light) 302 from a subject at a predetermined focal distance on the image sensor unit 12, and is configured to include a plurality of lenses.

[0033] The image sensor unit 12 is a part that disperses the visible light and infrared light imaged by the lens 11 using various filters and performs photoelectric conversion to output a plurality of pixel signals corresponding to predetermined wavelength components.

[0034] The signal processing unit 13 processes the output signal output from the imaging sensor unit 12, and performs image processing such as interior processing, image processing for removing the influence of infrared light that has passed through the second wavelength band during color shooting, gamma correction, white balance, RGB matrix correction, etc. on the image signal, and outputs an output signal of a visible image and an output signal of an infrared image.

[0035] The signal output control unit 14 transmits the visible image signal and infrared image signal "of the object photographed at the same time" output from the signal processing unit 13 to a specified controller device 200 connected to the network via the IF unit 16 in accordance with instructions from the communication control unit 15 or the control unit 23.

[0036] The communication control unit 15 is a part that controls the image signal output from the signal output control unit 14 via the IF unit 16, and transmits and receives control signals to and from the controller device 200 via the IF unit. It also executes the network protocol, application protocol, Web service, and the like.

[0037] The IF unit 16 is a communication IF portion that connects the image capturing apparatus 100 to a network 303 .

[0038] The abnormality detection unit 17 is a part that constantly or periodically monitors whether an abnormality has occurred in the hardware and software of the imaging device 100, and detects abnormalities. For example, when the imaging device 100 is removed from a predetermined installation location, when it is no longer possible to capture an image, when it is no longer possible to communicate over the network, or when unauthorized access has occurred.

[0039] The illuminance monitoring unit 18 is a part that constantly or periodically monitors the brightness of the shooting range of the imaging device 100 by using an illuminance sensor or the like. When it detects that the illuminance is insufficient, it notifies the control unit 23 of that fact and irradiates the infrared LED 24.

[0040] The GPS 19 is a section that acquires the current position of the imaging device 100 itself from position information received from a satellite. The acquired position information can also be notified to the controller device 200 via the IF section 16.

[0041] The clock 20 is a part that manages the current time information and sets / cancels the timer. The time information is automatically adjusted using general-purpose technology such as NTP (Network Time Protocol) or standard radio waves.

[0042] The memory 21 is a storage device (ROM (Read-Only Memory) area, FROM (Flash ROM) area) that stores programs, various setting information, and property information, and a storage device (RAM (Random Access Memory) area) that loads and temporarily stores these programs and data, and stores work data. Here, the recording device may use an external memory (USB memory or NAS (Network-Attached Storage)) or a portable medium (micro flash, SD card, magnetic tape, etc.) in addition to the built-in memory.

[0043] The maintenance IF unit 22 is an IF section with which a maintenance person for the imaging device 100 communicates to update a program or diagnose a failure when the program updates or the failure occurs. In addition, when the abnormality detection unit 17 detects an abnormality, the content of the detected abnormality can be automatically notified to a remote maintenance site.

[0044] The control unit 23 is a part that comprehensively controls the operations of each of the above-mentioned components.

[0045] On the other hand, the controller device 200 is composed of a user IF unit 201, a display unit 202, a clock 203, a memory 204, a recording and playback unit 205, a communication control unit 206, an IF unit 207, a camera management unit 208, a moving object area extraction unit 209, a face area detection unit 210, a face feature point detection unit 211, a face matching unit 212, a face DB 213, and a control unit 214.

[0046] The user IF unit 201 is a section where the user operates the controller device 200 using a remote control, a touch panel, a keyboard, a mouse, buttons, and the like.

[0047] The display unit 202 is a part that displays, on an external or built-in monitor, the operation screen of the controller device 200, visible images and infrared images received via the network 303, the results of face authentication, warning screens, and the like.

[0048] The clock 203 is a part that manages the current time information and sets / cancels the timer. The time information is automatically adjusted using general-purpose technology such as NTP or standard radio waves.

[0049] The memory 204 is a storage device (ROM area, FROM area) that stores programs, various setting information, and property information, and a storage device (RAM area) that loads and temporarily stores these programs and data, and stores work data. Here, the recording device may use an external memory (USB memory or NAS) or a portable medium (micro flash, SD card, DVD, Blu-ray (registered trademark) Disc, magnetic tape, etc.) in addition to the built-in memory.

[0050] The recording and reproducing unit 205 is a part that records and reproduces visible images and infrared images received via the network 303 and the IF unit 207, as well as metadata associated with these images, in the memory 204. If necessary, it performs encryption / decryption and compression / expansion of the data to be recorded.

[0051] The communication control unit 206 is a unit that transmits and receives control signals between the network 303 and the imaging device 100 via the IF unit 207. It also executes the network protocol, application protocol, Web services, and the like.

[0052] The IF unit 207 is a communication IF portion that connects the controller device 200 and the network 303 .

[0053] The camera management unit 208 is a part that manages one or more imaging devices 100 managed by the control device 200 via the network 303. This part creates, holds, updates, and deletes information about the imaging devices 100 under management (e.g., IP addresses, installation locations, manufacturer names, model names, installation dates / operating hours, functional specifications, maintenance contact information, etc.).

[0054] The moving object region extraction unit 209 is a part that extracts moving objects such as people, animals, and objects present in a visible image or an infrared image received via the IF unit 207 or recorded by the recording and playback unit 205, and acquires their position information. Methods for extracting moving objects in an image include creating a difference image (e.g., a difference image between the first and second images, or a difference image between the second and third images) from multiple consecutive images (e.g., three images) and extracting the moving object by comparing them, or extracting the moving object by generating a background image from the captured image and using a background subtraction method.

[0055] The face area detection unit 210 detects an area containing a human face directly from a visible image or an infrared image received via the IF unit 207 or recorded by the recording and playback unit 205, or from a moving object area extracted by the moving object area extraction unit 209. Detection methods include a high-speed face detection algorithm using Viola & Johns' integral image and a cascade type classifier.

[0056] The face feature point detection unit 211 is a part that detects feature points such as the eyes, nose, and mouth corners within the face area detected by the face area detection unit 210. This makes it possible to correct the image position so that the face features can be accurately extracted.

[0057] The face matching unit 212 is a part that selects the most suitable feature for identifying an individual from among the feature points detected by the face feature point detection unit 211, and performs matching using a face DB 213. Here, as features for identifying a face, a method that uses the entire shading information in the face region (for example, an eigen method that applies principal component analysis), a method that uses the interval and directional components of local shading changes in the face region as feature amounts (for example, elastic bunch graph matching), a method that combines both of these methods, etc. Also, as a matching method, a nearest neighbor method, linear discriminant analysis, etc. can be applied.

[0058] The face DB 213 is a part that stores data in which face images are registered in advance in a built-in or external storage medium for matching by the face matching unit 212. Images artificially generated by changing illumination or face direction from these registered images can also be registered. For example, in an entrance / exit management system, face images of users who are permitted to enter a specific area or users who are employees are registered. It is also possible to additionally register images that have been confirmed to be the person in question as a result of face authentication at a specific location. Here, this face DB 213 may be an external DB accessible via the network 303, rather than the controller device 200. For example, in a surveillance camera system at an airport or the like, a face DB of suspects, terrorists, etc. provided by the police or legal institutions is used. Also, the DB may be shared between multiple controller devices.

[0059] The control unit 214 is a part that comprehensively controls the operation of each of the above-mentioned components. Furthermore, if the face matching unit 212 finds that the user is not a pre-registered user (such as a suspicious person) or that the face matches a suspect, it is possible to automatically play back a report based on a predetermined format, notify the contact point of the administrator or the police, and send the report.

[0060] FIG. 2 shows an example of the configuration of the image sensor unit 12 of the camera unit 100. As shown in FIG.

[0061] The image sensor unit 12 is composed of a sensor body 2 , a color filter 3 , a cover glass 4 , and a DBPF 5 .

[0062] The sensor body 2 is a CCD (Charge Coupled Device) image sensor, in which a photodiode is arranged as a light receiving element for each pixel. A CMOS (Complementary Metal Oxide Semiconductor) image sensor may be used instead of the CCD image sensor.

[0063] The color filter 3 is provided on the sensor body 2, and is a portion in which red (R), green (G), blue (B), and infrared (IR) regions are arranged in a predetermined array for each pixel of the sensor body 2. Fig. 4 shows variations of the color filters used in this embodiment.

[0064] The cover glass 4 covers and protects the sensor body 2 and the color filter 3.

[0065] The DBPF5 is an optical filter formed on the cover glass 4. The DBPF5 is an optical filter that has a transmission characteristic in the visible light band, a blocking characteristic in a first wavelength band adjacent to the long wavelength side of the visible light band, and a transmission characteristic in a second wavelength band that is a part of the first wavelength band. The location of the DBPF5 is not limited, and it may be provided on the lens 11, for example.

[0066] 3 shows the transmittance spectrum of each of the R, G, B, and IR filters of the color filter 3, with the vertical axis representing transmittance and the horizontal axis representing wavelength. The wavelength range in the graph includes the visible light band and part of the near-infrared band, and shows a wavelength range of, for example, 300 nm to 1100 nm.

[0067] As shown by R (double line) on the graph, the R filter section has approximately maximum transmittance at a wavelength of 600 nm, and on the longer wavelength side, the approximately maximum transmittance is maintained even beyond 1000 nm.

[0068] As shown by graph G (wide dashed line), the G filter section has a maximum transmittance peak at a wavelength of about 540 nm, and a minimum transmittance portion at about 620 nm on the longer wavelength side. The G filter section also shows an increasing trend on the longer wavelength side than the minimum transmittance portion, with transmittance reaching a maximum at about 850 nm. On the longer wavelength side, transmittance remains maximum even beyond 1000 nm.

[0069] As shown by graph B (narrowly spaced dashed line), filter section B has a maximum transmittance peak at about 460 nm, and a minimum transmittance at about 630 nm on the longer wavelength side.The transmittance also tends to increase on the longer wavelength side, reaching a maximum at about 860 nm, and on the longer wavelength side, the transmittance remains maximum even beyond 1000 nm.

[0070] The IR filter section blocks light on the short wavelength side from about 780 nm and light on the long wavelength side from about 1020 nm, with the transmittance being greatest in the range from about 820 nm to 920 nm.

[0071] The transmittance spectrum of each of the R, G, B, and IR filter sections is not limited to that shown in Fig. 3, but it is believed that the color filters 3 that are currently in general use exhibit a transmittance spectrum close to this. Note that the value of 1 on the horizontal axis indicating the transmittance does not mean that 100% of light is transmitted, but indicates, for example, the maximum transmittance in the color filter 3.

[0072] Here, as shown by the graph DBPF (solid line), the DBPF5 used in this embodiment has high transmittance in two bands: the visible light band indicated by DBPF (VR) and the infrared band (second wavelength band) indicated by DBPF (IR) at a position slightly away from the visible light band on the long wavelength side. In addition, the DBPF (VR) as the band with high transmittance in the visible light band is, for example, a wavelength band of about 370 nm to 700 nm. In addition, the DBPF (IR) as the second wavelength band with high transmittance on the infrared side is, for example, a band of about 830 nm to 970 nm.

[0073] In this embodiment, the relationship between the transmittance spectrum of each filter portion of the above-mentioned color filter 3 and the transmittance spectrum of DBPF5 is defined as follows: That is, DBPF(IR), which is the second wavelength band that transmits infrared light in the transmittance spectrum of DBPF5, is included in wavelength band A shown in Fig. 2 where the R filter portion, G filter portion, and B filter portion all have approximately the maximum transmittance and the transmittance is approximately the same in each filter portion, and is also included in wavelength band B where light is transmitted with the maximum transmittance of the IR filter portion.

[0074] Here, the wavelength band A where the transmittance of each of the R, G, and B filter sections is the same is a portion where the difference in transmittance of each filter section is 10% or less in transmittance. Note that, on the shorter wavelength side than this wavelength band A, the transmittance of the G and B filter sections is lower than that of the R filter section, which has approximately the maximum transmittance. In DBPF5, the portion where the transmittance of each of the R, G, and B filter sections is different corresponds to the portion where the transmittance is extremely small and substantially blocks the light of DBPF5 between DBPF(VR), which is the portion where the transmittance of the visible light band is high, and DBPF(IR), which is the portion where the transmittance of the second wavelength band of the infrared light band is high. That is, on the infrared side, the transmission of light is cut off in the portion where the difference in the transmittance of each of the R, G, and B filter sections is large, and light is transmitted in the wavelength band A where the transmittance of each filter section is maximum on the longer wavelength side and the transmittance is the same.

[0075] From the above, in this embodiment, the DBPF5 used in place of the infrared cut filter has a region that transmits light not only in the visible light band but also in the second wavelength band on the infrared side, so that color photography using visible light is affected by the light that has passed through the second wavelength band. However, as described above, the second wavelength band does not transmit light in the parts where the transmittance differs in each of the R, G, and B filter sections, but transmits only light in wavelength bands where the transmittance of each filter section is maximized and the same transmittance.

[0076] In addition, in the second wavelength band of DBPF5, the light in the portion where the transmittance is maximum in the IR filter section is transmitted. Therefore, assuming that R, G, B, and IR filter sections are respectively provided in four very close pixels irradiated with the same light, in the second wavelength band, light passes through the R filter section, the G filter section, the B filter section, and the IR filter section in the same manner, and as light on the infrared side, the same amount of light reaches the photodiode of the image sensor body through each filter section including IR. In other words, the amount of light passing through the second wavelength band on the infrared side among the light passing through each of the R, G, and B filters is the same as the amount of light passing through the IR filter section. If we make the above-mentioned assumptions, the difference between the output signal of a pixel assumed as described above from the sensor body 2 that receives light that has passed through each of the R, G, and B filters and the output signal of a pixel assumed as described above from the sensor body 2 that receives light that has passed through an IR filter basically becomes the output signal of the visible light portion of each of the R, G, and B, with the infrared light that has passed through the R, G, and B filter sections cut out.

[0077] In reality, in the color filter 3, one of the R, G, B, and IR filter sections is disposed in each pixel of the sensor body 2, and it is highly likely that the amount of light of each color irradiated to each pixel will be different. For this reason, for example, a known interpolation method is used in each pixel to obtain the luminance of each color of each pixel, and the difference between the R, G, and B luminance of each pixel that has been interpolated and the IR luminance that has also been interpolated can be set as the R, G, and B luminance. Note that the image processing method for removing infrared light components from the luminance of each color of R, G, and B is not limited to this, and any method may be used as long as it can finally remove the influence of light that has passed through the second wavelength band from the luminance of each color of R, G, and B. In any method, the DBPF 5 removes the portion where the transmittance of the R, G, and B filter sections differs by more than 10% on the infrared side, i.e., the portion where the transmittance differs by more than a predetermined percentage, so that the process of removing the influence of infrared light in each pixel becomes easy.

[0078] As described above, by using the imaging sensor unit 12, it is possible to realize the imaging device 100 capable of both color photography and infrared photography. In general, it is conceivable to perform normal photography in color photography, and to perform infrared photography using infrared lighting that is difficult for humans to recognize at night, without using visible light lighting. For example, in various surveillance cameras and the like, it is conceivable to perform night photography using infrared light using infrared lighting when taking night photography in a place where night lighting is not required or where night lighting is preferable. It can also be used for purposes such as daytime photography and nighttime photography for observing wild animals.

[0079] When infrared light photography is used for nighttime photography, infrared light illumination is required because, like visible light, the amount of light produced by infrared light is insufficient at night.

[0080] The transmittance spectra (A) and (B) of DBPF5 shown in FIG. 21 were determined taking into consideration the transmittance spectra of the R, G, B, and IR filter sections and the emission spectrum of the light for infrared illumination, for example, an infrared LED for illumination.

[0081] FIG. 21 shows the transmittance spectra R, G, B, and IR of the filter sections of each color as in FIG. 2, the transmittance spectrum DBPF of DBPF5, and the emission spectrum IR-light of the LED lighting.

[0082] The second wavelength band indicated by DBPF(IR), which is the portion of the DBPF shown in FIG. 21(A) that transmits infrared light, is included in wavelength band A shown in FIG. 2 where the R filter section, G filter section, and B filter section all have approximately maximum transmittance and the transmittance is approximately the same in each filter section, similar to the DBPF shown in FIG. 2, and is also included in wavelength band B which transmits light with the maximum transmittance of the IR filter section.

[0083] In addition, the wavelength band of DBPF(IR) is set to include substantially the entire wavelength band that is the peak of the emission spectrum of the infrared illumination, which is included in both the above-mentioned wavelength band A and wavelength band B. When infrared photography is performed under infrared illumination rather than natural light at night, the second wavelength band indicated by DBPF(IR) does not need to be wider than the peak width of the optical spectrum of the infrared illumination, and when the spectrum of the infrared illumination is included in both the above-mentioned wavelength band A and wavelength band B, a peak portion of the transmittance of DBPF5 indicated by DBPF(IR) may be set as the second wavelength band with a peak width substantially similar to the peak width of the peak having an apex at, for example, about 860 in the emission spectrum of the infrared illumination.

[0084] That is, in FIG. 21(A), the peak in the emission spectrum of infrared light illumination indicated by IR-light is on the short wavelength side of the above-mentioned wavelength band A and wavelength band B, and the second wavelength band of DBPF indicated by DBPF(IR) overlaps with the peak of the emission spectrum of IR-light in the short wavelength portion of wavelength band A and wavelength band B.

[0085] 21(B) is similar to (A) in that the emission spectrum of infrared light illumination is added to the graph shown in FIG. 2, and the second wavelength band shown by DBPF(IR), which is the portion of the transmittance spectrum of DBPF5 with high transmittance on the infrared side, is aligned with the peak of the emission spectrum shown by IR-light of the above-mentioned infrared light illumination.

[0086] 21(B) uses infrared light illumination whose emission spectrum peak wavelength is longer than that of (A), and this peak is included in the above-mentioned wavelength bands A and B, and exists on the longer wavelength side of wavelength bands A and B. Correspondingly, a second wavelength band indicated by DBPF(IR) of DBPF5 is provided so as to overlap with the peak indicated by IR-light of the infrared illumination within the above-mentioned wavelength bands A and B.

[0087] 2 or 21, as long as the second wavelength band is included in both the above-mentioned wavelength band A and wavelength band B. Furthermore, when a wavelength band that is a peak in the emission spectrum of the infrared illumination used in nighttime infrared photography is determined, it is preferable that the wavelength band is included in both the above-mentioned wavelength band A and wavelength band B, and the second wavelength band of DBPF5 is aligned with the peak in the emission spectrum of the infrared illumination.

[0088] In such an image sensor, the second wavelength band that transmits light on the infrared side of DBPF5 is included in wavelength band A where the transmittance of each filter section is maximum on the infrared side of each of the R, G, B, and IR filter sections and the transmittance of each filter section is the same, and is also included in wavelength band B where the transmittance of the IR filter section is maximum. In other words, on the longer wavelength side than the visible light band, only the R filter section has the maximum transmittance of each of the R, G, and B filters, and the G and B filter sections do not have the maximum transmittance, so that light in the different parts where the transmittance of each of the R, G, and B filter sections is not the same is cut by DBPF5.

[0089] That is, since the R, G, B, and IR filter sections are configured to transmit light of the second wavelength band on the infrared side, the transmittance of each filter section on the infrared side is the same. If the same amount of light of the second wavelength band is irradiated, the amount of light transmitted through each of the R, G, B, and IR filter sections will be the same. This makes it possible to easily obtain an image in which the influence of infrared light passing through the second wavelength band of color during color shooting is suppressed by correcting the color based on the output signal from each of the R, G, and B filter sections as described above.

[0090] Furthermore, by making the second wavelength band correspond to the peak of the emission spectrum of the infrared light illumination contained in the above-mentioned wavelength band A and wavelength band B, the light of the infrared illumination can be used efficiently, and by narrowing the width of the second wavelength band, the influence of infrared light passing through the second wavelength band can be reduced during color shooting.

[0091] FIG. 5 is a block diagram showing the signal processing in the signal processing unit 12. As shown in FIG.

[0092] An outline of processing for the output signal from the image sensor unit 12 equipped with the color filter shown in FIG. 4 will be described.

[0093] The output signals of each pixel of R, G, B, and IR are sent to each of the internal processing blocks 21r, 21g, 21b, and 21ir. In each of the internal processing blocks 21r, 21g, 21b, and 21ir, an interpolation process using a known method is used to convert the R, G, B, and IR signals so that the image data of each frame of the above-mentioned color filter 3 becomes image data 20r in which all pixels are represented by red R, image data 20g in which all pixels are represented by green G, image data 20b in which all pixels are represented by blue B, and image data 20ir in which all pixels are represented by infrared IR, respectively.

[0094] Next, in the infrared light removal signal generating blocks 22r, 22g, 22b, and 22ir, signals to be subtracted from the R, G, and B color signals are generated from the IR signal in order to remove the influence of the infrared light received from the second wavelength band described above. The signals generated for each of the R, G, and B colors in the infrared light removal signal generating blocks 22r, 22g, and 22b are subtracted from the R, G, and B color signals. In this case, as described above, for the same pixel, the IR signal is basically removed from each of the R, G, and B signals, making processing easier. In reality, the sensitivity of each pixel of each color differs due to the characteristics of the filter section of each pixel, so a signal to be subtracted from each of the R, G, and B signals for each of the R, G, and B images is generated from the IR signal.

[0095] Next, in an image processing block 23, each of the R, G, and B signals undergoes a well-known RGB matrix process that converts each of the R, G, and B signals using a determinant to correct the color, a well-known white balance process that makes the output values ​​of each of the R, G, and B signals the same in white parts of the image, and a well-known gamma correction that is a correction for image output to a display or the like. Next, in a luminance matrix block 24, each of the R, G, and B color signals is multiplied by a coefficient to generate a luminance Y signal. In addition, the luminance Y signal is divided from the blue B signal and the red R signal to calculate the color difference signals of RY and BY, and the Y, RY, and BY signals are output.

[0096] In addition, the IR signal is basically output as a black and white gradation image.

[0097] Fig. 6 shows a communication flow for exchanging visible images, infrared images, and control commands between the imaging device 100 and the controller device 200 shown in Fig. 1. Here, the communication flow may be a proprietary communication protocol, but may also use a protocol established by ONVIF (Open Network Video Interface Forum) as a standard communication for surveillance cameras.

[0098] First, the imaging device 100 is placed in a predetermined location and connected to the network 303. When the power is turned on, the imaging device 100 starts up and the control unit 23 of the imaging device 100 executes an initial setting process. For example, this mainly includes starting up the hardware and setting initial parameters for the software, such as loading a program stored in the memory 21 and obtaining the current location from the GPS 19. Note that the imaging device 100 may be started up when it is connected to the network 303 using PoE (Power Over Ethernet) and a PoE-compatible hub (step 601).

[0099] When the necessary initial setting processing is completed, the control unit 23 of the imaging device 100 sets an IP address to be used in the communication control unit 15 and the IF unit 16. The IP address is set using a general-purpose network protocol, such as a method of directly connecting a PC, a pad terminal, or the like via the maintenance IF unit 22 and setting a static IP address, or a method of automatically setting an IP address using DHCP (Dynamic Host Configuration Protocol) (step 602).

[0100] When the IP address setting is completed, the control unit 23 of the imaging device 100 instructs the communication control unit 15 to notify the controller device 200 of its own existence. Protocols such as UPnP (Universal Plug and Play) and WS-Discovery (Web Services Dynamic Discovery) may be used as a method for automatically discovering devices present on the network. Furthermore, when making the notification, the manufacturer name, model name, installation location, date and time, etc. may be included (step 603). At this time, the installation location may be information that has been initially set in advance or information obtained from the GPS 19. Furthermore, information determined by the GPS 19 or the illuminance monitoring unit 18 as to whether the device is outdoors or indoors may be included.

[0101] The control unit 214 of the controller device 200 that has received the notification can recognize the presence of the imaging device 100 by acquiring the IP address of the imaging device 100. The control unit 214 notifies the administrator via the display unit 202 that a new imaging device 100 has been connected, and waits for an instruction from the administrator as to whether or not the control unit 214 will manage the imaging device 100. If an instruction is received from the administrator via the user IF unit 201, or if the controller device 200 checks the number of imaging devices 100 currently managed and finds that the maximum number has not been reached, the control unit 214 automatically instructs the communication control unit 206 to transmit a request to acquire installed function information to the imaging device 100 (step 605).

[0102] The control unit 23 of the imaging device 100 that has received the function information acquisition request acquires the function information stored in the memory 21, and instructs the communication control unit 15 to transmit the function information to the controller device 200. The function information includes, for example, device management information (support and parameter values ​​related to networks, systems, and security), imaging device performance information (parameter values ​​related to image quality such as backlight compensation, brightness, contrast, white balance, focus adjustment, and wide dynamic range, and parameter values ​​related to media profiles such as resolution, frame rate, and codec type), PTZ (pan-tilt-zoom) function information (coordinate system definition, movable parameters, preset positions, etc.), and analysis function information (supported analysis functions, types of authentication, and format of analysis results, etc.) (step 606).

[0103] Here, FIG. 7 shows one configuration of information related to the imaging device 100 of this embodiment. The imaging device 100 of this embodiment has four types of output image types 701: "visible image" and "infrared image", and four types of output modes 702 to be transmitted to the controller device 200: "output visible image only", "output infrared image only", "automatically switch between one of the two images depending on illuminance and time and output", and "output both visible image and infrared image simultaneously". In addition, the access destination information 703 of the visible image has URI / URL information that the controller device 200 accesses to obtain information about the visible image and the actual visible image from the imaging device 100, and similarly, the access destination information 704 of the infrared image has URI / URL information that the controller device 200 accesses to obtain information about the infrared image and the actual infrared image from the imaging device 100. In addition, the visible image information 705 includes a codec, transfer rate, resolution, etc. of the visible image that can be output, and the infrared image information 706 includes a codec, transfer rate, resolution, etc. of the infrared image that can be output. This configuration is an example, and other information may be included.

[0104] The control unit 214 of the controller device 200 that has received the functional information of the imaging device 100 notifies an administrator of the contents of the functional information via the display unit 202 or automatically confirms the contents, and if it is decided that the functional information should be managed by the controller device 200, adds the functional information to the camera management unit 208 as a subject to be managed. The camera management unit 208 stores and manages all or part of the functional information in the memory 204. The control unit 214 also confirms the analysis function and authentication function supported by the controller device 200 itself, and determines whether or not to use the images of the imaging device 100. Alternatively / additionally, the control unit 214 may check the analysis function information supported by the imaging device 100, and determine the authentication method and analysis method to be executed when the imaging device 100 is used (step 607).

[0105] When the control unit 214 of the controller device 200 decides to use the imaging device 100, the control unit 214 instructs the communication control unit 206 to transmit a device setting request to the imaging device 100 in order to set parameters that need to be changed or set among those included in the function information acquired in step 606. For example, in this embodiment, the output mode 702 is set to "output both visible and infrared images simultaneously" (step 608). Here, the output mode 702 may be determined based on, for example, the installation location of the imaging device 100.

[0106] The control unit 23 of the imaging device 100 that has received the device setting request checks whether the received setting is executable or not, and returns the execution result to the controller device 200 (step 609).

[0107] Next, the control unit 214 of the controller device 200 instructs the communication control unit 206 to transmit an access destination information acquisition request for acquiring protocols and parameters required for actually acquiring visible images and infrared images (step 610).

[0108] The control unit 23 of the imaging device 100 that has received the access destination information acquisition request instructs the communication control unit 15 to return access destination information for the media (e.g., media type, port number, transfer protocol, payload number, etc.) including the access destination information 703 for the visible image and the access destination information 704 for the infrared image (step 611).

[0109] The control unit 214 of the controller device 200 that has received the access destination information subsequently transmits to the imaging device 100 a request to acquire session information (DESCRIBE) necessary for receiving an image (step 612).

[0110] The control unit 23 of the imaging device 100 that has received the session information acquisition request instructs the communication control unit 15 to generate session information described using SDP (Session Description Protocol) and transmit it to the controller device 200 (step 613).

[0111] The control unit 214 of the controller device 200 that has received the session information instructs the communication control unit 206 to establish an RTSP session with the imaging device 100. Here, typically, separate RTSP sessions are established for the transfer of visible images and infrared images (step 614).

[0112] After establishing the RTSP session, the controller device 200 prepares to receive these images and prepares for face authentication (step 615), and the imaging device 100 prepares to transmit visible images and infrared images (step 616) and transmits the results (step 617).

[0113] When the control unit 214 of the controller device 200 confirms that all preparations are complete, it instructs the communication control unit 206 to transmit a streaming start request (PLAY) to the imaging device 100 (step 618).

[0114] The control unit 23 of the imaging device 100, which has received the streaming start request, instructs the signal output control unit 14 to output the image as requested by the controller device 200 in step 608, and instructs the communication control unit 15 to transmit the image output by the signal output control unit 14 to the imaging device 100 using RTP over the session established in steps 612 / 613 (step 620).

[0115] Furthermore, the control unit 214 of the controller device 200 starts receiving images (step 621).

[0116] Thereafter, the imaging device 100 executes RTP transfer of the visible image and the infrared image captured by the imaging device 100 (steps 621 and 622). Here, in order to reduce the processing load on the controller device side, the communication control unit 15 of the imaging device 100 may use a marker bit in the RTP header so that the break between frames can be identified.

[0117] Furthermore, the communication control unit 15 of the imaging device 100 transmits an RTCP transmission report to the controller device 200 every time a predetermined number of frames are transferred. To indicate that a visible image and an infrared image have been captured simultaneously, the same time stamp, frame number, packet count, etc. are stored in the report (step 623).

[0118] The control unit 214 of the controller device 200 receives the visible image and the infrared image from the imaging device 100, and stores these images in the memory 204 via the recording and reproducing unit 205, while executing face authentication using the moving object region extraction unit 209, the face region detection unit 210, the face feature point detection unit 211, and the face matching unit 212. Then, the control unit 214 controls interruption or stop of streaming as necessary (step 624).

[0119] The basic communication flow between the controller device 200 and the imaging device 100 has been described above.

[0120] Here, RTP communication is used in the above communication flow, but HTTP communication or other proprietary communication methods may also be used. Also, instead of transferring the visible image and the infrared image in separate streams, they may be transferred superimposed on the same stream (for example, a common header (including timestamp and sequence number) + first visible image + first infrared image + ..., etc.). Transferring both images at the same time increases the usage of the communication bandwidth, so the infrared image may be transferred every frame and the visible image every 30 frames. In this case as well, for frames captured at the same time, the same timestamp and frame number are used for the infrared image and the visible image.

[0121] Here, in step 623, the image capture device 100 sends a transmission report to the controller device 200, but similarly, the controller device 200 may send to the image capture device 100 a reception report including information regarding packet loss and transfer delays.

[0122] Also, in step 623, it was described that a transmission report with the same timestamp and frame number is sent to indicate that the visible image and the infrared image were captured simultaneously. However, other methods may be used, for example, by setting the timestamp and sequence number of the RTP header to be sent to the same value, or by setting the same timestamp and frame number in the extension header of the RTP header.

[0123] The control unit 214 of the controller device 200 instructs the recording and reproducing unit 205 to store the received visible image and infrared image in the memory 204, and detects people included in the images using the moving object region extraction unit 209, face region detection unit 210, face feature point detection unit 211, and face matching unit 212, and performs face authentication to determine whether or not the person is suspicious. At this time, there is a method of performing face authentication on both images to improve authentication accuracy, taking advantage of the advantage that visible images and infrared images of the same object can be obtained at the same time, and furthermore, synchronization of both images is easy because the same time stamp and sequence number are added, or a method of performing face authentication on only one of the images (for example, only the infrared image) and using the other image with the same sequence number when it is desired to compare and check both images (e.g., when additional information such as background and color is to be grasped, or when there is a part in the other image that is to be face-authenticated, etc.).

[0124] In addition, in this embodiment, "simultaneous output of both visible and infrared images" is set as the output mode 702 in step 608, but it may be changed depending on the time or surrounding environment, such as setting "simultaneous output of both visible and infrared images" during the day and "infrared images only" at night. Alternatively, when face authentication is performed while receiving either one of the images, if it is desired to obtain further information on a person who appears to be suspicious as a result of matching by the face comparison unit 212, it may be automatically switched to receiving both images midway.

[0125] If the control unit 214 of the controller device 200 determines, as a result of matching by the face matching unit 212, that there is a suspicious person or a suspected suspicious person in the image, it notifies an administrator via the display unit 202, or notifies or shares the information with other controller devices 200 via the IF unit 207, making it possible to track the suspicious person among multiple imaging devices 100.

[0126] (Embodiment 2) Next, a configuration of an imaging system according to a second embodiment of the present invention will be described.

[0127] 8 shows a configuration example of an imaging system according to the second embodiment of the present invention. Note that imaging device 100 according to the first embodiment and imaging devices 800 and 810 according to this embodiment can be installed in the same imaging system in a mixed manner, and controller device 200 can manage these imaging devices.

[0128] The imaging device 800 of this embodiment has a configuration in which a moving object region extraction unit 801 having the same function as the moving object region extraction unit 209 of the controller device 200 is installed in the imaging device 100 of the first embodiment. The other components are similar to those of the imaging device 100.

[0129] Of the visible image and infrared image output by the signal processing unit 13, the control unit 23 of the imaging device 800 inputs only the infrared image to the moving object region extraction unit 801. Reasons for using only the infrared image include that it is possible to detect objects that cannot be detected in a visible image, and that the contrast between a person and the background is greater than in a visible image, making it effective for human detection.

[0130] The moving object region extraction unit 801 extracts moving object regions in the input infrared image using the image, and outputs the number and position information of the moving object regions. These results are output to the control unit 25 or the signal output control unit 14. They may also be stored in the memory 22.

[0131] As described above, the imaging device 800 can constantly monitor the moving object area in the image by using the infrared image out of the visible image and the infrared image captured at the same timing, and can provide information on the moving object area extracted with high accuracy together with the visible image or the infrared image to the controller device 200. Since the controller device 200 can obtain information on the moving object area together with the image, it can reduce the image processing load.

[0132] Here, in order to reduce the amount of communication bandwidth used on the network, the control unit 23 of the imaging device 800 outputs an image from the signal output control unit 14 via the IF unit 16 only when a moving object area is extracted by the moving object area extraction unit 801, and if a moving object area cannot be extracted, it may be configured to not output an image from the signal output control unit 14, or to lower the frame rate of the image output from the signal output control unit 14.

[0133] In addition, the control unit 23 of the imaging device 800 may instruct the signal output control unit 14 to combine the moving object area extracted by the moving object area extraction unit 801 with the visible image and / or infrared image output by the signal processing unit 13, and generate / process an image in which the moving object area is surrounded by a rectangle on the image.

[0134] Similarly, an imaging device 810 of this embodiment has a configuration in which a face area detection unit 802 having the same function as the face area detection unit 210 of the controller device 200 is mounted on the imaging device 800. Other components have the same configuration as the imaging device 100.

[0135] The control unit 23 of the imaging device 810 inputs only the infrared image out of the visible image and infrared image output by the signal processing unit 13 to a moving object region extraction unit 801. The moving object region extraction unit 801 extracts moving object regions in the image using the input infrared image, and outputs the number and position information thereof to the control unit 23 or the signal output control unit 14, and at the same time inputs them to a face region detection unit 802. The face region detection unit 802 detects a region where a human face exists from the input moving object region, and outputs it to the control unit 23 or the signal output control unit 14.

[0136] As described above, the imaging device 810 can constantly monitor the moving object area in the image by using the infrared image out of the visible image and infrared image captured at the same timing, extract the moving object area with high accuracy, and further detect the area where a human face exists from the moving object area, and provide the information on the moving object area and the face area together with the visible image or infrared image to the controller device 200. Since the controller device 200 can obtain this information together with the image, the image processing load can be reduced.

[0137] Here, in order to reduce the usage of communication bandwidth on the network, the control unit 23 of the imaging device 810 may output an image from the signal output control unit 14 via the IF unit 16 only when a human face area is detected by the face area detection unit 802, and may not output an image from the signal output control unit 14 or may lower the frame rate of the image output from the signal output control unit 14 if a human face area cannot be detected even after extracting a moving object area. Similarly, in order to detect only an object, an image may be output from the signal output control unit 14 via the IF unit 16 only when a moving object area in which a human face area cannot be detected is detected.

[0138] In addition, the control unit 23 of the imaging device 810 may instruct the signal output control unit 14 to combine the face area extracted by the face area detection unit 802 (and the moving object area extracted by the moving object area extraction unit 801) with the visible image and / or infrared image output by the signal processing unit 13, and generate / process an image in which the face area is surrounded by a rectangle on the image.

[0139] The communication flow between the imaging devices 800, 810 and the controller device 200 is almost the same as that shown in FIG. 6 of the first embodiment, and only the differences will be described below.

[0140] First, in step 606 in Fig. 6, the functional information provided by the imaging devices 800 and 810 to the controller device 200 includes, as the above-mentioned analysis functional information, information as shown in Fig. 9 in addition to the information shown in Fig. 7 as an example. That is, in the functional information, the imaging device 800 includes information indicating that it is equipped with a "moving object region extraction function," and the imaging device 801 includes information indicating that it is equipped with a "moving object region extraction function" and a "face region detection function."

[0141] In this embodiment, this information is referred to as analysis metadata, and the functional information (analysis function information) includes, as shown in FIG. 9, analysis metadata type 901, analysis metadata output mode 902, moving object area metadata access destination information 903, face area metadata access destination information 904, moving object area / face area metadata access destination information 905, moving object area metadata information 906, and face area metadata information 907.

[0142] In step 607, the control unit 214 of the controller device 200 that has received the function information checks the analysis functions and authentication functions supported by the controller device 200 itself, and judges whether or not to use the analysis metadata output by the image capture device 800. For example, it is possible to select to use only the analysis metadata "position information of a moving object area" for both the image capture devices 800 and 810, or to not use the analysis metadata of the image capture device 800 and to use only the analysis metadata "position information of a face area" of the image capture device 810.

[0143] 10 shows a communication flow for transmitting visible images, infrared images, and analysis metadata between the image capturing devices 800, 810 and the controller device 200. In this description, it is assumed that the image capturing device 800 transmits "position information of a moving object region" and the image capturing device 810 transmits both analysis metadata of "position information of a moving object region" and "position information of a face region." It is also assumed that the image capturing devices 800, 810 establish a session for transferring analysis metadata in addition to the visible images and infrared images in step 614 in FIG. 6.

[0144] The imaging devices 800 and 810 start frame transfer of visible images and infrared images (steps 1001 and 1002), and every time a predetermined number of frames are transferred (steps 1003 and 1004), they transmit analysis metadata extracted by the moving object region extraction unit 801 and face region detection unit 802 (step 1005). Here, the analysis metadata may be sent at the timing when the moving object region or face region is detected.

[0145] The control unit 214 of the controller device 200 that has received the analysis metadata 1200 checks whether the analysis metadata 1200 includes information on a moving object region or information on a face region (step 1006). If the analysis metadata 1200 does not include information on a moving object region, the control unit 214 executes a moving object region extraction process using its own moving object region extraction unit 209 (step 1007).

[0146] On the other hand, if the information on the moving object area or the information on the face area is included, it is confirmed whether the information on the face area is included (step 1008).If the information on the face area is not included (i.e., only the information on the moving object area is included), the information on the received moving object area and the face area detection unit 210 included in the device are used to execute the detection process of the face area (step 1008).

[0147] On the other hand, if the facial area information is included, the facial feature points are extracted using the received facial area information and the facial feature point detection unit 211 contained in the device (step 1010), and matching is performed using the face matching unit 212 (step 10100).

[0148] Fig. 11 shows an image handled by the imaging devices 800 and 810. Image 1100 is an example of a visible image captured by the imaging devices 800 and 810. Image 1101 is obtained by removing the background from image 1100 and extracting only the moving object area. In this example image, three areas (A), (B), and (C) (areas surrounded by dotted line rectangles) have been extracted. Image 1102 is obtained by further extracting a face area from image 1101. In this example image, two areas (a) and (b) (areas surrounded by solid line rectangles) have been extracted.

[0149] FIG. 12 shows an example of the structure of the analysis metadata 1200 sent by the image capture devices 800 and 810 in step 1005. In FIG.

[0150] The analysis metadata 1200 is roughly composed of a communication header 1201 and a payload 1210. The communication header 1201 is similar to, for example, an RTP header or an HTTP header.

[0151] The payload 1210 stores analysis metadata, such as a frame number 1211 of an infrared image used to extract a moving object region or a face region, a frame number 1212 of a visible image, a maximum number 1213 of moving object regions that the imaging devices 800 and 810 can extract, a number 1214 (here, n) of moving object regions actually extracted by the moving object region extracting unit 801, coordinate information 1 to n (1215 to 1216) of the extracted moving object regions, a maximum number 1217 of face regions that the imaging device 810 can extract, a number 1218 (here, m≦n) of face regions actually extracted by the face region detecting unit 802, and coordinate information 1 to m (1219 to 1220) of the extracted moving object regions.

[0152] As described above, the imaging devices 800, 810 of this embodiment can provide the controller device 200 with not only visible images and infrared images, but also information on moving object regions and information on human regions accurately extracted using the infrared images, simultaneously with the required image output.

[0153] On the other hand, the controller device 200 can omit conventional procedures by immediately using the received moving object area information and human area information, and therefore can reduce the execution time of face authentication compared to conventional methods. This is effective in reducing the processing load of the controller device 200 when managing many imaging devices with one controller device 200.

[0154] Here, in this embodiment, an example has been described in which imaging devices 800, 810 transmit at least one of a visible image and an infrared image and analysis parameters to controller device 200, but in order to reduce the amount of data on the network, it is also possible to send the analysis parameters and an image of only the part indicated by the analysis parameters (moving object area, face area).

[0155] Furthermore, the control unit 23 of the imaging devices 800, 810 may hold the frame number of the corresponding image when the moving object region extraction unit 801 first detects a moving object region in an image, and may track the moving object region in images captured consecutively to that image until the subject of the moving object region is no longer present, and may add the frame number as auxiliary information to the coordinate information in the analysis metadata 1200 shown in Fig. 12. This allows the controller device 200 to easily grasp the frame number containing the moving object region and calculate the time by referring to the analysis metadata 1200.

[0156] (Embodiment 3) Next, a configuration of an imaging system according to a third embodiment of the present invention will be described.

[0157] The imaging devices of the above-mentioned first and second embodiments capture visible images and infrared images using one set of lens 11, imaging sensor unit 12, and signal processing unit 13. The imaging device of this embodiment has two sets of lenses 11, imaging sensor unit 12, and signal processing unit 13 arranged on the left and right, and is capable of capturing stereo images (distance images) consisting of two images, one on the left and one on the right, using visible light and one on the right using infrared light.

[0158] 13 shows an example of the configuration of an imaging system according to this embodiment. This imaging system is composed of one or more imaging devices 1300 and a controller device 1310.

[0159] As described above, the imaging device 1300 includes two sets of lenses 11, an imaging sensor unit 12, a signal processing unit 13, and also includes a correction parameter calculation unit 1301 and a distance calculation unit 1302. The two lenses 11(a) and 11(b) are arranged on the left and right so that their optical axes are parallel to each other. The other components are basically the same as those of the imaging devices 100, 800, and 810 of the first and second embodiments.

[0160] The correction parameter calculation unit 1301 sets parameters such as clip levels and signal level correction values ​​(for example, correction values ​​to be added to, subtracted from, multiplied by, or divided by signals such as a visible image signal, an infrared image signal, an infrared signal, or each color signal) so as to approximate the signal intensities (signal levels) of the respective visible images output from the two signal processing units 13(a) and (b), thereby approximating the signal levels of the two visible image signals (two infrared image signals). The amount of correction in the image signal correction processing unit 203 is set respectively by looking at the outputs from the two signal processing units 13(a) and (b), thereby aligning the levels of the image signals. The process of aligning the levels of the left and right image signals can be performed on both the infrared image signal and the visible image signal.

[0161] That is, the correction parameter calculation unit 1301 determines the amount of correction based on the signal levels of the image signals output from the two signal processing units 13(a)(b) so that the signal levels of the image signals output from the two signal processing units 13(a)(b) are similar to each other. This makes it possible to suppress errors and errors in the distance measurement caused by different parts of the subject being recognized as the same part (corresponding points) due to differences in luminance levels, for example, between the two image data.

[0162] The distance calculation unit 1302 calculates the distance to the object using two visible image signals or infrared image signals input from the two signal processing units 13(a) and (b), respectively. At this time, the same subject (corresponding points) are determined from the two images, and the distance is calculated in the same manner as in the conventional method by detecting the parallax as the difference in the positions of these same subjects on the images. That is, the corresponding points for measuring the difference are determined by image recognition, and the distance is calculated based on the parallax as the difference in the positions of the corresponding points on the images. Then, a stereo image (distance image) is generated based on the distance information corresponding to each pixel, and is output to the signal output control unit 14.

[0163] The signal output control section 14 is able to provide the controller device 1310 with the stereo image (distance image) generated by the distance calculation section 1302 in addition to the two visible images and infrared images captured on the left and right.

[0164] As described above, the imaging device 1300 can simultaneously capture a visible light image and an infrared image of a subject, and can calculate the distance from both images. In this case, since the positions of the visible image and the infrared image are aligned, it is possible to prevent the distance measured between the two images from changing.

[0165] Here, the distance calculation unit 1302 may use two visible images and two infrared images to calculate the respective distances, and then generate two stereo images (distance images) and output them as they are. Alternatively, the two generated stereo images may be compared, and if the difference in the distance information is within a predetermined threshold range, one of the distance images may be output, and if the difference exceeds the threshold, both distance images may be output, or a distance image that has been set to be output in advance (for example, a distance image calculated using an infrared image may be given priority, and a distance image showing a value close to the distance may be output), or the area where the difference is exceeded may be output individually as analysis metadata. FIG. 22 shows an example of an outline of the processing performed by the distance calculation unit 1302.

[0166] Furthermore, the control unit 23 of the imaging device 1300, in accordance with instructions from the controller device 200, uses the signal output control unit 14 to control which of the visible images and infrared images output from the two signal processing units 13(a) and (b) and the stereo image (distance image) output from the distance calculation unit 1302 is to be output via the IF unit 16. For example, when the imaging device 1300 is installed in a place where privacy protection is required (e.g., a toilet or a changing room), only the stereo image is output, and when the imaging device 1300 is installed in a place where high security is required, all images are output.

[0167] On the other hand, the controller device 1310 is equipped with a different moving object region extraction unit 1311, face region detection unit 1312, face feature point detection unit 1313, face matching unit 1314, and 3D face DB 1315 in order to perform analysis processing and authentication processing using a stereo image (distance image) in addition to a visible image and an infrared image or using only a stereo image, instead of the moving object region extraction unit 209, face region detection unit 210, face feature point detection unit 211, face matching unit 212, and face DB 213 of the controller device 200 of the first and second embodiments. As a result, for example, in face recognition, three-dimensional data on the concaves and convexes of the face can be obtained and used to accurately and easily detect the face region and the face feature points.

[0168] In addition, by acquiring a stereo image (distance image) from the imaging device 1300, the controller device 1310 can refer to the distance of the moving object area extracted by the moving object area extraction unit 209 and determine whether to perform face authentication if the distance is within a specified distance, or not to perform it otherwise.

[0169] FIG. 23 shows an example in which the controller device 1310 displays the appearances of the imaging devices 1300(a), (b), and (c) installed at different locations. The controller device 1310 performs 3D-based face recognition using a visible image or an infrared image and a distance image received from the imaging device 1300(a) installed at the entrance of an office, building, etc., and displays the execution result. This makes it easier to check visitors and suspicious people, and helps to reduce congestion at the reception desk. In addition, the controller device 1310 displays the number of people looking at the product shelves and information to the extent that a person cannot be identified (for example, gender, height, face direction, body direction, posture, etc.) using a distance image received from the imaging device 1300(b) installed in a store in a public facility or commercial facility. This makes it possible to determine the purchasing demographics of shoppers and their level of interest based on their gaze and posture, which is useful for strengthening marketing / sales power regarding products and displays. Using visible images, infrared images, and distance images received from an imaging device 1300(c) installed outdoors in an amusement park, park, etc., the controller device 1310 extracts people from the images, performs 3D-based face recognition, and if it is confirmed that the person is a pre-registered person, it displays that part in a distance image, and displays only people who cannot be confirmed in a visible image or infrared image. Alternatively, it displays information that does not identify a person (for example, gender, height, face direction, whether or not a child is present, posture, etc.). This helps ensure the safety of visitors and early detection of suspicious people.

[0170] Next, Fig. 14 shows another example of the configuration of the imaging system of this embodiment. This imaging system is composed of one or more imaging devices 1400 and a controller device 1410. Although not shown, the imaging device 100 of the first embodiment, the imaging devices 800 and 810 of the second embodiment, and the imaging device 1300 may be mixed on the network 303. The controller device 1410 can manage any of the imaging devices.

[0171] The imaging device 1400 includes two moving object region extraction units 1401(a) and (b) in the imaging device 1300. The moving object region extraction unit 1401 may be the same as the moving object region extraction unit 1311 in the controller device 1310. The other components are the same as those in the imaging device 1300.

[0172] The moving object region extraction unit 1401(a)(b) is a unit that uses infrared images output from the two signal processing units 13(a)(b) to extract moving object regions from each image. As in the second embodiment, information on these extracted moving object regions can be output to the signal output control unit 14 or the control unit 23 and provided to the controller device 1410. In addition, the moving object region extraction unit 1401(a)(b) can extract moving object regions using stereo images (distance images) output from the distance calculation unit 1302, and extract moving object regions with higher accuracy by comparing the results with the above-mentioned method. Alternatively, the moving object region may be extracted first using stereo images (distance images), and then the infrared images may be used to check only the extracted moving object region in more detail.

[0173] Here, the control unit 23 can refer to the information on the two moving object regions output from the moving object region extraction units 1401(a)(b), compare the number of extracted moving object regions and their positions, and transmit the comparison result as analysis metadata. The controller device 1410 can use the analysis metadata to select which of the left and right visible images or infrared images to use for face recognition. For example, when the number of moving object regions extracted by the moving object region extraction unit 1401(a) (or the moving object region extraction unit 1401(b)) is greater than the number of moving object regions extracted by the moving object region extraction unit 1401(a) (or the moving object region extraction unit 1401(b)), the control unit 23 of the imaging device 1400 sends the information on the moving object regions extracted by the moving object region extraction unit 1401(a) (or the moving object region extraction unit 1401(b)) and the visible image or infrared image output from the signal processing unit 13(a) (or the signal processing unit 13(b)).

[0174] On the other hand, the controller device 1410 includes the face area detection unit 210, face feature points 211, face matching unit 212, face DB 213, and overall determination unit 1411 of the controller device 200 described in the first embodiment in the controller device 1310.

[0175] As a result, the imaging device 1400 can execute face recognition processing (using visible images and infrared images) using the face area detection unit 210, face feature point 211, face matching unit 212, and face DB 213 described in the first embodiment, in combination with face recognition processing (using visible images, infrared images, and stereo images) using the face area detection unit 1312, face feature point detection unit 1313, face matching unit 1314, and 3D face DB 1315 described above. The overall determination unit 1411 is a part that performs a final determination of the person recognition result based on the results of executing both face recognition processes. As a result, by executing two different types of face recognition methods, more accurate face recognition can be performed.

[0176] Similarly, Fig. 15 shows another example of the configuration of the imaging system of this embodiment. This imaging system is composed of one or more imaging devices 1500 and a controller device 1510. Although not shown, imaging device 100 of embodiment 1, imaging devices 800 and 810 of embodiment 2, and imaging devices 1300 and 1400 may coexist on network 303. Controller device 1510 is capable of managing any of the imaging devices.

[0177] The imaging device 1500 includes two face area detection units 1502(a) and (b) in the imaging device 1400. The face area detection units 1502 may be the same as the face area detection unit 1312 in the controller device 1310. The other components are the same as those in the imaging device 1400.

[0178] The face area detection units 1501(a)(b) are units that extract a human face area using information on the moving object areas output from the two moving object area extraction units 1401(a)(b). The information on the extracted face areas can be output to the signal output control unit 14 or the control unit 23 and provided to the controller device 1510, as in the second embodiment.

[0179] Here, the control unit 23 can refer to the information on the two face areas output from the face area detection units 1501(a)(b), compare the number and positions of the extracted face areas, and the face orientation, and transmit the comparison result as analysis metadata. The controller device 1510 can use the analysis metadata to select an image more suitable for face recognition, thereby performing face recognition with higher accuracy.

[0180] On the other hand, the controller device 1510 is obtained by newly mounting an authentication method selection unit 1511, an iris detection unit 1512, an iris matching unit 1513, and an iris DB 1514 on the controller device 200 or the controller device 1410.

[0181] The authentication method selection unit 1511 is a part that selects whether to perform face authentication or iris authentication using the visible image, infrared image, stereo image (distance image), analysis parameter information, etc. received from the imaging device 1500. For example, if the target object is within a predetermined distance range, iris authentication is performed, and otherwise face authentication is performed. Alternatively, face authentication is normally performed, and iris authentication is also performed when the conditions for iris authentication are met.

[0182] The iris detection unit 1512 detects the position of the iris of a person's eye using an infrared image received from the imaging device 1500 and analysis parameters including a face area extracted from the image, and further detects the boundary between the iris and the white of the eye and the boundary of the iris and other objects, specifies the iris area, and generates a pupil code. Note that any well-known method may be applied for these methods.

[0183] An iris matching unit 1513 performs matching using an iris DB 1514 based on the information detected by the iris detection unit 1512, in the same manner as face authentication.

[0184] This enables the controller device 1510 to select the most appropriate biometric authentication method using the visible image, infrared image, stereo image (distance image), analysis parameter information, etc. received from the imaging device 1500, thereby enabling more accurate personal authentication to be performed.

[0185] FIG. 24 shows an example in which the controller device 1410 and the controller device 1510 process the captured image of the imaging device 1500 installed at an airport or a building entrance. In this example, the controller device 1410 acquires a visible image and face area and distance information as analysis parameters from the imaging device 1500, and performs 2D face recognition with a relatively light image processing load for face areas that are far away, and performs 3D face recognition with a heavy image processing load for face areas that are close. In addition, the controller device 1510 acquires a visible image, an infrared image, and face area and distance information as analysis parameters from the imaging device 1500, and performs face recognition using the visible image for face areas that are far away, and performs iris recognition using the infrared image for face areas that are close. In addition, in order to eliminate congestion due to waiting in line, it is also possible to determine people who are close to a specified position in the captured image and perform face recognition in that order.

[0186] 16 shows another example of the configuration of the imaging system according to the present embodiment. This imaging system is mounted on a mobile terminal such as a smartphone or a tablet.

[0187] 17 and 18 show examples of the configuration of functional information and analysis parameters related to the imaging devices 1400 and 1500 used in this embodiment.

[0188] The stereo images (distance images) generated by the imaging device can be provided to the controller device by the same transfer method as for visible images and infrared images, as shown in FIG.

[0189] Alternatively, it is possible to provide the information by adding it to the position information of the moving object region or the position information of the face region as shown in Fig. 18. In that case, only the distance information related to the coordinate region corresponding to the moving object region or the face region is extracted and added.

[0190] 20 shows an example of a configuration in which distance information is sent as part of the analysis parameters. Distance information is stored in payload 1210, and for example, distance information on extracted moving object regions (number n) is stored immediately after the coordinate information of the moving object regions (2001, 2002), and distance information on face regions (number m) is stored immediately after the coordinate information of the face regions (2003, 2004). In addition to this configuration, a configuration in which coordinate information and distance information are stored alternately may also be used.

[0191] 19 shows another example of the configuration of the imaging system of this embodiment. The imaging device 1900 of this imaging system includes one moving object region extracting unit 1901, and is configured to extract a moving object region using an infrared image from either one of the left and right signal processing units 13(a)(b). Alternatively, the imaging device 1900 is configured to extract a moving object region using either one of the infrared images and a stereo image output from the distance calculation unit 1302. [Explanation of symbols]

[0192] 2...sensor body, 3...color filter, 5...DBPF (optical filter), 11...lens (optical system), 12...imaging sensor unit, 13...signal processing unit, 14...signal output control unit, 15...communication control unit, 16...IF unit, 23...control unit, 100, 800, 810, 1300, 1400, 1500...imaging device, 200, 1310, 1410, 1510...controller device, 801, 1401...moving object area extraction unit, 802, 1501...face area detection unit, 1301...correction parameter calculation unit, 1302...distance calculation unit, 1600...mobile terminal.

Claims

1. A first filter section and a second filter section each having a characteristic of transmitting at least a visible light wavelength region and an infrared wavelength region and filtering light based on said characteristic; a first image sensor that receives light from a subject that has been filtered by the first filter unit and outputs a signal; and a second image sensor that receives light from the subject that has been filtered by the second filter unit and outputs a signal. a first signal processing unit that processes a signal from the first imaging element and outputs a visible light signal and an infrared signal, and a second signal processing unit that processes a signal from the second imaging element and outputs a visible light signal and an infrared signal; A control unit; Equipped with the control unit performs control so as to add or multiplex third data to at least one of first data based on one of the visible light signal output from the first signal processing unit and the infrared signal acquired at the same timing as the visible light signal, and second data based on one of the visible light signal output from the second signal processing unit and the infrared signal acquired at the same timing as the visible light signal, which corresponds to the first data, and transmit the third data to an outside. Imaging device.

2. The control unit further adds or multiplexes fourth data based on the other of the visible light signal output from the first signal processing unit and the infrared signal acquired at the same timing as the visible light signal, and fifth data based on the visible light signal output from the second signal processing unit and the infrared signal acquired at the same timing as the visible light signal, which corresponds to the fourth data, to at least one of the first data and the second data, and transmits the data to the outside. The imaging device according to claim 1 .

Images