Region extraction apparatus and method, and object detection apparatus and method

Abstract

Provided are a region extraction device and method for a pre-stage of target object detection that can appropriately extract a region in which a target object can exist in an imaging region as a determination region, and an object detection device and method that can effectively detect a target object by using a region extraction result, the target object emitting light having a specific narrow wavelength band. The region extraction method acquires a plurality of images including an image of a second narrow wavelength band corresponding to a first narrow wavelength band of light emitted by a target object and an image of a third narrow wavelength band different from the second narrow wavelength band from a multi-spectral camera, step (S100). Next, based on the acquired plurality of images, a region in the imaging region that emits light having a light emission spectrum other than the first narrow wavelength band is determined as a non-determination region, and one or more regions excluding the non-determination region from the imaging region are extracted as determination regions, steps (S110) to (S130).

Description

Technical Field

[0001] This invention relates to a region extraction device and method, and an object detection device and method, and more particularly to a technique for extracting a region in a camera area where a target object may exist as a determination region, and a technique for using the extraction results to detect the target object. Background Technology

[0002] The requirement is to detect only target objects that emit light with a specific wavelength.

[0003] For example, cameras and image recognition technologies capable of automatically recognizing LED (Light Emitting Diode) traffic lights are expected to become surveillance cameras at intersections and in-vehicle cameras for autonomous vehicles.

[0004] Patent Document 1 describes a monitoring camera device for detecting blue, yellow, and red traffic lights made of LEDs.

[0005] The surveillance camera described in Patent Document 1 includes: a traffic light detection mechanism for detecting the illumination of traffic lights composed of LEDs of traffic lights within the imaging area of ​​a solid-state imaging element; an on / off area detection mechanism for detecting on / off areas of each traffic light that are periodically and repeatedly on and off; and an RGB ratio detection mechanism for detecting the ratio of the RGB signals of the on / off areas output from the on / off area detection mechanism. The region within the imaging area, represented by the xy coordinates of three pixels connected in series horizontally or vertically and periodically and repeatedly on and off, is defined as a region containing red, yellow, and blue traffic lights. The ratio of the red (R), green (G), and blue (B) color signals (RGB signals) of the solid-state imaging element in the three on / off areas determines the red, yellow, and blue traffic lights.

[0006] Furthermore, Patent Document 2 describes a technique for distinguishing genuine banknotes, securities, and other counterfeit goods.

[0007] In genuine banknotes and the like, there exists a special ink that radiates (reflects) light in a specific narrow band. Regarding the method described in Patent Document 2, radiation within a first band containing the specific narrow band and other wavelengths is detected, as well as radiation within a second band that essentially contains only other wavelengths. The presence of radiation in the specific narrow band (the presence of the special ink) is determined by comparing the levels of the two detected radiations.

[0008] Previous technical documents

[0009] Patent documents

[0010] Patent Document 1: Japanese Patent Application Publication No. 2014-32436

[0011] Patent Document 2: Japanese Patent Publication No. 2001-516012 Summary of the Invention

[0012] The technical problem to be solved by the invention

[0013] The surveillance camera described in Patent Document 1 uses the sequential lighting / extinguishing of blue, yellow, and red traffic lights to detect areas where blue, yellow, and red traffic lights are present. The invention described in Patent Document 1 is not limited to the target object being detected, such as an object existing in the same position within the camera area, like a traffic light captured by the surveillance camera.

[0014] Furthermore, the method described in Patent Document 2 is a method for screening genuine and counterfeit banknotes, securities, etc. In particular, the target object to be detected is a special ink used on genuine banknotes, etc., that emits light in a specific narrow band, and the location where the special ink should be present is limited.

[0015] That is, the technologies described in Patent Documents 1 and 2 are all based on the premise that the target object exists in a specific position in the camera area, and cannot detect the target object that exists in any position in the camera area.

[0016] The present invention was made in view of this situation, and its object is to provide a region extraction apparatus and method for the pre-stage of target object detection, which can appropriately extract the region of the imaging area that may contain a target object emitting light with a specific narrow band of emission spectrum as a determination region, and an object detection apparatus and method that can effectively detect the target object by using the region extraction result.

[0017] means for solving technical problems

[0018] To achieve the above objective, the invention involved in the first aspect is a region extraction device, which includes a processor that extracts regions in a camera area where a target object may exist as determination regions. The target object emits light with a first narrow band emission spectrum. The processor performs the following processes: image acquisition processing, acquiring multiple images from a multispectral camera, including an image of a second narrow band corresponding to the first narrow band and an image of a third narrow band different from the second narrow band; non-determination region determination processing, detecting regions in the camera area that emit light with emission spectra other than the first narrow band emission spectrum based on images other than the second narrow band image from the multiple images, and determining the detected regions as non-determination regions; and determination region extraction processing, extracting one or more regions from the camera area after removing the non-determination regions as determination regions.

[0019] According to a first aspect of the invention, regions emitting or reflecting light with the emission spectrum of the first narrow band (i.e., regions without the emission spectrum of the first narrow band emitting or reflecting light from or reflecting light from the target object) are detected in the imaging area based on images other than the image of the second narrow band from a plurality of images. In these detected regions, light with an emission spectrum different from the emission spectrum of the target object is present at least, therefore the target object is not present. Therefore, regions where the target object is not present are considered non-determination regions and excluded from the regions where the target object might exist. That is, the remaining regions after excluding the non-determination regions from the imaging area are extracted as regions where the target object might exist (determination regions).

[0020] As a preprocessing step for target object detection, the detection performance of target objects is "improved" by narrowing down the determination area where target objects may exist.

[0021] Here, “improvement” includes the following two aspects of performance improvement: (a) “pseudo-objects” in non-determination areas are originally outside the determination objects, so no false detection will occur (suppressing false detection); (b) the determination areas in the camera area are reduced (strictly filtered), so the processing time for determining the target object is reduced (shortening processing time).

[0022] In the region extraction apparatus according to the second aspect of the present invention, it is preferable that the center wavelength of the second narrow band is within a range that is less than half the width of the emission spectrum of the first narrow band from the center wavelength of the first narrow band, and the center wavelength of the third narrow band is more than half the width of the emission spectrum of the first narrow band from the center wavelength of the second narrow band.

[0023] In the region extraction apparatus according to the third aspect of the present invention, it is preferable that the center wavelength of the second narrow band is the same as the center wavelength of the first narrow band, and the bandwidth of the second narrow band is within the bandwidth of the first narrow band. The fact that the center wavelength of the second narrow band is the same as the center wavelength of the first narrow band is not limited to the case of being completely the same, but also includes the case of being approximately the same.

[0024] In the area extraction device according to the fourth aspect of the present invention, it is preferable that the target object produces a strobe light at a frequency corresponding to the frequency of the commercial power supply, and the non-determination area determination process sets the area that does not produce strobe light as a non-determination area in the camera area.

[0025] In the region extraction apparatus according to the fifth aspect of the present invention, it is preferable that the non-determined region determination process defines the region below the lower third of the image area as a non-determined region. For example, this is because when using a vehicle-mounted multispectral camera to photograph the front, traffic lights are generally not present in the region below the lower third of the image area of ​​the vehicle-mounted camera.

[0026] In the area extraction apparatus according to the sixth aspect of the present invention, the target object is preferably a light-emitting body having a light-emitting diode. For example, a traffic light composed of a light-emitting diode can be designated as the target object.

[0027] Regarding the object detection apparatus according to the seventh aspect of the present invention, it includes the above-described region extraction apparatus, and the processor performs the following determination process, that is, based on the images of multiple narrow bands including the image of the second narrow band and the image of the third narrow band, it determines whether the object in the determination area is the target object.

[0028] In the object detection apparatus according to the eighth aspect of the present invention, the image acquisition process preferably acquires an image of a second narrow band, an image of a third narrow band, and an image of a fourth narrow band sandwiched between the second narrow band and located on the opposite side of the third narrow band from a multispectral camera. The determination process subtracts the images of the third and fourth narrow bands from the image of the second narrow band within the determination area, and determines whether the object within the determination area is a target object based on the subtraction result.

[0029] According to the eighth aspect of the present invention, by subtracting the images of two adjacent third and fourth narrow bands from the image of the second narrow band, it is possible to detect only the target object emitting light of the emission spectrum of the first narrow band.

[0030] In the object detection apparatus according to the ninth aspect of the present invention, the image acquisition process acquires an image of a second narrow band, an image of a third narrow band, and an image of a fourth narrow band sandwiched between the second narrow band and located on the opposite side of the third narrow band from a multispectral camera. The determination process performs a product-sum operation on the images of the second narrow band, the third narrow band, and the fourth narrow band within the determination area, and on the weight coefficient set for each image. Based on the result of the product-sum operation, it determines whether the object within the determination area is a target object.

[0031] According to the ninth aspect of the present invention, by using three images—the second narrow band image, the third narrow band image, and the fourth narrow band image—within the determination region and performing a product-sum operation with a weighting coefficient set for each image, it is possible to more effectively detect only the target object emitting light in the first narrow band emission spectrum.

[0032] In the object detection apparatus according to the tenth aspect of the present invention, the image acquisition process acquires an image of a second narrow band, an image of a third narrow band, and an image of a fourth narrow band sandwiched between the second narrow band and located on the opposite side of the third narrow band from a multispectral camera. The determination process performs a product sum operation on the images of the second narrow band, the third narrow band, and the fourth narrow band within the determination area and a weight coefficient set for each image. The result of the product sum operation is further subjected to a nonlinear operation. Based on the result of the nonlinear operation, it is determined whether the object within the determination area is a target object.

[0033] According to the tenth aspect of the present invention, by using three images—the second narrow band image, the third narrow band image, and the fourth narrow band image—within the determination region and a weighting coefficient set for each image, and further performing a nonlinear operation on the result of the multiplication operation, it is possible to further and better detect only the target object emitting light of the first narrow band emission spectrum.

[0034] In the object detection device according to the 11th aspect of the present invention, the preferred image acquisition process acquires images of the second narrow band, the third narrow band, and the fourth narrow band sandwiched between the second narrow band and located on the opposite side of the third narrow band from a multispectral camera. The determination process is based on a learned model, wherein the learned model inputs images of the second narrow band, the third narrow band, and the fourth narrow band within the determination area, and outputs a determination result as to whether an object within the determination area is a target object.

[0035] In the object detection device according to the 12th aspect of the present invention, the target object is preferably the blue, yellow and red traffic lights. The image acquisition process acquires multiple narrow-band images from a multispectral camera, including images of three narrow-bands corresponding to the emission spectra of the light emitted by the blue, yellow and red traffic lights, and images of three or more narrow-bands different from the three narrow-bands. The processor detects which traffic light is emitting light based on the images of six or more narrow-bands.

[0036] The invention involved in the 13th aspect is a processor-based region extraction method for extracting regions in a camera area where a target object may exist as determination regions, wherein the target object emits light with a first narrow band emission spectrum. The region extraction method includes the following steps: acquiring multiple images from a multispectral camera, including an image of a second narrow band corresponding to the first narrow band and an image of a third narrow band different from the second narrow band; detecting regions in the camera area that emit light with a emission spectrum other than the first narrow band emission spectrum based on the images other than the second narrow band in the multiple images, and determining the detected regions as non-determination regions; and extracting one or more regions from the camera area after removing the non-determination regions as determination regions.

[0037] The object detection method according to the 14th aspect of the present invention includes the region extraction method described above. The processor determines whether an object within the determination region is a target object based on multiple narrow band images including an image of a second narrow band and an image of a third narrow band.

[0038] Invention Effects

[0039] According to the present invention, a region in the imaging area that may contain a target object emitting light with a specific narrow wavelength spectrum can be appropriately extracted as a determination region. Furthermore, by performing this determination region extraction process as a preprocessing step for target object detection, target objects emitting light with a specific narrow wavelength spectrum can be effectively detected in the imaging area. Attached Figure Description

[0040] Figure 1 This is a schematic structural diagram illustrating an embodiment of the object detection device according to the present invention.

[0041] Figure 2 This is a schematic diagram illustrating an implementation of a multispectral camera.

[0042] Figure 3 This is the front view of the pupil splitting filter.

[0043] Figure 4 This is the front view of a narrowband filter.

[0044] Figure 5 This diagram shows an example of the transmission wavelength set in the narrowband filter section.

[0045] Figure 6 This is the front view of the polarization filter.

[0046] Figure 7 This diagram shows an example of the polarization direction set in each polarization filter section of a polarization filter.

[0047] Figure 8 It is a diagram showing the approximate structure of the pixel arrangement of an image sensor.

[0048] Figure 9 This is a diagram showing the general structure of an image sensor.

[0049] Figure 10 It means to Figure 9 The dashed lines indicate a cross-sectional view of the approximate structure of the pixels.

[0050] Figure 11 This is a diagram showing an example of the arrangement pattern of polarization filter elements set in each pixel block.

[0051] Figure 12 This is a diagram showing an example of the arrangement pattern of spectral filter elements set in each pixel block.

[0052] Figure 13 This is a main block diagram of the signal processing unit of a multispectral camera.

[0053] Figure 14 This is a schematic diagram illustrating the implementation of the processor.

[0054] Figure 15 This is a conceptual diagram illustrating an implementation method based on the decision processing of each decision processing unit.

[0055] Figure 16 This is a conceptual diagram representing other variations of the decision processing unit.

[0056] Figure 17 This is a conceptual diagram showing a modified example of a multispectral camera and a determination processing unit.

[0057] Figure 18 This is a flowchart illustrating an implementation method of the object detection method according to the present invention.

[0058] Figure 19 These are diagrams representing various images, etc., during the detection process of the target object based on the present invention.

[0059] Figure 20 This is a flowchart illustrating a comparative example relative to the object detection method involved in this invention.

[0060] Figure 21 It is a graph representing various images, etc., during the detection process of target objects based on comparison examples. Detailed Implementation

[0061] Hereinafter, with reference to the accompanying drawings, preferred embodiments of the region extraction apparatus and method, and the object detection apparatus and method of the present invention will be described.

[0062] Figure 1 This is a schematic structural diagram illustrating an embodiment of the object detection device according to the present invention.

[0063] Figure 1 The object detection device 1 shown is configured to include the area extraction device involved in this invention.

[0064] like Figure 1 As shown, the object detection device 1 consists of a multispectral camera 10 and a processor 100. The object detection device 1 is a device for detecting target objects that emit or reflect light at a specific narrow wavelength. In this example, the blue, yellow, and red traffic lights, which are composed of light-emitting diodes (LEDs), are used as the target objects to be detected.

[0065] The blue, yellow, and red LEDs used in traffic lights have light emission center wavelengths of 503nm, 592nm, and 630nm, respectively, emitting light with a narrow wavelength spectrum of approximately 30-50nm.

[0066] The multispectral camera 10 has multiple narrow-band filters that selectively transmit light from multiple narrow-band light sources, including blue LEDs, yellow LEDs, and red LEDs, and simultaneously captures multiple images that transmit light from the multiple narrow-band filters.

[0067] Multispectral camera

[0068] Figure 2 This is a schematic diagram illustrating an implementation of a multispectral camera.

[0069] Figure 2 The multispectral camera 10 shown includes an imaging optical system 11, an image sensor 20, and a signal processing unit 30.

[0070] [Image capturing optical system]

[0071] The imaging optical system 11 includes a lens 12 and a pupil splitting filter 14. The lens 12 images an optical image of the subject containing the target object onto the light-receiving surface of the image sensor 20.

[0072] The pupil segmentation filter 14 is disposed at or near the pupil position of the imaging optical system 11, dividing the pupil portion of the imaging optical system 11 into 9 optical regions. The pupil segmentation filter 14 is constructed by overlapping the narrow-band filter 16 and the polarization filter 18.

[0073] Figure 3 This is the front view of the pupil splitting filter.

[0074] like Figure 3 As shown, the pupil segmentation filter 14 has nine optical regions Sj (j = 1, 2, 3, 4, 5, 6, 7, 8, 9) that are divided into nine equal parts in the circumferential direction. Hereinafter, as needed, the optical region marked S1 is designated as the first optical region S1, the optical region marked S2 as the second optical region S2, the optical region marked S3 as the third optical region S3, the optical region marked S4 as the fourth optical region S4, the optical region marked S5 as the fifth optical region S5, the optical region marked S6 as the sixth optical region S6, the optical region marked S7 as the seventh optical region S7, the optical region marked S8 as the eighth optical region S8, and the optical region marked S9 as the ninth optical region S9 to distinguish the nine optical regions Sj.

[0075] Each optical region Sj is configured to transmit light of different wavelengths. Furthermore, among the nine optical regions Sj, the optical region groups consisting of the first optical region S1, the second optical region S2, and the third optical region S3; the optical region groups consisting of the fourth optical region S4, the fifth optical region S5, and the sixth optical region S6; and the optical region groups consisting of the seventh optical region S7, the eighth optical region S8, and the ninth optical region S9 are configured to transmit light with different polarization directions (transmission polarization orientations). This structure is achieved by combining a narrow-band filter 16 and a polarization filter 18 with the following structures.

[0076] Figure 4 This is the front view of a narrowband filter.

[0077] The narrowband filter 16 has nine narrowband filter sections F1 to F9, which are divided into nine equal parts in the circumferential direction. Hereinafter, as needed, the narrowband filter section marked F1 is designated as the first narrowband filter section F1, the narrowband filter section marked F2 is designated as the second narrowband filter section F2, the narrowband filter section marked F3 is designated as the third narrowband filter section F3, the narrowband filter section marked F4 is designated as the fourth narrowband filter section F4, the narrowband filter section marked F5 is designated as the fifth narrowband filter section F5, the narrowband filter section marked F6 is designated as the sixth narrowband filter section F6, the narrowband filter section marked F7 is designated as the seventh narrowband filter section F7, the narrowband filter section marked F8 is designated as the eighth narrowband filter section F8, and the narrowband filter section marked F9 is designated as the ninth narrowband filter section F9 to distinguish the nine narrowband filter sections F1 to F9.

[0078] Each narrowband filter section F1 to F9 corresponds to each optical region S1 to S9 of the pupil-splitting filter 14. That is, the first narrowband filter section F1 corresponds to the first optical region S1, the second narrowband filter section F2 corresponds to the second optical region S2, the third narrowband filter section F3 corresponds to the third optical region S3, the fourth narrowband filter section F4 corresponds to the fourth optical region S4, the fifth narrowband filter section F5 corresponds to the fifth optical region S5, the sixth narrowband filter section F6 corresponds to the sixth optical region S6, the seventh narrowband filter section F7 corresponds to the seventh optical region S7, the eighth narrowband filter section F8 corresponds to the eighth optical region S8, and the ninth narrowband filter section F9 corresponds to the ninth optical region S9.

[0079] Each narrowband filter section F1 to F9 is composed of a bandpass filter that allows light to pass through different narrowbands.

[0080] Figure 5 This diagram shows an example of the transmission wavelength set in each narrowband filter section.

[0081] The wavelength of light transmitted through the first narrow-band filter unit F1 is designated as band 1 Δf1, the wavelength of light transmitted through the second narrow-band filter unit F2 is designated as band 2 Δf2, the wavelength of light transmitted through the third narrow-band filter unit F3 is designated as band 3 Δf3, the wavelength of light transmitted through the fourth narrow-band filter unit F4 is designated as band 4 Δf4, the wavelength of light transmitted through the fifth narrow-band filter unit F5 is designated as band 5 Δf5, the wavelength of light transmitted through the sixth narrow-band filter unit F6 is designated as band 6 Δf6, the wavelength of light transmitted through the seventh narrow-band filter unit F7 is designated as band 7 Δf7, the wavelength of light transmitted through the eighth narrow-band filter unit F8 is designated as band 8 Δf8, and the wavelength of light transmitted through the ninth narrow-band filter unit F9 is designated as band 9 Δf9.

[0082] Preferably, the second band Δf2 is a band corresponding to the band of the blue LED, with the center wavelength of the second band Δf2 being the same as the center wavelength of the blue LED (503nm), and the bandwidth of the second band Δf2 being within the bandwidth of the blue LED band. Furthermore, the center wavelength of the second band Δf2 is not limited to being exactly the same as the center wavelength of the blue LED (503nm), but includes cases where it is approximately the same; at least it needs to be within a range of less than half the width of the blue LED's emission spectrum from the center wavelength of the blue LED.

[0083] The center wavelength of band 1 Δf1 is 503nm-λ1, which is λ1 shorter than the center wavelength of band 2 Δf2 (the center wavelength of the blue LED (503nm)). The center wavelength of band 3 Δf3 is 503nm+λ3, which is λ3 longer than the center wavelength of band 2 Δf2. That is, band 3 Δf3 is different from bands 1 Δf1 and 2 Δf2; it is a band sandwiched between band 2 Δf2 and located on the opposite side from band 1 Δf1.

[0084] Furthermore, λ1 and λ3 are preferably values ​​greater than half the width of the emission spectrum of the blue LED from the center wavelength of the second band Δf2.

[0085] Preferably, the fifth band Δf5 is the band corresponding to the band of the yellow LED, the center wavelength of the fifth band Δf5 is consistent with the center wavelength (592nm) of the yellow LED, and the bandwidth of the fifth band Δf5 is within the bandwidth of the yellow LED band.

[0086] Band 4 (Δf4) and band 6 (Δf6) are the bands preceding and following band 5 (Δf5), respectively. The center wavelength of band 4 (Δf4) is 592 nm - λ4, and the center wavelength of band 6 (Δf6) is 592 nm + λ6. Preferably, λ4 and λ6 are values ​​greater than or equal to half the width of the emission spectrum of the yellow LED from the center wavelength of band 5 (Δf5).

[0087] Preferably, the 8th band Δf8 is a band corresponding to the band of the red LED, the center wavelength of the 8th band Δf8 is consistent with the center wavelength of the red LED (630nm), and the bandwidth of the 8th band Δf8 is within the bandwidth of the red LED band.

[0088] Band 7 (Δf7) and band 9 (Δf9) are the bands preceding and following band 8 (Δf8), respectively. The center wavelength of band 7 (Δf7) is 630 nm - λ7, and the center wavelength of band 9 (Δf9) is 630 nm + λ9. Preferably, λ7 and λ9 are values ​​greater than or equal to half the width of the emission spectrum of the red LED from the center wavelength of band 8 (Δf8).

[0089] Figure 6 This is the front view of the polarization filter.

[0090] The polarization filter 18 has three polarization filter sections G1 to G3, which are divided into three equal parts in the circumferential direction. Hereinafter, the polarization filter section marked G1 is designated as the first polarization filter section G1, the polarization filter section marked G2 as the second polarization filter section G2, and the polarization filter section marked G3 as the third polarization filter section G3, to distinguish the three polarization filter sections G1 to G3. The first polarization filter section G1 corresponds to the first optical region S1 to the third optical region S3 of the pupil segmentation filter 14, the second polarization filter section G2 corresponds to the fourth optical region S4 to the sixth optical region S6 of the pupil segmentation filter 14, and the third polarization filter section G3 corresponds to the seventh optical region S7 to the ninth optical region S9 of the pupil segmentation filter 14.

[0091] Figure 7 This is a diagram showing an example of the polarization direction of each polarization filter section set in the polarization filter.

[0092] The polarization direction (transmission polarization orientation) is represented by the angle (azimuth angle) between the polarization transmission axis and the X-axis on the XY plane orthogonal to the optical axis L. Figure 7 In the symbol Aa, the symbol Ab represents the polarization transmission axis of the first polarization filter section G1, the symbol Ac represents the polarization transmission axis of the second polarization filter section G2, and the symbol Ac represents the polarization transmission axis of the third polarization filter section G3.

[0093] like Figure 7As shown, each polarization filter section G1 to G3 has a structure that transmits light with different polarization directions (transmission polarization orientation). The polarization direction (transmission polarization orientation) of the light transmitted through the first polarization filter section G1 is set as α1, the polarization direction (transmission polarization orientation) of the light transmitted through the second polarization filter section G2 is set as α2, and the polarization direction (transmission polarization orientation) of the light transmitted through the third polarization filter section G3 is set as α3. In the multispectral camera 10 of this embodiment, the first polarization filter section G1 is set to transmit light with an azimuth angle of 0° (α1 = 0°), the second polarization filter section G2 is set to transmit light with an azimuth angle of 60° (α2 = 60°), and the third polarization filter section G3 is set to transmit light with an azimuth angle of 120° (α3 = 120°).

[0094] The pupil-splitting filter 14 is constructed by superimposing the narrow-band filter 16 and the polarization filter 18 above on the same axis. The pupil-splitting filter 14 functions as follows.

[0095] Light passing through the first optical region S1 of the pupil-splitting filter 14 passes through the first narrow-band filter section F1 of the narrow-band filter 16 and the first polarization filter section G1 of the polarization filter 18. Therefore, light of the first band Δf1 is emitted from the first optical region S1 polarized along the polarization direction α1 (linear polarization). Light passing through the second optical region S2 of the pupil-splitting filter 14 passes through the second narrow-band filter section F2 of the narrow-band filter 16 and the first polarization filter section G1 of the polarization filter 18. Therefore, light of the second band Δf2 is emitted from the second optical region S2 polarized along the polarization direction α1 (linear polarization). Light passing through the third optical region S3 of the pupil-splitting filter 14 passes through the third narrow-band filter section F3 of the narrow-band filter 16 and the first polarization filter section G1 of the polarization filter 18. Therefore, light of the third band Δf3 is emitted from the third optical region S3 polarized along the polarization direction α1 (linear polarization).

[0096] Furthermore, light passing through the fourth optical region S4 of the pupil segmentation filter 14 passes through the fourth narrow-band filter section F4 of the narrow-band filter 16 and the second polarization filter section G2 of the polarization filter 18. Therefore, light of the fourth band Δf4 is emitted from the fourth optical region S4 along the polarization direction α2 (linear polarization). Light passing through the fifth optical region S5 of the pupil segmentation filter 14 passes through the fifth narrow-band filter section F5 of the narrow-band filter 16 and the second polarization filter section G2 of the polarization filter 18. Therefore, light of the fifth band Δf5 is emitted from the fifth optical region S5 along the polarization direction α2 (linear polarization). Light passing through the sixth optical region S6 of the pupil segmentation filter 14 passes through the sixth narrow-band filter section F6 of the narrow-band filter 16 and the second polarization filter section G2 of the polarization filter 18. Therefore, light of the sixth band Δf6 is emitted from the sixth optical region S6 along the polarization direction α2 (linear polarization).

[0097] Furthermore, light passing through the 7th optical region S7 of the pupil segmentation filter 14 passes through the 7th narrow-band filter section F7 of the narrow-band filter 16 and the 3rd polarization filter section G3 of the polarization filter 18. Therefore, light of the 7th band Δf7 is emitted from the 7th optical region S7 along the polarization direction d3 (linear polarization). Light passing through the 8th optical region S8 of the pupil segmentation filter 14 passes through the 8th narrow-band filter section F8 of the narrow-band filter 16 and the 3rd polarization filter section G3 of the polarization filter 18. Therefore, light of the 8th band Δf8 is emitted from the 8th optical region S8 along the polarization direction α3 (linear polarization). Light passing through the 9th optical region S9 of the pupil segmentation filter 14 passes through the 9th narrow-band filter section F9 of the narrow-band filter 16 and the 3rd polarization filter section G3 of the polarization filter 18. Therefore, light of the 9th band Δf9 is emitted from the 9th optical region S9 along the polarization direction α3 (linear polarization).

[0098] The camera optical system 11 is configured to move back and forth along the optical axis L as a whole. This allows for focusing.

[0099] [Image sensor]

[0100] Figure 8 It is a diagram showing the approximate structure of the pixel arrangement of an image sensor.

[0101] like Figure 8 As shown, the image sensor 20 has multiple pixels Pi (i = 1, 2, 3, 4, 5, 6, 7, 8, 9) on its light-receiving surface. The pixels Pi are arranged regularly at a predetermined interval along the horizontal direction (x direction) and the vertical direction (y direction).

[0102] The image sensor 20 of this embodiment consists of nine adjacent (3 x 3) pixels Pi forming a pixel block PB(x, y), which is arranged regularly along the horizontal (x-direction) and vertical (y-direction) directions. Hereinafter, as needed, the pixel symbol P1 is designated as pixel 1 P1, the pixel symbol P2 as pixel 2 P2, the pixel symbol P3 as pixel 3 P3, the pixel symbol P4 as pixel 4 P4, the pixel symbol P5 as pixel 5 P5, the pixel symbol P6 as pixel 6 P6, the pixel symbol P7 as pixel 7 P7, the pixel symbol P8 as pixel 8 P8, and the pixel symbol P9 as pixel 9 P9 to distinguish the nine pixels disposed in a pixel block PB(x, y). Each pixel Pi receives light with different characteristics.

[0103] Figure 9 This is a diagram showing the general structure of an image sensor. Figure 10 It represents a pixel ( Figure 9 A cross-sectional view of the approximate structure (the dashed part).

[0104] The image sensor 20 has a pixel array layer 21, a polarization filter element array layer 23, a spectral filter element array layer 25, and a microlens array layer 27. The pixel array layer 21, the polarization filter element array layer 23, the spectral filter element array layer 25, and the microlens array layer 27 are arranged sequentially from the image plane side to the object side.

[0105] The pixel array layer 21 is constructed by arranging multiple photodiodes 22 in a two-dimensional manner. Each photodiode 22 constitutes a pixel. The photodiodes 22 are arranged regularly along the horizontal direction (x-direction) and the vertical direction (y-direction).

[0106] The polarization filter element array layer 23 is constructed by arranging three types of polarization filter elements 24A, 24B, and 24C with different polarization directions (transmission polarization orientations) in a two-dimensional configuration. Hereinafter, as needed, the polarization filter element 24A will be designated as the first polarization filter element 24A, the polarization filter element 24B as the second polarization filter element 24B, and the polarization filter element 24C as the third polarization filter element 24C to distinguish the three types of polarization filter elements 24A, 24B, and 24C.

[0107] Each polarization filter element 24A, 24B, and 24C is arranged at the same interval as the pixel array layer 21, and is configured for each pixel. The polarization direction (transmission polarization orientation) of the light transmitted through the first polarization filter element 24A is set to β1, the polarization direction (transmission polarization orientation) of the light transmitted through the second polarization filter element 24B is set to β2, and the polarization direction (transmission polarization orientation) of the light transmitted through the third polarization filter element 24C is set to β3.

[0108] In the multispectral camera 10, the first polarization filter element 24A is set to transmit light with an azimuth angle of 0° (β1 = 0°), the second polarization filter element 24B is set to transmit light with an azimuth angle of 60° (β2 = 60°), and the third polarization filter element 24C is set to transmit light with an azimuth angle of 120° (β3 = 120°). Polarization filter elements 24A, 24B, and 24C are an example of the second optical filter.

[0109] In each pixel block PB(x,y), polarization filter elements 24A, 24B, and 24C are arranged in a regular pattern.

[0110] Figure 11 This is a diagram showing an example of the arrangement pattern of polarization filter elements set in each pixel block.

[0111] like Figure 11 As shown, in the multispectral camera 10 of this embodiment, a first polarization filter element 24A is provided on the first column of pixels in the pixel block, namely the first pixel P1, the fourth pixel P4 and the seventh pixel P7; a second polarization filter element 24B is provided on the second column of pixels in the pixel block, namely the second pixel P2, the fifth pixel P5 and the eighth pixel P8; and a third polarization filter element 24C is provided on the third column of pixels in the pixel block, namely the third pixel P3, the sixth pixel P6 and the ninth pixel P9.

[0112] The spectral filter element array layer 25 is constructed by arranging three types of spectral filter elements 26A, 26B, and 26C with different spectral transmittances in a two-dimensional manner. Hereinafter, as needed, the spectral filter element 26A is designated as the first spectral filter element 26A, the spectral filter element 26B is designated as the second spectral filter element 26B, and the spectral filter element 26C is designated as the third spectral filter element 26C to distinguish the three types of spectral filter elements 26A, 26B, and 26C. Each spectral filter element 26A, 26B, and 26C is arranged at the same interval as the photodiode 22 and is configured for each pixel.

[0113] Spectral filtering elements 26A, 26B, and 26C transmit light from each narrow-band filter section F1 to F9 of the transmission narrow-band filter 16 with different transmittances. In the multispectral camera 10, the first spectral filtering element 26A has the characteristic of transmitting more short-wavelength light in the visible light band, the second spectral filtering element 26B has the characteristic of transmitting more mid-wavelength light, and the third spectral filtering element 26C has the characteristic of transmitting more long-wavelength light. For example, spectral filtering elements 26A, 26B, and 26C can be color filters (B, G, R) provided in a general color image sensor.

[0114] Figure 12 This is a diagram showing an example of the arrangement pattern of spectral filter elements set in each pixel block.

[0115] like Figure 12 As shown, spectral filter elements 26A, 26B, and 26C are arranged regularly in each pixel block PB(x, y). In the multispectral camera 10 of this embodiment, a first spectral filter element 26A is provided on the first row of pixels, namely the first pixel P1, the second pixel P2, and the third pixel P3 within the pixel block; a second spectral filter element 26B is provided on the second row of pixels, namely the fourth pixel P4, the fifth pixel P5, and the sixth pixel P6 within the pixel block; and a third spectral filter element 26C is provided on the third row of pixels, namely the seventh pixel P7, the eighth pixel P8, and the ninth pixel P9 within the pixel block.

[0116] like Figure 9 As shown, the microlens array layer 27 is constructed by arranging a plurality of microlenses 28 in a two-dimensional manner. Each microlens 28 is arranged at the same spacing as the photodiode 22 and is configured for each pixel. The microlenses 28 are configured to efficiently focus light from the imaging optical system 11 onto the photodiode 22.

[0117] In the image sensor 20 configured as described above, each pixel Pi in each pixel block PB(x,y) receives light from the camera optical system 11 as follows.

[0118] Pixel 1 P1 receives light from the imaging optical system 11 via the first spectral filter element 26A and the first polarization filter element 24A. Pixel 2 P2 receives light from the imaging optical system 11 via the first spectral filter element 26A and the second polarization filter element 24B. Pixel 3 P3 receives light from the imaging optical system 11 via the first spectral filter element 26A and the third polarization filter element 24C.

[0119] Furthermore, pixel P4 receives light from the imaging optical system 11 via the second spectral filter element 26B and the first polarization filter element 24A. Pixel P5 receives light from the imaging optical system 11 via the second spectral filter element 26B and the second polarization filter element 24B. Furthermore, pixel P6 receives light from the imaging optical system 11 via the second spectral filter element 26B and the third polarization filter element 24C.

[0120] Furthermore, pixel P7 receives light from the camera optical system 11 via the third spectral filter element 26C and the first polarization filter element 24A. Pixel P8 receives light from the camera optical system 11 via the third spectral filter element 26C and the second polarization filter element 24B. Pixel P9 receives light from the camera optical system 11 via the third spectral filter element 26C and the third polarization filter element 24C.

[0121] Thus, each pixel Pi of pixel block PB(x,y) receives light with different characteristics through spectral filtering elements 26A, 26B, 26C and polarization filtering elements 24A, 24B, 24C with different combinations.

[0122] [Signal Processing Department]

[0123] Figure 13 This is a main block diagram of the signal processing unit of a multispectral camera.

[0124] like Figure 13 As shown, the signal processing unit 30 is a part that processes the image signal output from the image sensor 20 to generate image data acquired in each optical region Sj of the imaging optical system 11, and includes an analog signal processing unit 30A, an image generation unit 30B, and a coefficient storage unit 30C.

[0125] The image signal output from the image sensor 20 is applied to the analog signal processing unit 30A. The analog signal processing unit 30A includes a sample-and-hold circuit, a color separation circuit, and an AGC (Automatic Gain Control) circuit. The AGC circuit functions as a sensitivity adjustment unit, adjusting the gain of the amplifier that amplifies the input image signal to bring the signal level of the image signal into an appropriate range. The A / D converter converts the analog image signal output from the analog signal processing unit into a digital image signal. Furthermore, in the case where the image sensor 20 is a CMOS image sensor, the analog signal processing unit and the A / D converter are typically built into the CMOS image sensor.

[0126] like Figure 8 As shown, each pixel block PB(x,y) includes the 1st pixel P1, the 2nd pixel P2, the 3rd pixel P3, the 4th pixel P4, the 5th pixel P5, the 6th pixel P6, the 7th pixel P7, the 8th pixel P8, and the 9th pixel P9.

[0127] By separating and extracting the pixel signals of the first pixel P1, the second pixel P2, the third pixel P3, the fourth pixel P4, the fifth pixel P5, the sixth pixel P6, the seventh pixel P7, the eighth pixel P8, and the ninth pixel P9 from each pixel block PB(x,y), the image generation unit 30B generates nine image data D1 to D9.

[0128] However, interference (crosstalk) occurs in these nine image data D1 to D9. That is, light from each optical region Sj of the camera optical system 11 is incident on each pixel Pi, so the generated image becomes an image composed of images of each optical region Sj mixed in a predetermined ratio. Therefore, the image generation unit 30B removes the interference (crosstalk) by performing the following calculations.

[0129] Now, the pixel signal (signal value) obtained from the first pixel P1 of each pixel block PB(x,y) is set as x1, the pixel signal obtained from the second pixel P2 is set as x2, the pixel signal obtained from the third pixel P3 is set as x3, the pixel signal obtained from the fourth pixel P4 is set as x4, the pixel signal obtained from the fifth pixel P5 is set as x5, the pixel signal obtained from the sixth pixel P6 is set as x6, the pixel signal obtained from the seventh pixel P7 is set as x7, the pixel signal obtained from the eighth pixel P8 is set as x8, and the pixel signal obtained from the ninth pixel P9 is set as x9. Nine pixel signals x1 to x9 can be obtained from each pixel block PB(x,y). The image generation unit 30B calculates nine pixel signals X1 to X9 corresponding to each optical region S1 to S9 from these nine pixel signals x1 to x9 according to the following formula [Equation 2] using matrix A shown in [Equation 1], and removes interference.

[0130] [Formula 1]

[0131]

[0132] [Formula 2]

[0133]

[0134] In addition, pixel signal X1 is the pixel signal corresponding to the first optical region S1, pixel signal X2 is the pixel signal corresponding to the second optical region S2, pixel signal X3 is the pixel signal corresponding to the third optical region S3, pixel signal X4 is the pixel signal corresponding to the fourth optical region S4, pixel signal X5 is the pixel signal corresponding to the fifth optical region S5, pixel signal X6 is the pixel signal corresponding to the sixth optical region S6, pixel signal X7 is the pixel signal corresponding to the seventh optical region S7, pixel signal X8 is the pixel signal corresponding to the eighth optical region S8, and pixel signal X9 is the pixel signal corresponding to the ninth optical region S9.

[0135] Therefore, an image acquired in the first optical region S1 is generated from pixel signal X1, an image acquired in the second optical region S2 is generated from pixel signal X2, an image acquired in the third optical region S3 is generated from pixel signal X3, an image acquired in the fourth optical region S4 is generated from pixel signal X4, an image acquired in the fifth optical region S5 is generated from pixel signal X5, an image acquired in the sixth optical region S6 is generated from pixel signal X6, an image acquired in the seventh optical region S7 is generated from pixel signal X7, an image acquired in the eighth optical region S8 is generated from pixel signal X8, and an image acquired in the ninth optical region S9 is generated from pixel signal X9.

[0136] The reasons why interference can be removed according to the above [Equation 2] will be explained below.

[0137] Interference occurs because light from each optical region Sj mixes into each pixel Pi. Now, assuming that the proportion (interference amount (also called interference ratio)) of light incident on the j-th optical region Sj (j = 1 to 9) of the imaging optical system 11 and received by the i-th pixel Pi (i = 1 to 9) of each pixel block PB (x, y) is bij (i = 1 to 9, j = 1 to 9), then the following relationship holds between the pixel signal xi obtained by each pixel Pi of each pixel block PB (x, y) and the pixel signal Xj corresponding to each optical region Sj of the imaging optical system 11.

[0138] That is, regarding the pixel signal x1 obtained from the first pixel P1, the following formula holds ("*" is the symbol for multiplication operation).

[0139] [Formula 3]

[0140] b11*X1+b12*X2+b13*X3+b14*X4+b15*X5+b16*X6+b17*X7+b18*X8+b19*X9= x1

[0141] Similarly, the same formula as [Equation 3] holds for the pixel signals x2 to x9 obtained from the 2nd pixel P2 to the 9th pixel P9 respectively.

[0142] Furthermore, by solving the simultaneous equations consisting of nine equations for X1 to X9, the pixel signals of the original image (i.e., the pixel signals X1 to X9 corresponding to each optical region S1 to S9) can be obtained.

[0143] Here, the above simultaneous equations can be expressed by [Equation 5] using [Equation 4] matrix B.

[0144] [Formula 4]

[0145]

[0146] [Formula 5]

[0147]

[0148] The solution to the simultaneous equation consisting of 9 equations, namely X1 to X9, can be obtained by multiplying both sides of equation [Equation 5] by the inverse matrix B. -1 Use the formula [6] to calculate.

[0149] [Formula 6]

[0150]

[0151] Thus, the pixel signals X1 to X9 corresponding to each optical region S1 to S9 can be calculated from the signal values ​​(pixel signals) x1 to x9 of each pixel P1 to P9 based on the proportion of light incident on each optical region S1 to S9 of the imaging optical system 11 that is received by each pixel P1 to P9 of the pixel block PB(x,y).

[0152] Equation [2] will convert the inverse matrix B of Equation [6] into [Equation 2]. -1 Let A(B) be a matrix. -1 =A). Therefore, the elements aij of matrix A in [Equation 2] can be obtained by finding the inverse matrix B of matrix B. -1 To obtain the values, the elements bij (i = 1–9, j = 1–9) of matrix B represent the proportion (interference) of light incident on the j-th optical region Sj (j = 1–9) of the imaging optical system 11 being received by the i-th pixel Pi (i = 1–9) of each pixel block PB (x, y). When polarization-based transmittance is set as c and spectral transmittance is set as d, this proportion is calculated by the product of transmittance c and transmittance d.

[0153] Here, the polarization-based transmittance c is calculated by the square of the cosine of the angle difference between the transmitted polarization orientation of the light in the transmission optical region and the transmitted polarization orientation of the light received by the pixel.

[0154] Furthermore, the transmittance d based on spectral transmittance is determined according to the wavelength of light in the transmitted optical region and the spectral transmittance of the spectral filtering element set in the pixel.

[0155] For example, regarding the relationship between the j-th optical region Sj of the camera optical system 11 and the i-th pixel Pi of the image sensor 20, if the transmission polarization orientation of the light transmitted through the j-th optical region Sj is set as θj, and the transmission polarization orientation of the light received by the i-th pixel Pi is set as φi, then the polarization-based transmittance c is the square of the cosine (cos) of their angular difference (|θj-φi|). 2 We can use (|θj-φi|)) to calculate.

[0156] Furthermore, the transmittance d based on the spectral transmittance is calculated from the wavelength of the light transmitted through the j-th optical region Sj and the spectral transmittance of the spectral filtering elements 26A, 26B, and 26C disposed on the i-th pixel Pi. That is, it is calculated from the wavelength of the light transmitted through the j-th optical region Sj based on the spectral transmittance of the spectral filtering elements 26A, 26B, and 26C disposed on the i-th pixel Pi.

[0157] Figure 13 The coefficient storage unit 30C, as shown, stores each element aij of the 9x9 matrix A as a coefficient group. The image generation unit 30B obtains the coefficient group from the coefficient storage unit 30C, and calculates the pixel signals X1 to X9 corresponding to each optical region S1 to S9 of the camera optical system 11 from the pixel signals x1 to x9 obtained from each pixel P1 to P9 of each pixel block PR(x,y) according to the above-mentioned [Equation 2], thereby generating 9 image data for each optical region S1 to S9.

[0158] That is, the multispectral camera 10 in this example can simultaneously capture nine images, each transmitted through nine narrow-band filter sections F1 to F9, representing the first band Δf1 to the ninth band Δf9. The nine image data, representing the images of the first band Δf1 to the ninth band Δf9 simultaneously captured by the multispectral camera 10, are output to the processor 100.

[0159] <Processor>

[0160] Figure 14 This is a schematic diagram illustrating the implementation of the processor.

[0161] Figure 14The processor 100 shown includes a CPU (Central Processing Unit), dedicated circuitry for performing specific signal processing on image data, etc. It acquires nine image data points from the multispectral camera 10, ranging from band 1 Δf1 to band 9 Δf9. Based on these nine image data points, it functions as a region extraction device that extracts regions where target objects may exist as determination regions. Furthermore, it performs determination processing (target object detection processing) to determine whether an object within the extracted determination region is a target object.

[0162] The processor 100 includes a region extraction device 110, a first determination processing unit 120A, a second determination processing unit 120B, and a third determination processing unit 120C.

[0163] The region extraction device 110 performs image acquisition processing, acquiring multiple (nine) image data from the multispectral camera 10 in bands 1 to 9 (Δf9). Next, it performs region extraction processing as follows: based on image data from the nine acquired image data that are outside the bands of the emission spectrum of the light emitted by the target object, it detects regions in the camera area that emit light with emission spectra outside the emission spectrum of the light emitted by the target object, determines the detected regions as non-determination regions where the target object does not exist, and extracts one or more regions from the camera area after removing the non-determination regions as determination regions.

[0164] The target objects in this example are the blue, yellow, and red traffic lights made of LEDs.

[0165] The region extraction device 110 performs the following non-determination region determination process: when a lit blue signal light is detected, based on the 6 image data of the 4th band Δf4 to the 9th band Δf9 out of the 9 image data of the 1st band Δf1 to the 9th band Δf9 that do not contain or substantially do not contain the emission spectrum of the blue signal light, the region where no lit blue signal light is detected is determined as a non-determination region.

[0166] For example, for a pixel Pj in the camera area, six pixel data points representing the location of pixel Pj can be obtained from six image data points in nine image data points that do not contain or substantially do not contain the emission spectrum of the blue traffic light (bands 4 to 9, Δf9). However, if any one of the six pixel data points exceeds a threshold, pixel Pj is determined to be a pixel in a region (non-determination region) where the blue traffic light is not lit. By determining all pixels in the camera area, non-determination regions where the blue traffic light is not lit can be identified.

[0167] After determining the non-determination area where no lit blue traffic light exists, the area extraction device 110 extracts one or more areas from the camera area that are not non-determination areas as determination areas (first determination areas) where a lit blue traffic light may exist.

[0168] Similarly, when the region extraction device 110 determines the non-determination area where there is no lit yellow traffic light, it determines the non-determination area where there is no lit yellow traffic light based on five image data of the first band Δf1 to the third band Δf3, the eighth band Δf8 and the ninth band Δf9, which do not contain or substantially do not contain the emission spectrum of the yellow traffic light. After determining the non-determination area where there is no lit yellow traffic light, it extracts one or more areas from the camera area that are not the non-determination areas as the determination area (second determination area) where there may be lit yellow traffic light.

[0169] Furthermore, when the region extraction device 110 determines the non-determination area where there is no lit red traffic light, it determines the non-determination area where there is no lit red traffic light based on five image data of the first band Δf1 to the fifth band Δf5, which do not contain or substantially do not contain the emission spectrum of the red traffic light. After determining the non-determination area where there is no lit red traffic light, it extracts one or more areas from the camera area that are not the non-determination areas as the determination area (third determination area) where there may be lit red traffic light.

[0170] The region extraction device 110 outputs the first region information representing the first determination region extracted above to the first determination processing unit 120A, outputs the second region information representing the second determination region to the second determination processing unit 120B, and outputs the third region information representing the third determination region to the third determination processing unit 120C.

[0171] The first determination processing unit 120A, the second determination processing unit 120B, and the third determination processing unit 120C respectively determine whether the object in the first determination area, the object in the second determination area, and the object in the third determination area is the target object (the lit blue signal light, yellow signal light, and red signal light).

[0172] [Judgment and Processing Department]

[0173] Figure 15 This is a conceptual diagram illustrating an implementation method based on the decision processing of each decision processing unit.

[0174] The first determination processing unit 120A inputs three image data from the first band Δf1 to the third band Δf3 and information about the first region. Based on the three image data from the first band Δf1 to the third band Δf3 and information about the first region, it determines whether the object in the first determination region is the target object (the lit blue signal light).

[0175] Among the objects in the first determination area, it is not limited to lit blue signal lights, but also considers other light-emitting or reflective objects that emit blue light.

[0176] Other light-emitting or reflective bodies that emit blue light, even if they are blue, will not emit a narrow band of light with a center wavelength of 503nm and a wavelength width of about 30-50nm like a blue LED.

[0177] Therefore, the first determination processing unit 120A subtracts the image data of the first band Δf1 and the third band Δf3 from the image data of the second band Δf2, whose center wavelength is the same as the center wavelength of the blue LED. Based on the result of this subtraction, it determines whether the object in the first determination area is the target object (the blue signal light of the blue LED).

[0178] When the object in the first determination area is a blue signal light, the subtraction result will not decrease significantly. However, when other light-emitting or reflective objects, such as blue LEDs, do not have a light emission spectrum, the subtraction result will decrease significantly or become negative. Therefore, the first determination processing unit 120A can determine whether the object in the first determination area is a lit blue signal light based on the above subtraction result.

[0179] Similarly, the second determination processing unit 120B inputs three image data from the fourth band Δf4 to the sixth band Δf6 and the second region information, and determines whether the object in the second determination region is the target object (the lit yellow signal light) based on the three image data from the fourth band Δf4 to the sixth band Δf6 and the second region information.

[0180] Furthermore, the third determination processing unit 120C inputs three image data from the 7th band Δf7 to the 9th band Δf9 and the information of the third region, and determines whether the object in the third determination region is the target object (the lit red signal light) based on the three image data from the 7th band Δf7 to the 9th band Δf9 and the information of the third region.

[0181] According to the present invention, the region where no target object exists is determined as a non-determination region by the region extraction device 110. By performing preprocessing to extract the region from the camera area that is removed from the non-determination region as the determination region, the "pseudo-object" located in the non-determination region is originally outside the determination object, so false detection can be eliminated (false detection suppressed). Furthermore, since the determination region in the camera area is reduced (strictly screened), the processing time for the determination of the target object can be shortened.

[0182] Furthermore, in the determination process of target objects in the determination area, by using an image containing multiple narrow bands, including a second narrow band corresponding to the first narrow band and a third narrow band different from the second narrow band (in this example, the narrow bands before and after), the target object emitting light with the emission spectrum of the first narrow band emits light with the emission spectrum of the first narrow band, but excludes "false objects" emitting light with a band wider than the first narrow band, thereby enabling the detection of only the target object.

[0183] Information regarding the location of the target object (in this example, an illuminated traffic light) within the camera's field of view, detected by processor 100, and the color of the illuminated traffic light, is output to output destination 200. Output destination 200 may be used by monitoring systems, autonomous driving systems, or similar applications that utilize traffic light information.

[0184] Figure 16 This is a conceptual diagram representing other variations of the decision processing unit.

[0185] Figure 16 The first determination processing unit 120-1 shown determines whether an object in the first determination area is a blue traffic light by using five image data points from the first band Δf1 to the fifth band Δf5 within the first determination area. Specifically, the first determination processing unit 120-1 uses image data from the second band Δf2 (whose center wavelength is the same as the center wavelength of the blue LED), image data from the bands preceding and following it (i.e., the first band Δf1 and the third band Δf3), and image data from the fourth band Δf4 and the fifth band Δf5. It subtracts two or more image data points (four image data points from the first band Δf1, the third band Δf3 to the fifth band Δf5) from the image data of the second band Δf2. Based on the subtraction result, it performs nonlinear processing based on a nonlinear function (f) and determines whether the object in the first determination area is a blue traffic light based on the result of the nonlinear processing.

[0186] As a non-linear operation, for example, threshold-based binarization relative to the subtraction result can be considered.

[0187] Figure 16 The second determination processing unit 120-2, as shown, determines whether an object within the second determination area is a yellow traffic light. It inputs five image data points in the same manner as the first determination processing unit 120-1. Specifically, the second determination processing unit 120-2 uses image data from the fifth band Δf5 (whose center wavelength matches the center wavelength of the yellow LED), the preceding and following bands (the fourth band Δf4 and the sixth band Δf6), and the adjacent third band Δf3 and seventh band Δf7. It subtracts two or more image data points from the fifth band Δf5 image data (four image data points from the third, fourth, sixth, and seventh bands Δf7) and determines whether the object within the second determination area is a yellow traffic light based on the subtraction result.

[0188] Alternatively, a product summation operation can be performed using three or more image data (five image data from the third band Δf3 to the seventh band Δf7) containing image data of the fifth band Δf5 within the second determination region, and a weighting coefficient set for each image (each of the five image data). Based on the result of the product summation operation, it can be determined whether the object within the second determination region is a yellow traffic light.

[0189] In each determination and processing unit 120-1 and 120-2, not only are the images of adjacent bands simply subtracted from the image of the band corresponding to the target object, but the recognition accuracy of the target object is also improved by using images of more bands.

[0190] The third determination processing unit 120-3 determines whether the object in the third determination area is a red traffic light, and performs nonlinear processing (based on the learned model).

[0191] The learned model, for example, can be composed of a convolutional neural network, which is a machine learning model pre-processed using a learning dataset consisting of each learning image in multiple bands and its forward resolution data.

[0192] The image data of the third determination area, namely the image data of the 8th band Δf8 whose center wavelength is the same as the center wavelength of the red LED, and the image data of the bands before and after it, namely the image data of the 7th band Δf7 and the image data of the 9th band Δf9, are input into the third determination processing unit 120-3. The output is the recognition result indicating whether the object in the third determination area is a red traffic light.

[0193] For ease of explanation, the aforementioned determination processing units 120-1 to 120-3 determine whether the objects in the first determination area, the second determination area, and the third determination area are blue traffic lights, yellow traffic lights, and red traffic lights, respectively. However, it goes without saying that by changing the image data of multiple input bands, it is also possible to determine other colored traffic lights.

[0194] Figure 17 This is a conceptual diagram showing a modified example of a multispectral camera and a determination processing unit.

[0195] The multispectral camera 10 captures nine image data points from band 1 Δf1 to band 9 Δf9. However, as shown in Figure 17, it can also be a multispectral camera that omits band 6 Δf6 (center wavelength 592nm+λ6) and band 7 Δf7 (center wavelength 630nm-λ7) and captures seven image data points from band 1 Δf1 to band 5 Δf5, band 8 Δf8, and band 9 Δf9.

[0196] When capturing image data from the aforementioned seven bands, the second determination processing unit 120B-1, which determines whether an object within the second determination area is a yellow traffic light, uses three image data points—band 4 Δf4, band 5 Δf5, and band 8 Δf8—for determination processing. The second determination processing unit 120B-1, in the same manner as the second determination processing unit 120B, subtracts the image data of the preceding and following bands—band 4 Δf4 and band 8 Δf8—from the image data of band 5 Δf5, whose center wavelength matches the center wavelength of the yellow LED. Based on this subtraction result, it determines whether the object within the second determination area is the target object (the yellow traffic light with the yellow LED).

[0197] Furthermore, the third determination processing unit 120C-1, which determines whether the object in the third determination area is a red traffic light, uses three image data points—the 5th band Δf5, the 8th band Δf8, and the 9th band Δf9—for determination processing. The third determination processing unit 120C-1, in the same manner as the third determination processing unit 120C, subtracts the image data of the preceding and following bands—the 5th band Δf5 and the 9th band Δf9—from the image data of the 8th band Δf8, whose center wavelength is the same as the center wavelength of the red LED. Based on the subtraction result, it determines whether the object in the third determination area is the target object (the red traffic light of the red LED).

[0198] In addition, a multispectral camera is preferred to capture image data of 7 bands as described above. However, it is also possible to capture images of multiple narrow bands, including 3 narrow bands corresponding to the emission spectra of the light emitted by the blue, yellow and red traffic lights, and 3 or more narrow bands that are different from these 3 narrow bands. Furthermore, the processor can detect which traffic light is emitting light based on images of 6 or more narrow bands.

[0199] and, Figure 4 The area extraction device 110 shown includes a flicker detection unit 112, which can use the detection results of the flicker detection unit 112 in the extraction of the first determination area, the second determination area and the third determination area.

[0200] In recent years, red LEDs have become the most common LED technology for car brake lights. In this case, it's difficult to rule out red LED brake lights (false positives).

[0201] The flicker detection unit 112 is used to eliminate false objects such as brake lights with red LEDs.

[0202] The red LED brake lights of automobiles are powered by DC power and therefore emit light at a specified brightness level. On the other hand, since the red LEDs of traffic lights are powered by commercial power, they are affected by the frequency of the commercial power supply (50Hz or 60Hz), resulting in a flickering frequency corresponding to the frequency of the commercial power supply.

[0203] Therefore, the multispectral camera 10 captures image data (video image data) of each band at a frame rate (fps: frames per second) different from the flashing frequency of the red LED of a traffic light.

[0204] The flicker detection unit 112 detects areas (areas where flicker does not occur) where the brightness level does not change with consecutive frames based on the input video image data of each band.

[0205] The area extraction device 110 defines the area where there is no lit blue signal light and the area where no flickering occurs as the non-determination area, and extracts the area removed from the camera area as the first determination area.

[0206] Similarly, the area extraction device 110 sets areas where no yellow traffic lights are lit and areas where no strobe is produced as non-determination areas, sets areas removed from the camera area as non-determination areas as second determination areas, and sets areas where no red traffic lights are lit and areas where no strobe is produced as non-determination areas, and extracts areas removed from the camera area as third determination areas.

[0207] Therefore, the objects within each judgment area are not limited to objects that produce flashing. For example, when detecting a red signal light with a red LED, it is possible not to detect brake lights made of red LEDs from a car.

[0208] As a method for eliminating false objects such as red LED brake lights, the region extraction device 110 can designate the area below the lower third of the imaging area as a non-determination area. This is because, when the multispectral camera 10 is a vehicle-mounted camera, there are no traffic lights in the area below the lower third of the imaging area captured by the vehicle-mounted camera, and on the other hand, there are no vehicle brake lights or the like in the upper two-thirds of the imaging area. In this case, each determination area is not limited to the upper two-thirds of the imaging area, which can shorten the processing time.

[0209] [Object Detection Methods]

[0210] Figure 18 This is a flowchart illustrating an implementation method of the object detection method according to the present invention. Figure 19 These are diagrams illustrating various images, etc., during the detection process of the target object based on the present invention. Additionally, Figure 18 The object detection method shown includes a region extraction method that extracts regions that may contain the target object involved in this invention as determination regions.

[0211] exist Figure 18 In the process, multiple narrow-band images are acquired using a multispectral camera 10 (step S100).

[0212] Figure 19 (A) represents a broadband image of visible light. Figure 19 (B) represents 7 images for each narrow band. In this example, as... Figure 17 As shown, seven images were acquired, including band 1 Δf1 to band 5 Δf5, band 8 Δf8, and band 9 Δf9.

[0213] The processor 100 acquires multiple images of each narrow band from the multispectral camera 10 and performs the following processing steps.

[0214] In addition, for the sake of simplicity, we will explain the case where the red signal light with the lit red LED is set as the target object. However, the same process will be applied when the blue signal light with the lit blue LED and the yellow signal light with the lit yellow LED are set as the target objects.

[0215] The region extraction device 110 of the processor 100 detects regions without emission / reflection in the wavelength band of the target object based on multiple images (step S110). In this example, the narrow wavelength bands outside the wavelength band of the target object are band 1 Δf1 to band 5 Δf5. Therefore, in the five images of these narrow wavelength bands, if any pixel data exceeds the threshold, that pixel is not a pixel containing the target object. By judging all pixels in the imaging area, regions where the target object does not exist are detected within the imaging area.

[0216] Then, regions detected as areas where no target object exists are considered non-determination regions and excluded from the determination regions (step S120). The region extraction device 110 extracts only the remaining regions from the camera area after removing non-determination regions as determination regions (step S130).

[0217] Figure 19 (C) indicates the region to be determined as described above. Figure 19 In (C), the decision area (the third decision area) represented by the white rectangular frame indicates the area where a red signal light may be lit. In this example, four decision areas are extracted.

[0218] Next, the judgment processing department ( Figure 17 The third determination processing unit 120C-1 determines whether the object in the determination area is the target object (step S140). That is, if using Figure 17 As explained, the third determination processing unit 120C-1 subtracts the image data of the preceding and following bands, namely the fifth band Δf5 and the ninth band Δf9, from the image data of the eighth band Δf8, whose center wavelength is the same as the center wavelength of the red LED, and determines whether the object in the determination area is the target object (the red signal light of the red LED) based on the result of the subtraction.

[0219] When an object within the determination area is determined to be a target object ("Yes"), the object within the determination area is detected as a target object, and the detection result is output to the output destination 200 (step S150). On the other hand, when an object within the determination area is determined not to be a target object ("No"), the process proceeds to step S160.

[0220] In step S160, it is determined whether the above-mentioned determination process in all determination regions has ended. If the determination process in all determination regions has not ended ("No"), the process transitions to step S170, where the next determination region is set. When the next determination region is set, the process returns to step S140, and the processes of steps S140 to S160 are performed again for the next determination region.

[0221] When the above-mentioned determination process in all determination areas is completed in this manner, the target object detection based on the image acquired by one capture ends.

[0222] Figure 19 (D) is a diagram representing the determination region where a target object is determined to exist, and the target object within that determination region.

[0223] exist Figure 19 In (D), a white box represents the region where a target object is determined to exist, and white represents the target object within that region. In this example, such as Figure 19 As shown in (D), one target object (a red signal light with a lit red LED) is detected in the camera area at the position indicated by the white box.

[0224] According to the present invention, such as Figure 19 As indicated by the white box in (C), there may be a certain determination area of ​​the target object that is strictly screened, so the load in the determination processing unit is small, and the "pseudo-object" located in the non-determination area is originally outside the determination object, so false detection can be suppressed.

[0225] [Comparative Example]

[0226] Figure 20 Figure 21 is a flowchart showing a comparative example of the object detection method according to the present invention, and Figure 22 is a diagram showing various images, etc., in the process of detecting a target object based on the comparative example.

[0227] exist Figure 20 In step S200, a narrow-band image of the target object is captured. In the comparative example, the red signal light of the lit red LED is set as the target object, and the target object's band is the 8th band Δf8.

[0228] Figure 21 (A) represents a broadband image of visible light. Figure 21 (B) represents the image of the narrow band (band 8 Δf8). Figure 21 The image shown in (B) is a narrow-band image, but it includes multiple objects that emit / reflect light in other bands besides the narrow-band.

[0229] Next, based on the captured narrow-band image, regions of light emission / reflection are detected in the wavelength band of the target object (step S210). For example, in the narrow-band image, regions of pixel groups exceeding a threshold can be detected as regions of light emission / reflection in the wavelength band of the target object.

[0230] The region detected in step S210 is set as the determination region (step S220). Figure 21 (C) indicates the decision region set as described above. Figure 21 In (C), the area indicated by the white rectangular frame is the determination area. Figure 21 (C) shows the determination region ratio Figure 19 (C) shows a large number of determination areas. This is because non-determination areas where there is no clearly defined target object are not excluded. For example, areas of a white object also reflect the wavelength of light from the target object, and are therefore set as determination areas.

[0231] Next, it is determined whether the object in the determination area is the target object (step S230). This determination of whether the object is the target object can be performed through image recognition processing. For example, when the target object is a traffic light, the object's shape is circular or approximately circular. Therefore, if the shape of the object in the determination area matches the shape of the target object, it can be identified as the target object. Furthermore, if the shape near the determination area resembles the shape of a traffic light (e.g., when blue, yellow, and red lights are arranged), the detected object can be identified as a traffic light.

[0232] When it is determined that an object within the determination area is a target object ("Yes"), the object within the determination area is detected as a target object (step S240). On the other hand, when it is determined that an object within the determination area is not a target object ("No"), the process transitions to step S250.

[0233] In step S250, it is determined whether the above-mentioned determination process in all determination areas has ended. If the determination process in all determination areas has not ended (in the case of "No"), the process transitions to step S260, where the next determination area is set. After setting the next determination area, the process returns to step S230, and the processes of steps S230 to S250 are performed again for the next determination area.

[0234] When the above-mentioned determination process in all determination areas is completed in this manner, the target object detection based on the image acquired by one capture ends.

[0235] Figure 21 (D) is a diagram representing the determination region where a target object is determined to exist, and the target object within that determination region.

[0236] exist Figure 21In (D), a white box represents the region where a target object is determined to exist, and white represents the target object within that region. In this example, such as Figure 21 As shown in (D), two target objects were detected at the locations indicated by white boxes within the camera area.

[0237] According to the comparative example, multiple determination regions are set, thus increasing the processing load for identifying whether a target object exists in each determination region. Furthermore, within the determination regions, a large number of "pseudo-objects" exist, including narrow-band light that is generated / reflected by the target object and broadband light that generates / reflects white light, etc. As a result, the possibility of mistakenly detecting "pseudo-objects" as the target object increases. Additionally, based on the determination results obtained through the comparative example, such as... Figure 21 As shown in (D), the target object was detected in the approximate center of the camera area, but a "pseudo-object" was detected in the lower left area of ​​the camera area.

[0238] [other]

[0239] In this embodiment, the blue, yellow, and red traffic lights made of LEDs are designated as target objects. However, the target object is not limited to these; it can also be a single target object. For example, guide lights installed on airport runways, LED road rivets, other reflectors that only reflect narrow-band light can also be designated as target objects.

[0240] Furthermore, the number of narrow-band images captured simultaneously by the multispectral camera is not limited to this embodiment. A multispectral camera can be any camera that selectively transmits light through multiple narrow-band filters, each containing a second narrow-band corresponding to at least the band emitted by the target object (a first narrow-band) and a third narrow-band different from the second narrow-band, and simultaneously acquires multiple images transmitted through these multiple narrow-band filters. Moreover, when there is no interference between the multiple images simultaneously acquired from the image sensor of the multispectral camera, it is self-evident that interference removal processing is unnecessary.

[0241] Furthermore, in this embodiment, for example, the hardware structure of the processing unit that performs various processes such as CPU is as shown below, including various processors. These processors include common processors that execute software (programs) to perform the functions of various processing units, such as CPUs (Central Processing Units), FPGAs (Field Programmable Gate Arrays) and other processors whose circuit structure can be changed after manufacturing, such as Programmable Logic Devices (PLDs), and processors with circuit structures specifically designed for performing specific processes, such as Application Specific Integrated Circuits (ASICs), i.e., dedicated circuits.

[0242] A processing unit can be composed of one of these various processors, or it can be composed of two or more processors of the same or different types (e.g., multiple FPGAs or a combination of CPU and FPGA). Furthermore, a single processor can also constitute multiple processing units. As examples of a single processor constituting multiple processing units, firstly, as represented by a computer such as a client or server, a single processor is composed of a combination of one or more CPUs and software, and this processor performs the functions of multiple processing units. Secondly, as represented by a System-on-Chip (SoC), a processor is used to implement the overall system functions including multiple processing units using a single IC (Integrated Circuit) chip. Thus, various processing units are constructed using one or more of the aforementioned processors in a hardware structure.

[0243] Furthermore, more specifically, the hardware structure of these various processors is a circuit composed of combined semiconductor elements and other circuitry.

[0244] Furthermore, the present invention is not limited to the above-described embodiments, and it goes without saying that various modifications can be made without departing from the spirit of the present invention.

[0245] Symbol Explanation

[0246] 1-Object detection device, 10-Multispectral camera, 11-Imaging optical system, 12-Lens, 14-Pupil splitting filter, 16-Narrow band filter, 18-Polarization filter, 20-Image sensor, 21-Pixel array layer, 22-Photodiode, 23-Polarization filter element array layer, 24A-First polarization filter element, 24B-Second polarization filter element, 24C-Third polarization filter element, 25-Spectrum filter element array layer, 26-Spectrum filter element, 26A-First spectrum filter element, 26B-Second spectrum filter element, 26C- 3rd beam splitter filter element, 27-microlens array layer, 28-microlens, 30-signal processing unit, 30A-analog signal processing unit, 30B-image generation unit, 30C-coefficient storage unit, 100-processor, 110-region extraction device, 112-flicker detection unit, 120A, 120-1-first determination processing unit, 120B, 120B-1, 120-2-second determination processing unit, 120C, 120C-1, 120-3-third determination processing unit, 200-output destination, F1-first narrowband filter unit, F2-second narrowband filter unit, F3-third narrowband filter unit, F4-fourth narrowband filter unit, F5-fifth narrowband filter unit, F6-sixth narrowband filter unit. F7 - 7th narrowband filter section, F8 - 8th narrowband filter section, F9 - 9th narrowband filter section, G1 - 1st polarization filter section, G2 - 2nd polarization filter section, G3 - 3rd polarization filter section, L - optical axis, P1 - 1st pixel, P2 - 2nd pixel, P3 - 3rd pixel, P4 - 4th pixel, P5 - 5th pixel, P6 - 6th pixel, P7 - 7th pixel, P8 - 8th pixel, P9 - 9th pixel, PB - pixel block, Pi - i-th pixel, S1 - 1st optical region, S2 - 2nd optical region, S3 - 3rd optical region, S4 - 4th optical region, S5 - Optical region 5, S6-Optical region 6, S7-Optical region 7, S8-Optical region 8, S9-Optical region 9, S100-S170, S200-S260-Steps, X1-X9, x1-x9, Xj-Pixel signals, Δf1-Band 1, Δf2-Band 2, Δf3-Band 3, Δf4-Band 4, Δf5-Band 5, Δf6-Band 6, Δf7-Band 7, Δf8-Band 8, Δf9-Band 9.

Claims

1. A region extraction device, comprising a processor for extracting regions within a camera area where a target object may exist as determination regions, wherein, The target object emits light with a first narrow wavelength band of emission spectrum. The processor performs the following processing: Image acquisition processing involves acquiring multiple images from a multispectral camera, including an image of a second narrow band corresponding to the first narrow band and an image of a third narrow band different from the second narrow band. Non-determination region determination process: Based on the images other than the second narrow band in the plurality of images, detect regions in the imaging area that emit light with a light emission spectrum other than that of the first narrow band, and determine the detected regions as non-determination regions; and The determination region extraction process involves removing one or more non-determination regions from the camera area and extracting them as the determination region. The center wavelength of the second narrow band is within a range that is less than half the width of the emission spectrum of the first narrow band from the center wavelength of the first narrow band, and the center wavelength of the third narrow band is more than half the width of the emission spectrum of the first narrow band from the center wavelength of the second narrow band.

2. The region extraction device according to claim 1, wherein, The center wavelength of the second narrow band is the same as the center wavelength of the first narrow band, and the bandwidth of the second narrow band is within the bandwidth of the first narrow band.

3. The region extraction device according to claim 1 or 2, wherein, The target object produces a strobe effect at a frequency corresponding to that of commercial power supply. The non-determination area determination process defines the area in the camera area that does not produce the flicker as the non-determination area.

4. The region extraction device according to claim 1 or 2, wherein, The non-determination area determination process defines the area below the lower third of the camera area as the non-determination area.

5. The region extraction device according to claim 1 or 2, wherein, The target object is a light-emitting body with a light-emitting diode.

6. An object detection device comprising the region extraction device according to any one of claims 1 to 5 and the multispectral camera, The processor performs the following determination process: based on multiple narrow band images including the image of the second narrow band and the image of the third narrow band, it determines whether the object in the determination area is the target object.

7. The object detection device according to claim 6, wherein, The image acquisition process acquires images of the second narrow band, the third narrow band, and a fourth narrow band sandwiched between the second narrow band and located on the opposite side of the third narrow band from the multispectral camera. The determination process subtracts the images of the third and fourth narrow bands within the determination area from the image of the second narrow band within the determination area, and determines whether the object within the determination area is the target object based on the result of the subtraction.

8. The object detection device according to claim 6, wherein, The image acquisition process acquires images of the second narrow band, the third narrow band, and a fourth narrow band sandwiched between the second narrow band and located on the opposite side of the third narrow band from the multispectral camera. The determination process utilizes the images of the second narrow band, the third narrow band, and the fourth narrow band within the determination area, and performs a product-sum operation on the weight coefficients set for each image. Based on the result of the product-sum operation, it determines whether the object within the determination area is the target object.

9. The object detection device according to claim 6, wherein, The image acquisition process acquires images of the second narrow band, the third narrow band, and a fourth narrow band sandwiched between the second narrow band and located on the opposite side of the third narrow band from the multispectral camera. The determination process utilizes the images of the second narrow band, the third narrow band, and the fourth narrow band within the determination area, and performs a product-sum operation on the weight coefficient set for each image. The result of the product-sum operation is further subjected to a nonlinear operation. Based on the result of the nonlinear operation, it is determined whether the object within the determination area is the target object.

10. The object detection device according to claim 6, wherein, The image acquisition process acquires images of the second narrow band, the third narrow band, and a fourth narrow band sandwiched between the second narrow band and located on the opposite side of the third narrow band from the multispectral camera. The determination process is based on the learned model. The learned model takes into account the images of the second narrow band, the third narrow band, and the fourth narrow band within the determination region, and outputs the determination result of whether the object within the determination region is the target object.

11. The object detection device according to claim 6, wherein, The target objects are the blue, yellow, and red traffic lights. The image acquisition process acquires multiple narrow-band images from the multispectral camera, including images of three narrow-band light emission spectra corresponding to the emission spectra of the light emitted by the blue, yellow, and red traffic lights, and images of three or more narrow-band light emission spectra different from the three narrow-band light emission spectra. The processor detects which traffic light is emitting light based on images from more than six narrow bands.

12. A region extraction method for a processor that extracts regions within a camera area where target objects may exist, wherein... The target object emits light with a first narrow wavelength band of emission spectrum. The region extraction method includes the following steps: The step of acquiring multiple images from a multispectral camera, including an image of a second narrow band corresponding to the first narrow band and an image of a third narrow band different from the second narrow band; The step of detecting regions emitting light with emission spectra other than those of the first narrow band in the imaging area based on the images other than the images of the second narrow band, and determining the detected regions as non-determination regions; and The step of extracting one or more regions from the camera area that are not considered as the determination region. The center wavelength of the second narrow band is within a range that is less than half the width of the emission spectrum of the first narrow band from the center wavelength of the first narrow band, and the center wavelength of the third narrow band is more than half the width of the emission spectrum of the first narrow band from the center wavelength of the second narrow band.

13. An object detection method, comprising the region extraction method of claim 12, The processor determines whether an object within the determination area is the target object based on multiple narrow band images including the image of the second narrow band and the image of the third narrow band.

Images