Surveillance camera, image processing method, and program

Abstract

To improve the image quality of a subject and to improve the efficiency of observation work even when a phenomenon that degrades the image quality of the imaged subject occurs.SOLUTION: A monitoring camera includes an imaging element configured to have a plurality of pixels and acquiring, for each pixel, a blue component signal, a red component signal, and a green component signal corresponding to light from a subject incident through fog or haze, an optical filter disposed between the subject and the imaging element, which passes light of a specific wavelength having a wavelength equal to or greater than the specific wavelength among the light from the subject, and a processor that generates and outputs the captured image of the subject on the basis of one of the blue component signal, the red component signal, and the green component signal for each pixel corresponding to the light of the specific wavelength, and the blue component signal for each pixel corresponding to the light of the specific wavelength.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to a surveillance camera, an image processing method, and a program. [Background technology]

[0002] Patent Document 1 discloses an imaging device that includes an imaging element that captures an image of a subject, a switching device that can switch an optical filter inserted between the subject and the imaging element between a first filter that blocks infrared light and a second filter that blocks visible light, a determination unit that determines the presence or absence of a fluctuation phenomenon (e.g., a heat haze phenomenon) caused by the presence of an air layer in which atmospheres of different densities are mixed between the subject and the imaging element based on the captured image of the subject, and a control unit that controls the switching device to control switching of the optical filter according to the determination result. This reduces the influence of fluctuation phenomena such as the heat haze phenomenon included in the captured image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2012-90152 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the configuration of Patent Document 1, when it is determined that a flickering phenomenon such as a heat haze phenomenon is occurring, an optical filter that blocks visible light is uniformly used for the entire imaging surface, so the image of the captured subject becomes dark. In particular, when monitoring a distant subject (e.g., an airport, a distant part of a long-distance bridge, or a port) as viewed from a surveillance camera, if an optical filter that blocks visible light is used, the subject becomes dark overall, making it difficult to determine the subject's condition, and there is a possibility that the efficiency of the observation work will decrease. For this reason, it can be said that there is room for improvement over the conventional technology in terms of suppressing deterioration in the image quality of the captured subject when a phenomenon that deteriorates the image quality of the captured subject occurs (for example, not only the above-mentioned heat haze phenomenon, but also meteorological phenomena such as fog or mist).

[0005] The present disclosure has been devised in consideration of the above-mentioned conventional circumstances, and aims to provide a surveillance camera, an image processing method, and a program that improve the image quality of an image of a subject, thereby making observation work more efficient, even when a phenomenon occurs that degrades the image quality of the captured image of the subject. [Means for solving the problem]

[0006] The present disclosure provides an imaging device comprising: an imaging element configured with a plurality of pixels, which acquires, for each pixel, a blue component signal, a red component signal, and a green component signal corresponding to light from a subject incident through fog or mist; an optical filter disposed between the subject and the imaging element, which passes specific wavelength light having a wavelength equal to or greater than a specific wavelength among the light from the subject; a processor which generates and outputs an image of the subject based on any of the blue component signal, the red component signal, and the green component signal corresponding to the specific wavelength light for each pixel and the blue component signal corresponding to the specific wavelength light for each pixel; and a second processor which specifies a region of interest in an image captured by the imaging element, When the processor determines that the fog or haze is occurring in the specified region of interest, it selects, according to the degree of the fog or haze occurring in the region of interest, one of a blue component signal, a red component signal, and a green component signal corresponding to the specific wavelength light for each pixel, and a blue component signal corresponding to the specific wavelength light for each pixel. and provide surveillance cameras.

[0008] The present disclosure also provides an image processing method executed by a surveillance camera, comprising the steps of: passing, by an optical filter included in the surveillance camera, specific wavelength light having a wavelength equal to or greater than a specific wavelength among light from a subject that is incident through fog or mist; and acquiring, for each pixel, a blue component signal, a red component signal, and a green component signal corresponding to the specific wavelength light by an image sensor having a plurality of pixels included in the surveillance camera; identifying a region of interest in an image captured by the imaging device; generating and outputting a captured image of a subject based on any one of a blue component signal, a red component signal, and a green component signal for each pixel corresponding to the specific wavelength light and a blue component signal for each pixel corresponding to the specific wavelength light. and when it is determined that the fog or mist is occurring in the specified region of interest, selects one of a blue component signal, a red component signal, and a green component signal for each pixel corresponding to the specific wavelength light and a blue component signal for each pixel corresponding to the specific wavelength light according to the degree of the fog or mist occurring in the region of interest. The present invention provides an image processing method.

[0010] The present disclosure also provides a method for detecting a specific wavelength of light from a subject that is incident through fog or mist by an optical filter provided in the surveillance camera, the method comprising the steps of: acquiring, for each pixel, a blue component signal, a red component signal, and a green component signal corresponding to the specific wavelength of light by an image sensor having a plurality of pixels provided in the surveillance camera; identifying a region of interest in an image captured by the imaging device; generating and outputting a captured image of a subject based on any one of a blue component signal, a red component signal, and a green component signal for each pixel corresponding to the specific wavelength light, and a blue component signal for each pixel corresponding to the specific wavelength light; When it is determined that the fog or mist is occurring in the specified region of interest, one of a blue component signal, a red component signal, and a green component signal corresponding to the specific wavelength light for each pixel and a blue component signal corresponding to the specific wavelength light is selected according to the degree of the fog or mist occurring in the region of interest. , and provide programs.

[0012] These comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. Effect of the Invention

[0013] According to the present disclosure, even if a phenomenon occurs that deteriorates the quality of a captured image of a subject, the quality of the image of the subject can be improved to make observation work more efficient.

[0014] Further advantages and benefits of certain aspects of the present disclosure will become apparent from the specification and drawings, in which such advantages and / or benefits are provided by some of the embodiments and features described in the specification and drawings, respectively, but not necessarily all of which are provided to obtain one or more identical features. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a diagram showing an example of a system configuration of a surveillance camera system according to first and second embodiments. [Diagram 2] FIG. 1 is a block diagram showing an example of a hardware configuration of a surveillance camera according to first and second embodiments. [Diagram 3] FIG. 1 is a diagram showing an example of an outline of the operation of a surveillance camera according to the first embodiment; [Figure 4A] A diagram showing a situation where fog or haze occurs over almost the entire area of ​​the image. [Figure 4B] A diagram showing a foggy or hazy condition in the distant area of ​​interest in the image. [Diagram 5] FIG. 1 is a diagram showing an example of transition of an operational state of a surveillance camera according to the first embodiment; [Figure 6] FIG. 2 is a diagram showing a schematic example of a relationship between a spatial frequency and a first threshold value A and a second threshold value B. [Figure 7] 1 is a flowchart showing a chronological order of an example of a procedure for determining whether fog or mist is occurring in a surveillance camera according to the first embodiment. [Figure 8] 1 is a flowchart showing an example of an image processing operation procedure of a surveillance camera according to a first embodiment when fog or mist is occurring. [Figure 9] FIG. 9 is a diagram showing an example of an image generated by the image processing operation procedure of FIG. 8; [Figure 10] FIG. 13 is a diagram showing an example of an outline of the operation of a surveillance camera according to a second embodiment; [Figure 11]11 is a flowchart showing an example of an image processing operation procedure of a surveillance camera according to a second embodiment when fog or mist is occurring. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, with reference to the drawings as appropriate, a detailed description will be given of an embodiment that specifically discloses a surveillance camera, an image processing method, and a program according to the present disclosure. However, more detailed description than necessary may be omitted. For example, detailed description of already well-known matters or duplicate description of substantially the same configuration may be omitted. This is to avoid the following description becoming unnecessarily redundant and to facilitate understanding by those skilled in the art. Note that the attached drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0017] (Embodiment 1) Fig. 1 is a diagram showing an example of a system configuration of a surveillance camera system 100 according to embodiments 1 and 2. As shown in Fig. 1, the surveillance camera system 100 includes a surveillance camera 1 and a server 50 to which a monitor MN1 and an operation unit MH1 are connected. The surveillance camera 1 and the server 50 are connected via a network NW1 so as to be able to transmit and receive data to and from each other.

[0018] The network NW1 is, for example, a wireless network conforming to any of a wireless LAN (Local Area Network) such as Wi-Fi (registered trademark), Bluetooth (registered trademark), and WiGig (Wireless Gigabit), but is not limited to these. The network NW1 may be a wired network such as a USB (Universal Serial Bus) cable or a wired LAN. In the following description, an image captured by the surveillance camera 1 (hereinafter referred to as a "captured image") includes not only the captured image data, but also the camera ID (Identification) of the surveillance camera 1 that captured the image and information on the capture date and time.

[0019] The surveillance camera 1 is installed in an outdoor location with a good view, such as a weather station, airport, bridge, or port, in order to monitor a subject (for example, a point in a distant monitoring area as seen from the installation position of the surveillance camera 1). However, the installation position of the surveillance camera 1 is not limited to the above-mentioned locations, and may be a pole (not shown) installed on the side of a road or a parking lot if it is outdoors. The surveillance camera 1 captures a subject within a predetermined angle of view (for example, a distant monitoring area as seen from the installation position of the surveillance camera 1). Here, the subject may be, for example, an airport, a distant part of a long-distance bridge, a port, a mountain, or a city, but is not limited to these. In addition, the surveillance camera 1 is equipped with artificial intelligence (AI), and uses the installed artificial intelligence to determine whether or not a meteorological phenomenon that causes deterioration of image quality, such as fog or mist, is occurring in the captured image of the subject (see below for details).

[0020] In the surveillance camera 1, a trained model is stored in the trained model memory 152 (see FIG. 2). This trained model is a parameter group that characterizes the function of the AI ​​mounted on the surveillance camera 1. In other words, the trained model is a parameter group for determining the presence or absence of fog or mist, which is the determination target of the AI ​​mounted on the surveillance camera 1. The trained model may be, for example, a parameter group for using a segmentation technique for determining the presence or absence of fog or mist. This trained model is generated by an external terminal (not shown) and stored in advance in the surveillance camera 1.

[0021] The surveillance camera 1 operates in an operation mode corresponding to one of the following states: a daytime state in which neither fog nor mist occurs (see state CON1 in FIG. 5 described later), a nighttime state (see state CON2 in FIG. 5 described later), and a daytime state in which fog or mist occurs (see state CON3 in FIG. 5 described later). Details will be described later with reference to FIG. 5.

[0022] The server 50 is, for example, an information processing device such as a personal computer, a smartphone, a tablet terminal, or a server computer machine with high-performance specifications. The server 50 communicates data with the surveillance camera 1 via the network NW1.

[0023] The server 50 includes a communication IF circuit 51, a processor 52, a memory 53, and a database 54. In the attached drawings, the interface is abbreviated to "IF" for convenience.

[0024] The communication IF circuit 51 communicates data with the surveillance camera 1 via the network NW1 described above. The communication IF circuit 51 receives captured image data generated and transmitted by the surveillance camera 1, for example, and outputs the data to the processor 52.

[0025] The processor 52 is configured using, for example, a CPU (Central Processing Unit), a DSP (Digital Signal Processor), a GPU (Graphical Processing Unit), or an FPGA (Field Programmable Gate Array). The processor 52 functions as a controller that manages the overall operation of the server 50, and performs control processing for managing the operation of each part of the server 50, input / output processing of data between each part of the server 50, calculation processing of data, and storage processing of data. The processor 52 operates according to the programs and data stored in the memory 53. The processor 52 uses the memory 53 during operation, and temporarily stores data or information generated or acquired by the processor 52 in the memory 53.

[0026] When the processor 52 receives the captured image data received by the communication IF circuit 51, it executes various image processing or analysis processing for observing the state of the subject shown in the captured image. The processor 52 may execute this image processing or analysis processing using a trained model (not shown) that has been trained and generated for each processing, or a database 54 described later. This allows the processor 52 to observe the state of the subject with high accuracy. The processor 52 may respond with the results of the image processing or analysis processing to the surveillance camera 1 or other external device (for example, a terminal carried by a person requesting the observation) connected via the network NW1, or may output (display) the results on the monitor MN1.

[0027] The memory 53 is configured using, for example, a RAM (Random Access Memory) and a ROM (Read Only Memory), and temporarily stores programs necessary for executing the operation of the server 50, and further, data or information generated during operation. The RAM is, for example, a work memory used during operation of the server 50. The ROM, for example, stores and holds in advance programs for controlling the server 50.

[0028] The database 54 is configured using, for example, a hard disk drive (HDD) or a solid state drive (SSD). The database 54 stores various kinds of matching data (not shown) used for comparing subjects or objects that are the subject of image processing or analysis processing for the above-mentioned observation.

[0029] The monitor MN1 is a display device configured using, for example, an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence). The monitor MN1 displays, for example, the results of image processing or analysis processing executed by the server 50. The monitor MN1 may be included in the server 50.

[0030] The operation unit MH1 is an input device that accepts an input operation by a user, such as a mouse, a keyboard, a touch pad, a touch panel, etc. The operation unit MH1 transmits a signal corresponding to the input operation by the user to the server 50. The operation unit MH1 may be included in the server 50.

[0031] Fig. 2 is a block diagram showing an example of the hardware configuration of the surveillance camera 1 according to the first and second embodiments. In addition to the surveillance camera 1, Fig. 2 also shows an external storage medium M1 that is inserted into and removed from the surveillance camera 1. The external storage medium M1 is, for example, a storage medium such as an SD card.

[0032] As shown in Figure 2, the surveillance camera 1 includes a lens 11, a lens driving unit LM1, a visible light cut filter CF1, an IR cut filter CF2, a filter driving unit FM1, an image sensor 12, a memory 13, a processor 14, an AI processor 15, a communication IF circuit 16, an IR illumination unit 17, an external storage medium IF 18, and a pan-tilt driving unit PTM1.

[0033] The lens 11, which is an example of an imaging unit, includes, for example, a focus lens and a zoom lens, and receives incident light ICL1, which is light reflected by a subject, and forms an optical image of the subject on the light receiving surface (imaging surface) of the image sensor 12 via at least one of a visible light cut filter CF1 and an IR cut filter CF2 as necessary. This incident light ICL1 is, for example, light that enters (transmits) through fog or mist occurring in the atmosphere during the day, light that enters (transmits) through an atmosphere without fog or mist occurring during the day, or light that enters (transmits) through an atmosphere at night. Lenses with various focal lengths or shooting ranges can be used for the lens 11 depending on the installation location of the surveillance camera 1 or the shooting purpose.

[0034] The lens driver LM1, which is an example of an imaging unit, drives the lens 11 based on a control signal from the processor 14 to adjust camera parameters related to the lens 11 (for example, the position of the focus lens, the position of the zoom lens corresponding to the zoom magnification).

[0035] The visible light cut filter CF1, which is an example of an imaging unit, is driven by the filter drive unit FM1 so as to be effective (i.e., used) when it is determined (see below) that fog or haze is occurring in the daytime atmosphere. The visible light cut filter CF1 has a spectral characteristic of passing light having a specific wavelength (e.g., 560±20 [nm: nanometers]) or more among the incident light ICL1 (an example of light from a subject) transmitted through the lens 11 (see FIG. 3). Note that, when it is determined (see below) that neither fog nor haze is occurring in the daytime atmosphere or at night, the visible light cut filter CF1 is driven by the filter drive unit FM1 so as to be ineffective and open (i.e., not used).

[0036] The IR cut filter CF2, which is an example of an imaging unit, is driven by the filter driving unit FM1 so as to be effective (i.e., used) when it is determined (see below) that neither fog nor mist is occurring in the daytime atmosphere. The IR cut filter CF2 has a spectral characteristic of passing visible light (e.g., light having a wavelength of 400 to 760 [nm]) and blocking near-infrared light (e.g., light having a wavelength of 780 [nm] or more). Note that when it is determined (see below) that fog or mist is occurring in the daytime atmosphere or at night, the IR cut filter CF2 is driven by the filter driving unit FM1 so as to be ineffective and open (i.e., not used).

[0037] A filter driving unit FM1, as an example of an imaging unit, drives each of the visible light cut filter CF1 and the IR cut filter CF2 to control whether or not each of them is enabled (for example, by placing the corresponding filter on the optical path of the incident light ICL1) based on a control signal from the processor 14.

[0038] The image sensor 12 as an example of an imaging section is configured to have a plurality of pixels (for example, pixels each having a Bayer array RGB color filter provided on each light receiving element), and is, for example, an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The image sensor 12 performs photoelectric conversion to convert light received on a light receiving surface (imaging surface) configured by the above-mentioned plurality of pixels into an electric signal. As a result, the image sensor 12 can obtain, for each pixel, a blue component signal (B signal), a red component signal (R signal), and a green component signal (G signal) corresponding to light from a subject. The image sensor 12 outputs an electric signal (analog signal) corresponding to the light received on the light receiving surface to the processor 14. This analog signal is converted into digital captured image data by the processor 14 as an example of an imaging section. As a result, captured image data is generated by the processor 14.

[0039] Memory 13 is configured using, for example, a RAM and a ROM, and temporarily stores programs necessary for executing the operation of surveillance camera 1, as well as data or information generated during operation. The RAM is, for example, a work memory used when surveillance camera 1 is operating. The ROM stores and holds, for example, a program according to the present disclosure for controlling surveillance camera 1 in advance. In other words, processor 14 executes the program stored in the ROM to enable surveillance camera 1, which is a computer, to execute various processes related to image processing according to the present disclosure.

[0040] The memory 13 also stores, for example, thresholds (specifically, the first threshold A and the second threshold B (> the first threshold A)) of the amount of change in the high-frequency spatial frequency (see FIG. 6) used by the AI ​​processor 15 when making a comparison. The memory 13 also stores, as a reference value f1, a predetermined high-frequency spatial frequency among the calculation results of the spatial frequency based on the pixels corresponding to the subject (target area) to be observed under normal circumstances (i.e., a state in which neither fog nor mist occurs) and the entire area of ​​the captured image. The fact that the predetermined high-frequency spatial frequency is the reference value f1 indicates that an edge portion of the captured image (in other words, the clarity of the captured image) exists. Therefore, if the high-frequency spatial frequency among the calculated spatial frequencies is less than the reference value f1, it indicates that the edge of the captured image is unclear due to a meteorological phenomenon such as fog or mist.

[0041] Processor 14 is configured using, for example, a CPU, DSP, GPU, or FPGA. Processor 14 functions as a controller that manages the overall operation of surveillance camera 1, and performs control processing for managing the operation of each part of surveillance camera 1, input / output processing of data between each part of surveillance camera 1, calculation processing of data, and storage processing of data. Processor 14 operates according to programs and data stored in memory 13. Processor 14 uses memory 13 during operation, and temporarily stores data or information generated or acquired by processor 14 in memory 13.

[0042] The processor 14, which is an example of an imaging unit, may generate digital captured image data by performing predetermined signal processing on the electrical signal output from the imaging element 12. The processor 14 sends the generated captured image data to the AI ​​processor 15.

[0043] Processor 14 also has a timer TM1, and can ascertain the current time based on the output of timer TM1, and outputs a control signal to instruct irradiation of IR light to IR illumination unit 17 during the night (in other words, from around sunset to around dawn). Timer TM1 has a circuit that measures the elapsed time from a predetermined reference time (i.e., the current time), and sends the measurement output (count value) to processor 14. This allows processor 14 to identify the current time.

[0044] Furthermore, processor 14, as an example of a control unit, switches the operation mode (see FIG. 5) based on the output of timer TM1 and the processing result of AI processor 15 (for example, the result of determining whether or not fog or mist is occurring in the atmosphere in which a subject (for example, a point in a distant monitoring area as seen from the installation position of surveillance camera 1) is visible). FIG. 5 is a diagram showing an example of transitions in the operation state of surveillance camera 1 according to embodiment 1. As shown in FIG. 5, surveillance camera 1 operates in an operation mode corresponding to one of a daytime state CON1 in which neither fog nor mist is occurring, a nighttime state CON2, and a daytime state CON3 in which fog or mist is occurring.

[0045] For example, the processor 14 determines to operate in an operation mode corresponding to the state CON2 during the night (for example, the time period from 18:00 to 6:00) based on the output of the timer TM1. That is, in order to capture an image of a subject at night, the processor 14 opens (i.e., does not use) each of the visible light cut filter CF1 and the IR cut filter CF2, and captures an image of the subject using a color R (Red), G (Green), and B (Blue) signal under irradiation of IR light by the IR illumination unit 17 (for example, an IR LED). Note that the time period of the night may differ depending on the season, and data indicating the time period of the night may be stored in the memory 13 for each season or month. Each pixel of the image sensor 12 has sensitivity even in the wavelength region of near-infrared light, and by irradiating IR light, the processor 14 can capture an image of the subject even in a dark environment. However, since it is not possible to reproduce colors correctly, the processor 14 generates a monochrome image of the subject. Note that the determination of daytime and nighttime may be performed using an output value of an illuminance sensor (not shown) provided in the surveillance camera 1 without using the timer TM1.

[0046] For example, the processor 14 operates in an operation mode selected from state CON1 or state CON3 based on the processing result of the AI ​​processor 15 (see above) during a time period other than nighttime (i.e., daytime, for example, from 6:00 to 18:00) according to the output of the timer TM1. Specifically, when the AI ​​processor 15 determines that there is no fog or mist in the atmosphere where a subject (for example, a point in a distant monitoring area as seen from the installation position of the monitoring camera 1) can be seen, the processor 14 decides to operate in the operation mode corresponding to state CON1. ​​That is, in order to capture a subject in the daytime when there is no fog or mist in the atmosphere, the processor 14 enables (i.e., uses) only the IR cut filter CF2 and captures the subject using the above-mentioned color RGB signal. This allows the processor 14 to obtain color image data based on light from which the wavelengths of near-infrared light that affect color reproduction have been removed.

[0047] Furthermore, when the AI ​​processor 15 determines that fog or mist is occurring in the atmosphere where a subject (for example, a point in a distant monitoring area as seen from the installation position of the monitoring camera 1) can be seen, the processor 14 decides to operate in an operation mode corresponding to the state CON3. That is, in order to capture a subject in the daytime when fog or mist is occurring in the atmosphere, the processor 14 enables (i.e., uses) only the visible light cut filter CF1 that blocks visible light having a wavelength less than a specific wavelength (see above), and captures the subject using the above-mentioned color RGB signal or B signal (see FIG. 3). As a result, the processor 14 can obtain high-quality captured image data in which the effect of fog or mist is reduced, although the amount of visible light is reduced compared to the state CON1. ​​Furthermore, if the degree of fog or mist is not high (for example, the amount of change in high-frequency spatial frequency is less than the second threshold B shown in FIG. 6), the processor 14 can obtain captured image data in a state in which the amount of reduction in the amount of light is suppressed compared to the state in which all visible light is cut as shown in Patent Document 1.

[0048] If the AI ​​processor 15 determines that the fog or haze in the atmosphere has cleared during the day, the processor 14 operates in an operation mode that has transitioned from state CON3 to state CON1. ​​Similarly, if the AI ​​processor 15 determines that fog or haze has appeared in the atmosphere during the day, the processor 14 operates in an operation mode that has transitioned from state CON1 to state CON3.

[0049] The AI ​​processor 15 as an example of the second processor is configured using, for example, a CPU, a DSP, a GPU, an FPGA, and a memory, and uses AI (artificial intelligence) to extract pixels corresponding to a distant object (target area) to be observed from the data of the captured image of the object captured by the image sensor 12. The AI ​​processor 15 calculates spatial frequencies for the area configured by the extracted pixels and the entire captured image. The AI ​​processor 15 detects and determines whether or not fog or mist is occurring in the distant object (target area) to be observed, based on a comparison between the calculation result of the spatial frequency and a threshold value (see above) stored in the memory 13. The AI ​​processor 15 is configured to include an AI calculation processing unit 151 and a learning model memory 152. Note that the calculation of the individual spatial frequencies for the area configured by the extracted pixels and the entire captured image, as well as the processing of detecting and determining whether or not fog or mist is occurring in the distant object (target area) to be observed may be performed by the processor 14 instead of the AI ​​processor 15, and the same applies in the following description.

[0050] The AI ​​arithmetic processing unit 151 forms an AI (artificial intelligence) based on a trained model (see FIG. 2) stored in the learning model memory 152. For example, the AI ​​arithmetic processing unit 151 inputs data of an image of a subject captured by the image sensor 12 using the trained model, which is a parameter group for using a segmentation technique, and detects and extracts pixels corresponding to a distant subject (target area) from the input captured image data. The AI ​​arithmetic processing unit 151 or the processor 14 calculates spatial frequencies for the area composed of the extracted pixels and the entire captured image based on the extraction results, and determines whether or not fog or mist is occurring in the distant subject (target area) to be observed based on the calculation results of the spatial frequency and a threshold value (see above) stored in the memory 13, and returns the determination result to the processor 14.

[0051] Here, the determination of the occurrence or non-occurrence of fog or mist and the intensity (thickness) of fog or mist by the AI ​​processor 15 or processor 14 will be described with reference to Figs. 4A, 4B and 6. Fig. 4A is a diagram showing a situation where fog or mist occurs in almost the entire area AR1 in an image PIC1. Fig. 4B is a diagram showing a situation where fog or mist occurs in a distant target area AR2 in an image PIC2. Fig. 6 is a diagram showing a typical example of the relationship between spatial frequency, the first threshold value A and the second threshold value B. In Fig. 6, the horizontal axis is the spatial frequency. That is, for each of the spatial frequency characteristics Y1, Y2, Y3, a predetermined high-frequency spatial frequency is calculated, and the calculation results are f1 (see above), f2, and f3.

[0052] When a meteorological phenomenon such as fog or mist occurs in the atmosphere, the atmosphere (air) of almost the entire area AR1 of the image PIC1 or the distant target area AR2 of the image PIC2 appears hazy, degrading the image quality of the image PIC1 or the image PIC2. The AI ​​arithmetic processing unit 151 extracts pixels (e.g., almost the entire area AR1) corresponding to a distant subject (target area) to be observed in the image PIC1 input to the AI ​​arithmetic processing unit 151 based on a trained model using an existing segmentation technique. Similarly, the AI ​​arithmetic processing unit 151 extracts pixels (e.g., a distant target area AR2) corresponding to a distant subject (target area) to be observed in the image PIC2 input to the AI ​​arithmetic processing unit 151 based on a trained model using an existing segmentation technique.

[0053] The AI ​​calculation processor 151 calculates the spatial frequency for each of almost the entire area AR1 and the entire area of ​​the image PIC1, and determines whether the difference (amount of change) between a predetermined high-frequency portion (high-frequency spatial frequency) f2 of the spatial frequency corresponding to each area and a reference value f1 of the predetermined high-frequency portion (high-frequency spatial frequency) of the spatial frequency of the corresponding area stored in memory 13 is equal to or greater than a first threshold value A stored in memory 13. When the AI ​​calculation processor 151 determines that the amount of change (see above) in the high-frequency spatial frequency in almost the entire area AR1 or the entire area of ​​the image PIC1 is equal to or greater than the first threshold value A, it determines that fog or mist has occurred in the image PIC1.

[0054] Similarly, the AI ​​calculation processor 151 calculates the spatial frequency for each of the distant target area AR2 and the entire area of ​​the image PIC2, ​​and determines whether the difference (amount of change) between a predefined high-frequency portion (high spatial frequency) f2 of the spatial frequency corresponding to each area and a reference value f1 of the predefined high-frequency portion (high spatial frequency) of the spatial frequency of the corresponding area stored in the memory 13 is equal to or greater than a first threshold value A stored in the memory 13. When the AI ​​calculation processor 151 determines that the amount of change (see above) in the high-frequency spatial frequency in the distant target area AR2 or the entire area of ​​the image PIC2 is equal to or greater than the first threshold value A, it determines that fog or mist has occurred in the image PIC2.

[0055] The learning model memory 152 is composed of a memory such as a RAM, a ROM, or a flash memory. The learning model memory 152 stores a learned model for fog or mist detection processing created in advance by a learning process. The AI ​​calculation processing unit 151 reads out the learned model from the learning model memory 152, extracts pixels corresponding to a distant subject (target area) to be observed in the image input to the AI ​​calculation processing unit 151, and can detect and determine the presence or absence of fog or mist with high accuracy.

[0056] Note that the processor 14 may simply determine the presence or absence of fog or mist based on, for example, the output of the timer TM1 (i.e., the current time) instead of performing highly accurate detection processing of the presence or absence of fog or mist using the AI ​​processor 15. This is based on the fact that if it is possible to specify in advance a time period during which fog or mist is likely to occur in consideration of the regional characteristics around the installation position of the surveillance camera 1, it is possible to simply determine the presence or absence of fog or mist even if the current time is within a time period during which fog or mist is likely to occur.

[0057] In addition, when the surveillance camera 1 receives a coordinate designation notification from the server 50 specifying coordinates in the captured image where fog or mist is expected to occur, the processor 14 or the AI ​​processor 15 may determine whether fog or mist is occurring by limiting the specified coordinates included in the coordinate designation notification (for example, coordinates in the captured image data generated by the image sensor 12 or the processor 14).

[0058] The communication IF circuit 16 performs data communication (transmission and reception) with the server 50 connected via the network NW1. The communication IF circuit 16 transmits, for example, captured image data generated by the processor 14 to the server 50. When the communication IF circuit 16 receives a coordinate designation notification (see above) indicating the presence or absence of fog or mist from the server 50, the communication IF circuit 16 may send the designated coordinates (see above) included in the coordinate designation notification to the processor 14 or the AI ​​processor 15.

[0059] The IR illumination unit 17 starts irradiating the imaging area with IR light RD1 having a near-infrared wavelength band based on a control signal (e.g., an instruction to start irradiating IR light) from the processor 14. The IR illumination unit 17 ends the irradiation of the IR light RD1 to the imaging area based on a control signal (e.g., an instruction to end irradiation of IR light RD1) from the processor 14. Furthermore, the IR illumination unit 17 irradiates the currently irradiated IR light RD1 by increasing or decreasing the intensity based on a control signal (e.g., an instruction to adjust the intensity of IR light RD1) from the processor 14.

[0060] An external storage medium M1 such as an SD card is inserted into the external storage medium IF18.

[0061] The pan / tilt driving unit PTM1 adjusts camera parameters related to pan or tilt rotation (e.g., the amount of pan rotation, the amount of tilt rotation) based on a control signal from the processor 14 to perform pan rotation, tilt rotation, or both of the surveillance camera 1. If the surveillance camera 1 is configured not to perform pan rotation or tilt rotation, the pan / tilt driving unit PTM1 may be omitted from the configuration of the surveillance camera 1.

[0062] Next, an outline of the operation of the processor 14 of the surveillance camera 1 according to the first embodiment with respect to generation of a captured image in a foggy or misty state CON3 will be described with reference to Figs. 3 and 6. Fig. 3 is a diagram showing an example of an outline of the operation of the surveillance camera 1 according to the first embodiment. A graph GPH1 shown in Fig. 3 shows the spectral sensitivity characteristic of the image sensor 12, with the horizontal axis indicating wavelength [nm] and the vertical axis indicating sensitivity. Below the graph GPH1, a hue range CLB1 and various characteristics PRY1 used to generate data for the captured image are shown.

[0063] According to the graph GPH1, the blue component signal (hereinafter sometimes abbreviated as "B signal"), the red component signal (hereinafter sometimes abbreviated as "R signal"), and the green component signal (hereinafter sometimes abbreviated as "G signal") for each pixel acquired by the image sensor 12 have equivalent sensitivity in the near-infrared wavelength band (for example, the wavelength band a1 of 850 [nm]). Therefore, even when the image sensor 12 having an array of RGB color filters is used, imaging of near-infrared light is possible.

[0064] As described above, the visible light cut filter CF1 passes light having a wavelength equal to or greater than a specific wavelength (e.g., 560±20 [nm: nanometers]) among the incident light ICL1 (an example of light from a subject) transmitted through the lens 11. In other words, the visible light cut filter CF1 blocks light having a wavelength less than the specific wavelength. Therefore, as shown by the characteristic PRY1, when the visible light cut filter CF1 is effective, all light having a wavelength in the wavelength band shown in the hue range CLB1 (e.g., UV light to IR light) does not enter the image sensor 12, and only light having a wavelength equal to or greater than the specific wavelength enters the image sensor 12.

[0065] Here, the specific wavelength is, for example, 560±20 [nm], but when a meteorological phenomenon such as fog or mist occurs, light having a wavelength less than the specific wavelength is easily scattered by the water vapor that constitutes the fog or mist, which becomes an obstacle to generating a clear image of the subject. Therefore, in the first embodiment, the surveillance camera 1 generates a captured image based on light having a wavelength equal to or greater than the specific wavelength (for example, a wavelength of 560±20 [nm]) by utilizing the characteristic that the longer the wavelength of light, the less likely it is to be scattered by the water vapor that constitutes the fog or mist. This allows the surveillance camera 1 to generate a clear captured image even when fog or mist is occurring.

[0066] However, since the specific wavelength is a wavelength in the wavelength band of visible light (e.g., 400 to 760 nm), it is possible to generate a captured image using R signals, G signals, and B signals (i.e., RGB signals) based on light having a wavelength equal to or greater than the specific wavelength and in the wavelength band of visible light (e.g., a wavelength band corresponding to the hues of yellow, orange, and red).

[0067] Therefore, if the degree of fog or haze is not so thick (see FIG. 6), the processor 14 of the surveillance camera 1 uses an RGB signal based on light having a wavelength equal to or greater than a specific wavelength and having a wavelength band of visible light (e.g., a wavelength band corresponding to hues of yellow, orange, and red) in addition to the wavelength band of near-infrared light, and can generate a captured image in which the amount of light reaching the image sensor 12 is greater than a monochrome captured image captured only with near-infrared light (IR light) and the subject's appearance is clear even if fog or haze occurs. For example, when the processor 14 determines that the amount of change in high-frequency spatial frequency (see above) in almost the entire area AR1 or the entire area of ​​the image PIC1 is less than the second threshold value B, the processor 14 determines that the degree of fog or haze is not so thick and generates a monochrome captured image in which the subject's appearance is clear even if fog or haze occurs using an RGB signal based on light having a wavelength equal to or greater than a specific wavelength and having a wavelength band of visible light (e.g., a wavelength band corresponding to hues of yellow, orange, and red) in addition to the wavelength band of near-infrared light. Although the method of generating a captured image has been described above by taking the image PIC1 as an example, the method is similar even if the image PIC2 is taken as an example, and therefore the description will be omitted.

[0068] On the other hand, when the degree of fog or haze is thick (see FIG. 6), the surveillance camera 1 can generate a captured image in which the subject's appearance is clear even when the degree of fog or haze is thick by using only the B signal (see wavelength band a1) that has high sensitivity to near-infrared light (for example, a wavelength band of 780 [nm] or more) but has almost no sensitivity to a wavelength band of visible light of a specific wavelength or more. For example, when the processor 14 determines that the amount of change in high-frequency spatial frequency (see above) in almost the entire area AR1 or the entire area of ​​the image PIC1 is equal to or greater than the second threshold value B, the processor 14 determines that the degree of fog or haze is thick, and generates a monochrome captured image in which the subject's appearance is clear even when fog or haze occurs by using only the B signal (see symbol a1) that has high sensitivity to near-infrared light (for example, a wavelength band of 780 [nm] or more) but has almost no sensitivity to a wavelength band of visible light of a specific wavelength or more. Note that the method of generating the captured image has been described here by taking the image PIC1 as an example, but the same can be achieved by taking the image PIC2 as an example, so a description thereof will be omitted.

[0069] That is, the processor 14 of the surveillance camera 1 uses a different method for generating a captured image depending on the magnitude of the amount of change in high-frequency spatial frequency (see above) in almost the entire area AR1 or the entire area of ​​the image PIC1 and the second threshold value B. This allows the surveillance camera 1 to generate a high-quality captured image in which the effects of fog or mist are adaptively reduced depending on the level (e.g., density) of the fog or mist detected by the AI ​​processor 15, for example.

[0070] Next, an example of the operational procedure of the surveillance camera 1 according to the first embodiment will be described with reference to Fig. 7 and Fig. 8. Fig. 7 is a flowchart showing an example of the operational procedure of the surveillance camera 1 according to the first embodiment in chronological order to determine whether or not fog or mist is occurring. Fig. 8 is a flowchart showing an example of the image processing operational procedure of the surveillance camera 1 according to the first embodiment in chronological order while fog or mist is occurring. Note that in Figs. 7 and 8, fog or mist is conveniently illustrated as "fog / haze."

[0071] In FIG. 7, the AI ​​processor 15 inputs data of an image captured by the image sensor 12 (St1). The AI ​​processor 15 uses the trained model read from the trained model memory 152 to extract pixels corresponding to a distant object (target area) from the data of the image of the object input in step St1 (St2). The AI ​​processor 15 calculates spatial frequencies for an area formed by the pixels extracted in step St2 and the entire captured image input in step St1 (St3). That is, in step St3, the spatial frequency of pixels corresponding to a distant object (target area) to be observed (see, for example, almost the entire area AR1 in FIG. 4A) and the spatial frequency of the entire image PIC1 are calculated.

[0072] Here, if the calculation results of the spatial frequency of the distant object (target area) to be observed in the captured image within the current angle of view of the surveillance camera 1 and the entire captured image in normal times (i.e., when no fog or haze occurs) have not been registered (St4, YES), the AI ​​processor 15 associates the calculation results in step St3 (i.e., the calculation results of the spatial frequency in the pixel corresponding to the distant object (target area) to be observed (see, for example, almost the entire area AR1 in FIG. 4A) and the spatial frequency in the entire image PIC1) with the area information indicating the coordinate position of the pixel extracted in step St2 and stores them in the memory 13 (St5). After step St5, the processing of the AI ​​processor 15 returns to step St1.

[0073] On the other hand, if the calculation results of the spatial frequencies of the distant object (target area) to be observed in the captured image within the current angle of view of the surveillance camera 1 and the entire captured image in normal times (i.e., when no fog or mist is occurring) are registered in the memory 13 (St4, NO), the AI ​​processor 15 determines whether or not the change (decrease) in a specific high-frequency spatial frequency of the target area or the entire captured image between the initial registration time already registered in the memory 13 and the current time is equal to or greater than the first threshold A (see FIG. 6) stored in the memory 13 (St6). If the AI ​​processor 15 determines that the change (decrease) in the high-frequency spatial frequency is less than the first threshold A (St6, NO), the processor 14 outputs the captured image data input in step St1 as is (St7).

[0074] On the other hand, when the AI ​​processor 15 determines that the amount of change (amount of decrease) in the high spatial frequency is equal to or greater than the first threshold A (St6, YES), it determines that fog or mist is occurring in the captured image input in step St1 (St8). The AI ​​processor 15 returns the determination result in step St8 to the processor 14. Note that steps St3 to St8 may be performed by the processor 14.

[0075] In FIG. 8, the processor 14 generates a control signal for enabling the visible light cut filter CF1 and opening the IR cut filter CF2 based on the determination result in step St8, and sends the control signal to the filter driver FM1 (St11).

[0076] After step St11, the processor 14 determines whether or not the change (decrease) in a specific high-frequency spatial frequency of the target area or the entire captured image between the time of initial registration already registered in the memory 13 and the current time (for example, when the captured image data was input in step St1) is equal to or greater than a second threshold value B (see Figure 6) stored in the memory 13 (St12).

[0077] When the processor 14 determines that the change (decrease) in the high spatial frequency is equal to or greater than the second threshold value B (St12, YES), it generates monochrome captured image data using only the pixel values ​​of the B signal corresponding to light in the near-infrared wavelength band acquired by the image sensor 12 for the entire size of the captured image (St13).

[0078] On the other hand, if the processor 14 determines that the change (decrease) in the high-frequency spatial frequency is less than the second threshold value B (St12, NO), it generates monochrome captured image data for the entire size of the captured image using pixel values ​​of RGB signals corresponding to light having a wavelength equal to or greater than a specific wavelength, acquired by the image sensor 12 (St14).

[0079] Fig. 9 is a diagram showing an example of an image PIC3 generated by the image processing operation procedure of Fig. 8. The image PIC3 shown in Fig. 9 is generated using only pixel values ​​of B signals corresponding to light in the wavelength band of near-infrared light, or pixel values ​​of RGB signals corresponding to light having a wavelength equal to or greater than a specific wavelength. In other words, in either case, light in a wavelength band (e.g., wavelengths less than a specific wavelength) that is easily scattered by water vapor in a situation where fog or mist is occurring is not used for imaging by the imaging element 12, so that an image PIC3 in which a distant monitoring area (e.g., target area AR2) is clearly shown can be obtained.

[0080] As described above, the surveillance camera 1 of embodiment 1 comprises an image sensor 12 having a plurality of pixels, which acquires, for each pixel, a blue component signal, a red component signal, and a green component signal corresponding to light from a subject incident through fog or mist, an optical filter (e.g., a visible light cut filter CF1) which is disposed between the subject and the image sensor 12 and passes a specific wavelength light (e.g., light having a wavelength of 560±20 nm or more) having a specific wavelength or more among the light from the subject, and a processor 14 which generates and outputs an image of the subject based on any of the blue component signal, red component signal, and green component signal for each pixel corresponding to the specific wavelength light and the blue component signal for each pixel corresponding to the specific wavelength light.

[0081] As a result, even if a phenomenon occurs that degrades the image quality of the captured subject (for example, a meteorological phenomenon such as fog or mist), the surveillance camera 1 blocks light in a wavelength range that is easily scattered by the water vapor that makes up the fog or mist (i.e., light having a wavelength less than a specific wavelength), thereby improving the image quality of the subject and making observation work (for example, remotely monitoring a distant subject from the location where the surveillance camera 1 is installed) more efficient.

[0082] The surveillance camera 1 further includes a second processor (e.g., AI processor 15) that specifies a region of interest (e.g., a distant surveillance area) of the image captured by the imaging element 12. When the processor 14 determines whether fog or mist is occurring in the specified region of interest, it selects one of a blue component signal, a red component signal, and a green component signal for each pixel corresponding to a specific wavelength light, and a blue component signal for each pixel corresponding to a specific wavelength light, depending on the level of fog or mist occurring in the region of interest. This allows the surveillance camera 1 to generate a high-quality captured image in which the effects of fog or mist are adaptively reduced depending on the level (e.g., density) of the fog or mist detected by the AI ​​calculation processing unit 151, for example.

[0083] Furthermore, the processor 14 or the second processor (e.g., AI processor 15) determines that fog or mist has occurred in the region of interest when the amount of reduction (amount of change) in the high-frequency spatial frequency of the image portion of the region of interest is equal to or greater than the first threshold value A. This allows the surveillance camera 1 to easily determine whether fog or mist has occurred in the region of interest to be observed, based on a comparison between the amount of reduction in the predetermined high-frequency spatial frequency in the input captured image and the first threshold value A.

[0084] In addition, the second processor (e.g., AI processor 15) determines whether or not fog or mist is occurring in the region of interest based on a trained model using artificial intelligence formed by a learning process. This allows the surveillance camera 1 to easily and highly accurately determine whether or not fog or mist is occurring in the region of interest being observed using existing AI technology (e.g., segmentation technology).

[0085] In the first embodiment, the specific wavelength is, for example, 560±20 [nanometers]. This allows the surveillance camera 1 to generate a highly clear captured image with reduced effects of fog or mist by utilizing the characteristic that light having a wavelength of, for example, less than 560±20 [nm] is easily scattered by water vapor that constitutes fog or mist.

[0086] The surveillance camera 1 further includes a second optical filter (e.g., an IR cut filter CF2) that passes near-infrared light from the subject. When a predetermined condition (e.g., a state in which neither fog nor mist occurs during the day) is met, the processor 14 generates a captured image of the subject based on the light that has passed through the second optical filter. This allows the surveillance camera 1 to capture images by cutting out IR light during the day when neither fog nor mist occurs, and therefore generate a captured image using a highly accurate color RGB signal.

[0087] (Embodiment 2) In the first embodiment, an example is described in which a captured image capable of reducing the effects of fog or mist is generated using only light having a wavelength equal to or greater than a specific wavelength (e.g., 560 [nm]) that passes through the visible light cut filter CF1. In the second embodiment, an example is described in which a captured image capable of reducing the effects of fog or mist is generated mainly by signal processing without using the visible light cut filter CF1. The surveillance camera 1 according to the second embodiment has a configuration in which the visible light cut filter CF1 is omitted from the hardware configuration example shown in FIG. 2. Therefore, in the description of the hardware configuration of the surveillance camera 1 according to the second embodiment, the description of the same contents as those in the first embodiment will be simplified or omitted, and only the different contents will be described.

[0088] Next, an outline of the operation of the processor 14 of the surveillance camera 1 according to the second embodiment with respect to generation of a captured image in a foggy or misty state CON3 will be described with reference to Fig. 10 and Fig. 6. Fig. 10 is a diagram showing an example of an outline of the operation of the surveillance camera 1 according to the second embodiment. In the graph GPH1 shown in Fig. 10, the horizontal axis indicates wavelength [nm] and the vertical axis indicates sensitivity, and the spectral sensitivity characteristic of the image sensor 12 is shown. Below the graph GPH1, the hue range CLB1 and various characteristics PRY2 used to generate data of the captured image are shown. In the description of Fig. 10, the same elements as those in Fig. 3 are given the same reference numerals, and the description is simplified or omitted, and different contents will be described.

[0089] In the second embodiment, the processor 14, when generating captured image data, selects an R signal corresponding to light having a wavelength equal to or greater than the specific wavelength (e.g., 560±20 [nm]) described with reference to the first embodiment in order to configure a luminance signal of the captured image with reduced effects of fog or mist. In addition, when generating captured image data, the processor 14 colors red parts in the captured image captured with RGB signals orange or red, and makes blue or green background parts of the sky, sea, mountains, etc. in the same captured image monochrome (black and white), thereby generating a captured image in which only the parts of a color of interest such as orange or red are colored.

[0090] That is, in the second embodiment, the visible light cut filter CF1 is omitted, and light having a wavelength in the wavelength band indicated by the hue range CLB1 (for example, UV light to IR light) is incident on the image sensor 12. However, since light having a wavelength less than a specific wavelength (for example, 560±20 [nm]) is easily scattered by water vapor that constitutes fog or mist, the visible light cut filter CF1 is not used in the second embodiment, but the processor 14 selects and uses light having a wavelength equal to or greater than a specific wavelength (see above) from the wavelength band of light incident on the image sensor 12 in order to form a luminance signal of the captured image. Specifically, an R signal having a high sensitivity to wavelengths equal to or greater than the specific wavelength (see above) is used.

[0091] Processor 14 also checks the hue of the signal corresponding to the light incident on image sensor 12 (i.e., light having a wavelength in the wavelength band indicated in hue range CLB1 (e.g., UV light to IR light)), and performs a coloring process in which red parts are colored orange or red, and other color parts that are not red (e.g., blue or green parts of the sky, sea, mountains, etc. described above) are made monochrome. In this way, processor 14 can apply red or orange coloring to attention areas such as red parts of the color RGB signal (e.g., a life jacket worn by a person within the field of view), thereby generating an image that can be made to stand out or call attention.

[0092] Next, an example of the operation procedure of the surveillance camera 1 according to the second embodiment will be described with reference to Fig. 7 and Fig. 11. Fig. 11 is a flow chart showing, in time series, an example of the image processing operation procedure of the surveillance camera 1 according to the second embodiment while fog or mist is occurring. Note that the operation procedure of the surveillance camera 1 for determining whether or not fog or mist is occurring is the same as that shown in Fig. 7, and therefore the description thereof will be omitted here.

[0093] 11, the processor 14 generates a control signal for opening the IR cut filter CF2 based on the determination result in step St8, and sends the control signal to the filter driving unit FM1 (St21). The processor 14 extracts only the R signal (R pixel) from the RGB signal for each pixel acquired by the imaging element 12, and generates a luminance signal (luminance level) of the captured image (St22). The processor 14 also converts the pixel value (RGB pixel value) of the RGB signal for each pixel acquired by the imaging element 12 into each value in the HSV (Hue Saturation Brightness) space (i.e., hue value, saturation value, and brightness value) (St23).

[0094] The processor 14 determines whether the hue value of each pixel converted in step St23 is in the orange or red region (St24). If the processor 14 determines that the hue value of the pixel is in the orange or red region (St24, YES), it generates an image of the pixel colored red for that pixel (St25). On the other hand, if the processor 14 determines that the hue value of the pixel is not in either the orange or red region (St24, NO), it generates an image of the pixel in monochrome (black and white) without coloring that pixel (St26).

[0095] After step St25 or step St26, the processor 14 judges whether the process of step St25 or step St26 has been performed for all pixels (St27). If it is judged that the process of step St25 or step St26 has not been performed for all pixels (St27, NO), the process of the processor 14 returns to step St24. That is, the processor 14 repeatedly executes the process of step St25 or step St26 for each pixel until the process of step St25 or step St26 has been performed for all pixels.

[0096] On the other hand, if the processor 14 determines that the processing of step St25 or step St26 has been performed for all pixels (St27, YES), it generates captured image data (see FIG. 9) by synthesizing the images after processing of step St25 or step St26 for each of all pixels (St28).

[0097] As described above, the surveillance camera 1 according to the second embodiment includes an image sensor 12 having a plurality of pixels, which acquires a blue component signal, a red component signal, and a green component signal for each pixel, which correspond to light from a subject incident through fog or mist, and a processor 14 which generates a luminance signal based on the red component signal (R signal) for each pixel corresponding to the light from the subject, and generates a first pixel signal (an image of a pixel colored red) colored with a color (e.g., red) corresponding to a specific hue (e.g., orange or red) and a second pixel signal (e.g., an image of a monochrome pixel) not colored with a color (e.g., red) corresponding to the specific hue (e.g., orange or red) based on the blue component signal, red component signal, and green component signal for each pixel corresponding to the light from the subject. The processor 14 generates and outputs a captured image of the subject based on the luminance signal, the first pixel signal, and the second pixel signal.

[0098] As a result, even if a phenomenon occurs that degrades the image quality of the captured image of the subject (for example, a meteorological phenomenon such as fog or mist), surveillance camera 1, unlike embodiment 1, can block light in the wavelength range that is easily scattered by the water vapor that makes up the fog or mist (i.e., light having a wavelength less than a specific wavelength) without having a visible light cut filter CF1, thereby improving the image quality of the subject and making observation work more efficient.

[0099] The surveillance camera 1 further includes a second processor (e.g., AI processor 15) that identifies an area of ​​interest (e.g., a distant surveillance area) in the image captured by the imaging element 12. When fog or mist occurs in the identified area of ​​interest, the processor 14 generates a luminance signal, a first pixel signal, and a second pixel signal. This allows the surveillance camera 1 to generate a high-quality captured image with reduced effects of fog or mist when fog or mist is detected by the AI ​​calculation processing unit 151, for example.

[0100] Furthermore, the processor 14 or the second processor (e.g., the AI ​​processor 15) determines that fog or mist has occurred in the region of interest when the amount of reduction in high-frequency spatial frequency in the image portion of the region of interest is equal to or greater than the first threshold value A. This allows the surveillance camera 1 to easily determine whether fog or mist has occurred in the region of interest to be observed, based on a comparison between the amount of reduction in the predetermined high-frequency spatial frequency in the input captured image and the first threshold value A.

[0101] In addition, the second processor (e.g., AI processor 15) determines whether the fog or mist is occurring in the region of interest based on a trained model using artificial intelligence formed by a learning process. This allows the surveillance camera 1 to easily and accurately determine whether fog or mist is occurring in the region of interest being observed using existing AI technology (e.g., segmentation technology).

[0102] The surveillance camera 1 further includes a second optical filter (e.g., an IR cut filter CF2) that passes near-infrared light from the subject. When a predetermined condition (e.g., a state in which neither fog nor mist occurs during the day) is met, the processor 14 generates a captured image of the subject based on the light that has passed through the second optical filter. This allows the surveillance camera 1 to capture images by cutting out IR light during the day when neither fog nor mist occurs, and therefore generate a captured image using a highly accurate color RGB signal.

[0103] Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications, corrections, substitutions, additions, deletions, and equivalents within the scope of the claims, and it is understood that these also naturally belong to the technical scope of the present disclosure. In addition, the components in the various embodiments described above may be arbitrarily combined within the scope of the invention. [Industrial Applicability]

[0104] The present disclosure is useful as a surveillance camera, an image processing method, and a program that improve the image quality of an image of a subject and increase the efficiency of observation work even when a phenomenon occurs that degrades the image quality of the captured image of the subject. [Explanation of symbols]

[0105] 1. Surveillance cameras 11 Lens 12 Image sensor 13,53 Memory 14, 52 processors 15 AI Processor 151 AI processing unit 152 Learning Model Memory 16, 51 Communication IF circuit 17 IR lighting section 18 External storage medium IF 50 Servers 54 Database 100 Surveillance Camera System FM1 Filter driver LM1 Lens drive unit M1 external storage medium PTM1 Pan-tilt drive unit

Claims

1. an image sensor including a plurality of pixels, the image sensor acquiring, for each pixel, a blue component signal, a red component signal, and a green component signal corresponding to light from a subject incident through fog or mist; an optical filter disposed between the subject and the image sensor, the optical filter passing a specific wavelength light having a wavelength equal to or longer than a specific wavelength among the light from the subject; a processor that generates and outputs a captured image of a subject based on any one of a blue component signal, a red component signal, and a green component signal for each pixel corresponding to the specific wavelength light and a blue component signal for each pixel corresponding to the specific wavelength light; A second processor that identifies a region of interest in an image captured by the imaging device, When the processor determines that the fog or mist is occurring in the specified region of interest, the processor selects, according to the degree of the fog or mist occurring in the region of interest, one of a blue component signal, a red component signal, and a green component signal corresponding to the specific wavelength light for each pixel, and a blue component signal corresponding to the specific wavelength light for each pixel. Surveillance camera.

2. The processor determines that the fog or mist is occurring in the region of interest when a reduction in high spatial frequency of the image portion of the region of interest is equal to or greater than a first threshold value. The surveillance camera according to claim 1 .

3. The second processor identifies the region of interest based on a trained model using artificial intelligence formed by a learning process. The surveillance camera according to claim 1 .

4. The specific wavelength is 560±20 [nanometers]. The surveillance camera according to claim 1 .

5. a second optical filter that blocks near-infrared light from the light from the subject; The processor generates a captured image of the subject based on the light that has passed through the second optical filter when a predetermined condition is satisfied. The surveillance camera according to claim 1 or 2.

6. An image processing method performed by a surveillance camera, comprising: a step of passing specific wavelength light having a wavelength equal to or longer than a specific wavelength among light from a subject incident through fog or mist by an optical filter provided in the surveillance camera; acquiring, for each pixel, a blue component signal, a red component signal, and a green component signal corresponding to the specific wavelength light by an image sensor having a plurality of pixels included in the surveillance camera; identifying a region of interest in an image captured by the imaging device; generating and outputting a captured image of a subject based on any one of a blue component signal, a red component signal, and a green component signal for each pixel corresponding to the specific wavelength light, and a blue component signal for each pixel corresponding to the specific wavelength light, when it is determined that the fog or mist is occurring in the specified region of interest, selecting one of a blue component signal, a red component signal, and a green component signal corresponding to the specific wavelength light for each pixel and a blue component signal corresponding to the specific wavelength light for each pixel according to the level of the fog or mist occurring in the region of interest; Image processing methods.

7. The computer is a surveillance camera. a step of passing specific wavelength light having a wavelength equal to or longer than a specific wavelength among light from a subject incident through fog or mist by an optical filter provided in the surveillance camera; acquiring, for each pixel, a blue component signal, a red component signal, and a green component signal corresponding to the specific wavelength light by an image sensor having a plurality of pixels included in the surveillance camera; identifying a region of interest in an image captured by the imaging device; generating and outputting a captured image of a subject based on any one of a blue component signal, a red component signal, and a green component signal for each pixel corresponding to the specific wavelength light, and a blue component signal for each pixel corresponding to the specific wavelength light; when it is determined that the fog or mist is occurring in the specified region of interest, selecting one of a blue component signal, a red component signal, and a green component signal corresponding to the specific wavelength light for each pixel and a blue component signal corresponding to the specific wavelength light for each pixel according to the level of the fog or mist occurring in the region of interest; program.

Images