Imaging device, control method for imaging device, program, and system

Abstract

To suppress false detection about a subject's intrusion into a specific area.SOLUTION: Imaging means of an imaging device acquires a first image captured at a predetermined angle of view using a first wavelength and a second image captured at the predetermined angle of view using a second wavelength different from the first wavelength. Identification means identifies an area in which a subject is detected in the first image as a first region, and identifies an area in which the subject is detected in the second image as a second region. Determination means determines whether at least a part of the first region or the second region has entered a preset detection area.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a process for determining an intrusion into a predetermined area based on an image. [Background technology]

[0002] Conventionally, it has been possible to detect whether a third party has invaded a predetermined area by analyzing images captured by an imaging device such as a surveillance camera. Patent Document 1 describes a device that analyzes images captured by a surveillance camera and detects an intruder in an intrusion prohibited area.

[0003] Patent document 1 has a specification unit that detects a subject in a captured image as a rectangular area, and detects whether the subject has entered a no-entry area by determining whether all of the points of interest in the rectangular area are included in the no-entry area. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6341388 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the above-mentioned technology, there is a concern that it may be difficult to correctly detect an intrusion when brightness changes due to external light. For example, when strong light is irradiated onto the subject by lighting, a shadow may be generated depending on the installation state of the lighting or the irradiation state, and the shadow may be erroneously detected as part of the subject.

[0006] The present invention has been made in consideration of the above problems, and has an object to suppress erroneous detection of the intrusion of a subject into a specific area. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, an imaging device as one aspect of the present invention has an imaging means for acquiring a first image captured at a predetermined angle of view using a first wavelength and a second image captured at the predetermined angle of view using a second wavelength different from the first wavelength, an identification means for identifying an area in the first image where a subject is detected as a first area and an area in the second image where the subject is detected as a second area, and a determination means for determining whether at least a portion of the first area or the second area is within a predetermined detection area, wherein when the determination means determines that at least a portion of the first area is within the detection area, the determination means determines whether at least a portion of the second area is within the detection area. Effect of the Invention

[0008] According to the present invention, it is possible to suppress erroneous detection regarding the intrusion of a subject into a specific area. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a block diagram showing the configurations of an imaging apparatus and a client apparatus. [Diagram 2] FIG. 4 is a schematic diagram illustrating an analysis process of the imaging device. [Diagram 3] FIG. 4 is a schematic diagram illustrating an analysis process of the imaging device. [Figure 4] 10 is a flowchart illustrating an analysis process operation of the imaging device. [Diagram 5] FIG. 4 is a schematic diagram illustrating an analysis process of the imaging device. [Figure 6] 10 is a flowchart illustrating an analysis process operation of the imaging device. [Figure 7] FIG. 4 is a schematic diagram illustrating an analysis process of the imaging device. [Figure 8] 10 is a flowchart illustrating an analysis process operation of the imaging device. [Figure 9] FIG. 2 is a block diagram illustrating an imaging section of the imaging device. [Figure 10] 4 is a timing chart illustrating an analysis processing operation of the imaging device. [Figure 11] 4 is a timing chart illustrating an analysis processing operation of the imaging device. [Figure 12] FIG. 2 is a block diagram showing the configuration of an imaging apparatus and an external information device. [Figure 13] FIG. 4 is a schematic diagram illustrating an analysis process of the imaging device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] <Embodiment> An image processing device according to an embodiment of the present invention will be described with reference to Fig. 1. Fig. 1(A) is a block diagram showing the functional configuration of an image capturing device 100, which is an image processing device according to an embodiment of the present invention, and a client device 190.

[0012] The imaging device 100 has a visible light imaging element 121 for capturing a visible light image of an imaging region that is an imaging target, and a non-visible light imaging element (infrared light imaging element) 122 for acquiring a non-visible light image of the imaging region. The visible light imaging element 121 can receive, for example, visible light and generate an image signal of a visible light image. The non-visible light imaging element 122 can receive, for example, non-visible light such as infrared light and generate an image signal of a non-visible light image. The visible light imaging element 121 and the non-visible light imaging element 122 may be, for example, a CMOS (Complementary Metal Oxide Semiconductor) sensor, and can convert a subject image formed on an imaging surface into an electrical signal and output it. The image signals, which are electrical signals output from the visible light imaging element 121 and the non-visible light imaging element 122, are input to the image processing unit 140.

[0013] The imaging unit 110 guides light from an imaging region to a visible light imaging element 121 or a non-visible light imaging element 122 to generate an image signal. The imaging unit 110 includes an optical unit (optical mechanism) 111 and an optical unit (optical mechanism) 112. The optical unit 111 includes, for example, a control mechanism for zooming, focusing, aperture, and image stabilization, and a lens group. The optical unit 112 separates light incident on the visible light imaging element 121 from light incident on the non-visible light imaging element 122. The optical unit 112 may be, for example, a prism or a dichroic mirror for wavelength separation. In this embodiment, the optical unit 112 is a dichroic mirror, and is configured to transmit visible light and reflect infrared light. The visible light component transmitted through the dichroic mirror is received by the visible light imaging element 121 arranged behind the dichroic mirror. Moreover, the infrared light component reflected by the dichroic mirror is received by the non-visible light imaging element 122 .

[0014] The visible light imaging element 121 and the non-visible light imaging element 122 are each connected to the control unit 130 and the image processing unit 140. The visible light imaging element 121 and the non-visible light imaging element 122 are controlled by the control unit 130, for example, so that they can capture images of an object (subject) in synchronization with each other.

[0015] The imaging unit 110 includes a driving mechanism (not shown) for controlling the attitude. The optical axis direction of the optical unit may be changeable. Here, the imaging unit 110 is described as a two-axis driving mechanism capable of so-called pan-tilt driving (hereinafter, PT driving). The attitude control mechanism (not shown) is composed of a mechanical driving mechanism including a gear mechanism and a DC motor or a stepping motor as a driving source. The operation of the driving mechanism, such as the driving amount, driving direction, driving speed, and driving acceleration, is controlled by the control unit 130. The attitude control mechanism may be configured to connect a pan head independent of the imaging device. For example, the pan head and the imaging device 100 may be detachably connected via a mount, and when connected, information can be exchanged via the communication unit 170. The attitude control mechanism is not limited to the two-axis PT driving, and a manipulator with multiple joints may be used.

[0016] The control unit 130 is at least one computer (e.g., a CPU) and controls the overall operation of the imaging device 100. The control unit 130 may also compress image data to generate compressed image data. For example, the control unit 130 may perform still image compression or moving image compression. Examples of image compression methods include standards such as H.264, H.265, MPEG, and JPEG. The control unit 130 may also generate image data in any format, including mp4 or avi format.

[0017] The control unit 130 can output image data or compressed image data as described above. For example, the control unit 130 can store the image data in a storage medium within the imaging device 100 or outside the imaging device 100 by outputting the image data. As a specific example, the control unit 130 can record the (compressed) image data in a storage unit 150 or a recording medium (not shown) mounted on the imaging device 100. The control unit 130 can distribute the (compressed) image data by transmitting the (compressed) image data to a client device 190, which is an information processing device outside the imaging device 100, via a communication unit 170 and a network 180. The control unit 130 can execute analysis processing such as determining attributes of a subject image in the image data by transmitting the image data to an analysis processing unit 160. The control unit 130 can also transmit data of the analysis result analyzed by the analysis processing unit 160 to the client device 190.

[0018] The image processing unit 140 can perform various image processing on the captured image data. For example, the image processing unit 140 generates a visible light image and a non-visible light image by performing image processing such as pixel interpolation processing or color conversion processing on the image signal obtained from the imaging element. The image processing unit 140 may also perform correction processing such as pixel defect correction or lens correction, or detection processing for adjusting the black level, focus, exposure, or the like. The image processing unit 140 may also perform demosaic processing, white balance processing, gamma correction processing, edge enhancement processing, noise suppression processing, or the like. The image processing unit 140 can store the image data after such image processing in the storage unit 150.

[0019] The storage unit 150 can store programs or data. The storage unit 150 may have a non-volatile memory and a RAM. The non-volatile memory can store a control program that defines the processing procedure by the control unit 130, or various parameters that the control unit 130 uses for processing. The RAM can be used as a work area for the control unit 130, and can also be used as a storage area for the image processing unit 140 to perform image processing.

[0020] The analysis processing unit 160 is an analysis processing unit that analyzes image data and audio data captured by the imaging device 100. The analysis processing unit 160 selectively executes at least one of pre-analysis processing, analysis processing, and post-analysis processing on the captured image.

[0021] The analysis pre-processing is a process that is executed on a captured image before executing the analysis process described below. In the analysis pre-processing of the present embodiment, as an example, a process of dividing the captured image to generate divided images is executed.

[0022] The analysis process is a process of outputting information obtained by analyzing an input image. In the analysis process of this embodiment, as an example, a process is executed in which a human body detection process, a face detection process, a moving object detection process, etc. are executed using a divided image obtained by the analysis pre-processing as an input, and the analysis process result is output. The analysis process may be, for example, a process configured to output the position of an object in a divided image using a machine learning model that has been trained to be able to detect an object included in an image. In addition, the analysis process may be a process configured to output not only the position information of an object as coordinate information, but also the position of the object as image data binarized like a silhouette.

[0023] The post-analysis process is a process executed after the analysis process is executed. In the post-analysis process of the present embodiment, as an example, the position information of the detected object may be superimposed on the display position of the object as a rectangular frame. Alternatively, the post-analysis process may be executed as a process of extracting the shape of the object from the analysis image output as binarized image data, and superimposing the shape of the object on the original captured image as a color-enhanced silhouette image.

[0024] Alternatively, the results of multiple analysis processes may be superimposed on the image data to output coordinate information of the object. The analysis results are stored in the storage unit 150 as coordinate information or image data, and are used by the control unit 130 for each process as appropriate.

[0025] The analysis process may be a process of detecting an object in an image by performing pattern matching and outputting its position. For example, it may be a face recognition process of determining whether or not a person stored in advance is included in an image. For example, the degree of match between the image feature amount of a person stored in advance and the image feature amount of a person detected from an input image is calculated, and if the degree of match is equal to or greater than a threshold, it is determined that the person is the person stored in advance. In addition, for the purpose of privacy protection, it may be a process of superimposing a predetermined mask image or performing a mosaic process on the person detected from the input image. In addition, it may be a behavior analysis process of detecting whether a person in an image is performing a specific behavior using a learning model that has learned a specific behavior of a person by machine learning. Furthermore, it may be a process of determining what kind of area in an image is. For example, it may be a process of determining what kind of area in an image is using a learning model that has learned buildings, roads, people, the sky, etc. by machine learning. As described above, the executable analysis process can be applied to both image analysis processing using machine learning and image analysis processing without machine learning.

[0026] The analysis processing unit 160 may be configured with, for example, an ASSP (Application Specific Standard Product) configured to be able to execute an inference process using a NN (Neural Network) model. The analysis processing unit 160 may be configured with an FPGA (Field Programmable Gate Array). An NN model for each inference process and parameters for executing the inference process may be stored in advance in the storage unit 150. When selecting an analysis process to be executed by the analysis processing unit 160, the NN model and parameters for the inference process can be deployed in the inference processing unit of the FPGA or ASSP, thereby changing the analysis processing function to be executed. On the other hand, the imaging device 100 may not have an internal analysis processing unit 160 for analyzing a captured image. For example, the imaging device 100 may be configured with a plurality of information processing devices (cloud, server, etc.) connected via a network.

[0027] According to another embodiment, an imaging device transmits captured image data to an external information processing device via a network. The information processing device that receives the image data can analyze the image using an analysis processing unit included in the information processing device and transmit the analysis result to the imaging device. The imaging device may be configured to make a judgment based on the received analysis result and notify a client device of a warning.

[0028] The communication unit 170 is a network processing circuit. The communication unit 170 can convert image data (compressed image data) into a communication signal conforming to a communication protocol and transmit the signal to the network 180, for example.

[0029] The non-visible light illumination unit (infrared light illumination unit) 80 is an illumination mechanism having a light source that irradiates non-visible light having a peak frequency outside the wavelength range of visible light. For example, it is composed of an LED light source that emits infrared light. Even in a low-illumination environment where sufficient brightness cannot be obtained in a visible light image captured using a visible light wavelength (first wavelength), it is possible to irradiate non-visible light (infrared light) having a non-visible light wavelength (second wavelength) different from the visible light wavelength (first wavelength) from the non-visible light illumination unit. This allows sufficient brightness to be obtained in the non-visible light image.

[0030] The client device 190 is communicably connected to the imaging device 100 via the network 180, and can communicate with the imaging device 100. The client device 190 may be an information processing device such as a personal computer. The client device 190 has, for example, a communication unit 191, a control unit 192, a storage unit 193, a display unit 194, and an operation unit 195.

[0031] The communication unit 191 is a network processing circuit, and is capable of communicating with the imaging device 100 via the network 180 .

[0032] The control unit 192 is at least one computer (e.g., a CPU), and controls the overall operation of the client device 190. The control unit 192 can receive image data from the imaging device 100 and perform decompression processing as necessary. The control unit 192 can also control the operation of the imaging device 100 by transmitting control information (control signal) for controlling the imaging device 100 to the imaging device 100 via the communication unit 191.

[0033] The storage unit 193 can store image data received from the imaging device 100. The display unit 194 can receive image data distributed by the imaging device 100 via the communication unit 191. In addition, the display unit 194 can display a user interface (UI) to the user of the client device 190, or an image received from the imaging device 100, or the like.

[0034] The operation unit 195 accepts operation input from a user. The user can perform input for controlling the imaging device 100 or the client device 190 by operating a mouse or a keyboard in accordance with a user interface (UI) such as icons displayed on the display unit 194.

[0035] Each unit of the imaging device 100 and the client device 190 may be configured with dedicated hardware or may be realized by software. In the embodiment described below, the control unit 130, the image processing unit 140, and the control unit 192 are realized by software. For example, a processor such as a CPU executes a program stored in a storage medium such as the storage unit 150, thereby realizing the functions of the control unit 130, the image processing unit 140, and the control unit 192. The analysis processing unit 160 may be configured as an external device connected to the imaging device 100. For example, it can be configured as an inference processing device connected to the imaging device 100 via an I / F such as a USB. It can also be configured to analyze voice data and image data sent from the control unit 130 to the analysis processing unit 160 and return the analysis result to the control unit 130.

[0036] 1B is a block diagram showing an example of a hardware configuration for realizing functions of the control unit 130, image processing unit 140, control unit 192, etc., of the imaging device 100 and the client device 190. The CPU 10, which is a processor, executes an operating system (OS), a control program, a processing program, etc. stored in a hard disk device (HD) 15 to realize processing corresponding to this embodiment. The CPU 10 can also control data transmission and reception with an external device via an interface (I / F) 18.

[0037] The ROM 11 stores various data such as a basic I / O program and an application program for executing a predetermined process. The RAM 12 temporarily stores various data and can function as a main memory or a work area for the CPU 10. The drive 13 can access a medium (recording medium) 14, and can load, for example, a program stored in the medium 14 into the system shown in FIG. 1(B). The HD 15 is a large-capacity memory, such as a hard disk. The HD 15 can store an application program, an OS, a control program, or related programs. The system shown in FIG. 1(B) may have a non-volatile storage device such as a flash memory instead of the HD 15. The ROM 11, the RAM 12, and the HD 15 can function as the storage unit 150 shown in FIG. 1(A).

[0038] The input device 16 acquires a user's input to the imaging device 100 or the client device 190. The input device 16 may be, for example, a keyboard, a pointing device (such as a mouse), or a touch panel, and corresponds to the operation unit 195 shown in FIG. 1(A). The output device 17 can output a command input from the input device 16, and a response to this command from the imaging device 100 or the client device 190, and the like. The output device 17 may be, for example, a display, a speaker, or a headphone terminal, and corresponds to the display unit 194 shown in FIG. 1(A).

[0039] The interface (I / F) 18 mediates data exchange between the imaging device 100 or the client device 190 and an external device. For example, the I / F 18 may include a wireless communication module. This wireless communication module may include known circuits including an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module card, or a memory.

[0040] The I / F 18 may also include a wired communication module for wired connection. This wired communication module may enable communication between the imaging device 100 or the client device 190 and other devices via one or more external ports. The external ports are connected to other devices directly or indirectly via a network according to standards such as Ethernet (registered trademark), USB, and IEEE1394. The I / F 18 may also include various software components for processing data. The I / F 18 may also be realized using software that realizes functions equivalent to those described above. The I / F 18 may function as the communication unit 170 or the communication unit 191 shown in FIG. 1(A). The system bus 19 controls the flow of data within the system shown in FIG. 1(B).

[0041] The flow of processing of the imaging device 100 in this embodiment will be described below with reference to Fig. 2, Fig. 3, and Fig. 4. Specifically, the imaging device 100 monitors the subject area (first area, second area) and detects whether or not a subject is entering a predetermined area that has been determined in advance as an intrusion prohibition area (detection area).

[0042] First, the environment in which the imaging device 100 is installed and the subject to be detected will be described with reference to Fig. 2. Fig. 2 is a schematic diagram for explaining the analysis process of the imaging device of this embodiment. Fig. 2(A) is a schematic diagram showing an example of the installation environment in which the imaging device 100 is installed. Fig. 2(A) shows a state in which a person, subject 203, is viewing an exhibit 202 in an exhibition hall.

[0043] In the example of Fig. 2(A), the imaging device 100 is installed near the ceiling in an exhibition hall. Furthermore, an exhibit 202 installed on a wall is lit up by lighting 201. A shadow 204 caused by the lighting 201 is cast on a subject 203 existing around the exhibit 202. Note that, although Fig. 2(A) shows an example in which the imaging device 100 is installed in an exhibition hall, it may be installed outside the exhibition hall, for example, in a facility such as an art gallery or a museum.

[0044] Fig. 2(B) shows a state in which a prohibited area (entry prohibited area) is set for detecting and warning a viewer who has entered the exhibit 202. In Fig. 2(B), a judgment boundary line 210 is set in the space around the exhibit 202 to detect the intrusion of a viewer (e.g., subject 203). The area inside the judgment boundary line 210 is set as an intrusion prohibited area 211.

[0045] In this embodiment, when a viewer (e.g., subject 203) reaches out his / her hand and approaches or attempts to touch exhibit 202, it is determined (judged) that the viewer has crossed the judgment boundary line 210 and entered entry-prohibited area 211, and a process of issuing a warning is performed. In other words, when the subject crosses the judgment boundary line, it is judged that the subject has entered the entry-prohibited area. In FIG. 2(B), a judgment boundary line 212 is also set on the ground of the exhibition hall, and the area inside the judgment boundary line 212 is set as entry-prohibited area 213.

[0046] FIG. 2C is a schematic diagram showing a state in which an intrusion of a viewer (subject 203) into an intrusion-prohibited area 211 and an intrusion-prohibited area 213 is detected in an image captured by the imaging device 100 capturing an image of the exhibition hall.

[0047] Here, for example, in order to determine whether the subject has entered the intrusion prohibited area 211, the area of ​​the subject 203 is detected as a rectangular frame (rectangular area, detection frame) 214 by the analysis processing unit 160. That is, the analysis processing unit 160 detects the subject from the captured image, and specifies the area in the image of the detected subject as a rectangular frame (first area). In addition, the analysis processing unit 160 specifies the shadow 204 as a rectangular frame (rectangular area, detection frame) 215 which is another object. That is, the shadow of the subject is detected from the captured image, and specifies the area in the image of the detected shadow of the subject as a rectangular frame (first area).

[0048] 2C, the rectangular frame 215 of the shadow 204 encroaches on the intrusion forbidden area 211. Furthermore, the rectangular frame 215 of the shadow 204 encroaches on the intrusion forbidden area 213. Therefore, the imaging device 100 is in a state in which it has notified the client device 190 of two warnings.

[0049] Next, a method for setting the intrusion forbidden area and the judgment boundary line will be described with reference to Fig. 3. Fig. 3 is a schematic diagram showing a method for setting the judgment boundary line and the intrusion forbidden area. Fig. 3(A) shows a method for setting the judgment boundary line on the space seen from the image captured by the imaging device 100. Fig. 3(B) shows a method for setting the judgment boundary line and the intrusion forbidden area on the ground seen from the image captured by the imaging device 100. Fig. 3(C) shows an example of setting the intrusion forbidden area by combining Fig. 3(A) and Fig. 3(B).

[0050] A user can set an entry-prohibited area by operating the UI of the client device 190. In the imaging device 100 of this embodiment, the current position of a subject within the shooting angle of view is detected with respect to the set entry-prohibited area, and it is determined whether the subject has entered the entry-prohibited area. Here, image recognition technology such as machine learning and DL (Deep Learning) is used to detect coordinate information of the detected subject, and it is compared with the coordinate information of the set entry-prohibited area to determine whether the subject has entered the entry-prohibited area.

[0051] The intrusion forbidden area can be set, for example, by the user operating a UI or the like to place a plurality of designated points (designated positions) at arbitrary positions. It can also be set by drawing a judgment boundary line, or by a combination of these. Here, in the example of FIG. 3, the arbitrary designated points are represented by 301 to 307, and the judgment boundary line is represented by 311. Furthermore, the user can set an intrusion forbidden area 321 and an allowed area (intrusion allowed area) 322 on one side and the other side of each judgment boundary line (with the judgment boundary line in between). In the example of FIG. 3(A), the area on the left side of the judgment boundary line 311 is set as the intrusion forbidden area 321, and the area on the right side of the judgment boundary line 311 is set as the allowed area 322.

[0052] Furthermore, when multiple judgment boundaries are connected to surround an arbitrary area, the interior of the area (inside the area) can be set as either a permitted area in which passage is permitted, or a prohibited area. Furthermore, the judgment boundary can be selected to be set in space or on the ground. For example, when the judgment boundary is set on the ground, a judgment is made on the judgment boundary taking into account the depth direction within the angle of view of the imaging device 100. The judgment process will be described later.

[0053] 3A shows a judgment boundary line 311 when setting an intrusion-prohibited area in space. A user can set a judgment boundary line when setting an intrusion-prohibited area 321 in space by operating a UI or the like to place (set) an indication point 301 and an indication point 302 on a captured image.

[0054] In Fig. 3(B), pointer points 303 to 307 are placed on the ground to set an intrusion-prohibited area 323. In Fig. 3(B), an area outside the intrusion-prohibited area 323 is set as an allowed area 324.

[0055] By increasing the number of designated points (designated positions), it is possible to set more complex shapes. There is no particular restriction on the number of designated points that can be set, but since the judgment process is performed for each judgment boundary line, it is preferable to limit the number to a certain extent in order to reduce the load of calculation processing.

[0056] Here, in order to detect the intrusion of a subject into the intrusion-prohibited area set on the ground, it is preferable to detect the intrusion of a person, who is the subject, at a point where the subject is in contact with the ground. That is, when judging the intrusion into the intrusion-prohibited area set at a position that is the ground or floor, the control unit 130 sets the position of the subject's feet as a feature point (feature position). Then, when it is detected that the feature point has intruded into the intrusion-prohibited area set at a position that is the ground or floor (whether the feature point is included in the intrusion-prohibited area), it is judged that the subject has intruded into the intrusion-prohibited area. Alternatively, when it is detected that the feature point has straddled the judgment boundary line, it is judged that the subject has intruded into the intrusion-prohibited area set at a position that is the ground or floor. When the subject is detected by a rectangular frame, the bottom side of the rectangular frame is the contact point of the foot position with the ground. Therefore, when a point, a line segment, or an area on the bottom side of the rectangular frame intrudes (enters) into the prohibited area, it is judged that the subject has intruded into the prohibited area.

[0057] If the subject is in front of (near) the intrusion-prohibited area, there is a high possibility that the upper part of the subject will be photographed overlapping the intrusion-prohibited area. If the detected rectangular frame in the photographed image is judged to overlap the intrusion-prohibited area only based on whether it overlaps with the intrusion-prohibited area, an erroneous judgment will be made. However, if the intrusion into the intrusion-prohibited area is judged to be set at the bottom side of the rectangular frame, that is, the ground or floor part at the above-mentioned characteristic point (characteristic position), such an erroneous judgment can be prevented.

[0058] FIG. 3C shows a process for determining an intrusion into an intrusion-prohibited area by a combination of a determination boundary line 311 set in the space and a determination area 325 set on the ground. The determination area 325 is synonymous with the intrusion-prohibited area, and the method for setting the determination area 325 is the same as the method for setting the intrusion-prohibited area described above. Here, if it is determined that the subject 203 has intruded into the intrusion-prohibited area when the subject has intruded into the determination area 325 and a part of the subject has crossed the determination boundary line 311. In other words, if a subject that is not in the determination area 325 crosses the determination boundary line 311, it is not determined that the subject has entered the intrusion-prohibited area. In other words, it is possible to prevent erroneous detection when the photographing angle of view of the imaging device 100 is crossed in front of the intrusion-prohibited area. In this way, by performing a determination by a combination of areas, it is possible to set a finer area.

[0059] 3(A) to 3(C), when it is determined that at least a part of the detection area (rectangular frame) of the subject detected by analyzing the captured image has crossed the determination boundary line, the imaging device 100 determines that the subject has entered the entry prohibited area. Then, a warning is notified to the client device 190.

[0060] The determination of intrusion into the intrusion forbidden area may be made based on the overlapping area between the intrusion forbidden area and the detection area (rectangular frame) of the subject. That is, if at least a part of the detection area of ​​the subject overlaps with the intrusion forbidden area on the image, it may be determined that the subject has intruded into the intrusion forbidden area. It may also be determined that the subject has intruded into the intrusion forbidden area if at least a part of the detection area of ​​the subject overlaps with the judgment boundary line. It may also be determined that the subject has intruded into the intrusion forbidden area if it is detected that the coordinates of the subject have crossed the judgment boundary line. Either method can produce equivalent results.

[0061] Next, a series of processing steps in which the imaging device 100 detects whether an object (subject) has entered an intrusion prohibited area will be described with reference to Fig. 4. Fig. 4 is a flowchart illustrating the analysis processing operation of the imaging device 100 of this embodiment. Each operation (process) shown in the flowchart of Fig. 4 is controlled by the control unit 130 of the imaging device 100 executing a program stored in the storage unit 150. Also, by adding an S to the beginning of each process (step), the notation of the process (step) will be omitted.

[0062] First, as a preparation before starting the process shown in FIG. 4, a predetermined shooting angle of view is set in the imaging device 100. Furthermore, the entry-prohibited area is set as explained in FIG. 3. Furthermore, an analysis processing model for detecting the subject is set. Here, the entry-prohibited area is used for the sake of explanation, but instead of the entry-prohibited area, an intrusion detection area may be set as an area where intrusion is permitted (permitted area) or an area where intrusion is detected. Once these settings are completed, a process is performed to determine whether or not the subject has entered the entry-prohibited area for each shooting (each time a captured image is obtained) according to the process shown in FIG. 4. Note that the entry of the subject into the entry-prohibited area means that the subject has crossed the determination boundary line.

[0063] In S401, the control unit 130 sets the shooting mode to a shooting mode for a visible light image (first image) and starts shooting using visible light (normal shooting). When visible light shooting starts, reading of an image signal by the visible light imaging element 121 starts. The read image signal is then developed by the image processing unit 140 and stored in the storage unit 150 as predetermined image data (visible light image data). The image signal by the visible light imaging element 121 is then processed by the image processing unit 140 and transmitted to the analysis processing unit 160.

[0064] In S402, the analysis processing unit 160 detects the subject from the visible light image data acquired in S401 and transmitted from the control unit 130, and acquires the area in which the subject is detected as a detection area (first area). Then, the control unit 130 judges whether the subject has entered a predetermined area. In other words, the control unit 130 judges whether the detection area (first area) of the subject detected from the visible light image data has entered a previously set entry-prohibited area. As an example, for the sake of simplicity, the detection area detected based on the subject is detected as a rectangular frame, and based on the coordinate information of the detected detection area, it is judged whether at least a part of the rectangular frame has entered the entry-prohibited area (crossed the judgment boundary line). That is, the control unit 130 judges whether the detection area (first area) of the subject has entered the entry-prohibited area by comparing the coordinate information of the detection area of ​​the detected subject with the coordinate information of the set entry-prohibited area.

[0065] If it is determined that the detection area of ​​the detected subject has not entered the intrusion forbidden area (has not crossed the judgment boundary line) (No in S402), the subject is not in the intrusion forbidden area and the process proceeds to S407. In other words, unless the subject is approaching the judgment boundary line, neither invisible light is emitted nor an invisible light image is captured. On the other hand, if it is determined that the detection area of ​​the detected subject has entered the intrusion forbidden area (has crossed the judgment boundary line) (Yes in S402), the process proceeds to S403.

[0066] In S403, when starting shooting with invisible light (infrared light shooting), the control unit 130 switches the shooting mode of the imaging device 100 to a shooting mode for shooting an invisible light image (second image). Then, the control unit 130 sends an instruction to the invisible light illumination unit 80 to start emitting invisible light. In this process, infrared light is emitted as invisible light, so the invisible light image becomes an infrared image. In this way, the irradiation of invisible light is performed when it is determined in the determination process of S402 that the detection area (first area) of the subject has entered the entry prohibited area.

[0067] In S404, the control unit 130 starts capturing images using non-visible light. When non-visible light capturing is started, the non-visible light imaging element 122 starts reading out an image signal. The read-out image signal is then developed by the image processing unit 140 and stored in the storage unit 150 as predetermined image data (non-visible light image data). The image signal is then processed by the image processing unit 140 from the non-visible light imaging element 122 and transmitted to the analysis processing unit 160. Here, it is desirable to configure the subject position on the developed visible light image to match the subject position (subject coordinates) on the non-visible light image. In this way, the non-visible light image (second image) is acquired after non-visible light is irradiated.

[0068] The control unit 130 may control the non-visible light imaging element 122 to stop driving or to enter a power saving state while visible light imaging is being performed, thereby making it possible to reduce the driving power of the imaging element.

[0069] In S405, the analysis processing unit 160 first detects the subject from the non-visible light image data acquired in S401 and transmitted from the control unit 130, and specifies the area in which the subject is detected as a detection area (second area). Details of the detection process at this time will be described later. Then, the control unit 130 judges whether the subject has entered a predetermined area. In other words, the control unit 130 judges whether the detection area (second area) of the subject detected from the non-visible light image data has entered a previously set entry-prohibited area. Note that in this process, similarly to S402, the detection area detected based on the subject is specified as a rectangular frame, and based on the coordinate information of the specified detection area, it is judged whether at least a part of the rectangular frame has entered the entry-prohibited area (crossed the judgment boundary line). That is, the control unit 130 judges whether the detection area (second area) of the subject has entered the entry-prohibited area by comparing the coordinate information of the detection area of ​​the subject with the coordinate information of the set entry-prohibited area. In this way, when the detection area (first area) of the subject identified from the visible light image has entered the intrusion prohibited area, the control unit 130 determines whether the detection area (second area) of the subject identified from the non-visible light image has entered the intrusion prohibited area.

[0070] Here, if it is determined that the detection area of ​​the detected subject has not entered the intrusion forbidden area (has not crossed the judgment boundary line) (No in S406), the subject has not entered the intrusion forbidden area and the process proceeds to S407. This means that the intrusion of the detection area into the intrusion forbidden area detected in S402 has occurred due to a false detection caused by the shadow of the subject. In this case, the control unit 130 may determine that the intrusion of the subject into the intrusion forbidden area is the intrusion of the shadow of the subject, and output information indicating that it is a false detection. That is, if the control unit 130 determines that the detection area (first area) detected from the visible light image has entered the intrusion forbidden area, but determines that the detection area (second area) specified from the invisible light image has not entered the intrusion forbidden area, it determines that it is a false detection and outputs information to that effect. On the other hand, if it is determined that the detection area of ​​the detected subject has entered the intrusion forbidden area (crossed the judgment boundary line) (Yes in S406), the process proceeds to S406.

[0071] In S406, the control unit 130 notifies the client device 190 of a predetermined warning, since the detection area of ​​the subject detected from the non-visible light image in S405 has entered the entrance forbidden area (crossed the determination boundary line).

[0072] In S407, the control unit 130 judges whether or not to end the process. As an example, it judges whether a control signal (command) related to an operation to stop shooting of the imaging device 100 or the process related to this flow has been received from the client device 190, or whether an operation to turn off the power supply of the imaging device 100 has been performed. Then, if it is judged that the process is to be ended, the process in this flow is ended. On the other hand, if it is judged that the process is not to be ended, the process returns to S401, and the same process is repeated while continuing shooting.

[0073] Here, the analysis processing unit 160 performs the analysis processing on each captured image frame data, but if there is almost no change in the captured image, the number of frames of the captured image on which the analysis processing is performed may be thinned out. For example, it is also effective to configure the analysis processing to be performed at predetermined time intervals, such as analyzing the captured image every second or every few seconds.

[0074] Also, while there are lighting means that respond sufficiently quickly to the frame rate at which a captured image is read out, such as a camera strobe, there are also lighting means that respond slowly. In the case of such lighting that takes time to switch, a judgment boundary for lighting may be provided to start irradiating infrared lighting one step before the judgment boundary for detecting intrusion. When the subject approaches the intrusion-prohibited area, after crossing the boundary for lighting, the subject reaches the actual judgment boundary after a predetermined time. That is, a pre-detection area (pre-boundary) is set in advance in an area that is outside the intrusion-prohibited area on the image. Then, when the control unit 130 determines that the detection area of ​​the subject has crossed this pre-detection area (pre-boundary), it sends an instruction to the invisible light illumination unit 80 to irradiate invisible light. Alternatively, the distance between the subject and the judgment boundary may be observed, and when it is determined that the subject has approached the judgment boundary by a predetermined distance, invisible light may be irradiated. With such a configuration, it is possible to buy time for the illumination to stabilize before the subject reaches the actual judgment boundary after starting to irradiate invisible light.

[0075] Fig. 5 is a schematic diagram showing the detection results of a subject in a visible light image and a non-visible light image. Fig. 5(A) shows a rectangular frame representing a subject detected in a visible light image captured by the imaging device 100 in S402 superimposed on the captured image. Rectangular frame 214 shows the result of detecting a person 501, and rectangular frame 215 shows the result of detecting a shadow 502 of the person. Fig. 5(A) shows a state in which rectangular frame 215, which is the result of erroneously detecting the shadow of a subject (person 501), is inside both the intrusion forbidden area 211 and the intrusion forbidden area 213, and therefore it is erroneously determined that person 501 has intruded into the intrusion forbidden area.

[0076] 5(B) shows a state in which the determination result of the detection of a subject is superimposed on the non-visible light image captured by the imaging device 100 in S405. Person 504 is the same subject as person 501. The non-visible light image is a grayscale image having only luminance values.

[0077] Since person 504 is illuminated by infrared illumination emitted by non-visible light illumination unit 80 of imaging device 100, shadow 505 that occurs on person 504 occurs behind person 504. In addition, since it is an non-visible light image, the influence of the lighting installed in the venue, which mainly contains visible light wavelengths, is reduced, and the occurrence of shadow 502 is suppressed. Therefore, rectangular area 506, which is the detection result on the non-visible light image, can be detected as a rectangular area appropriate for the size of person 504 (minimizing the influence of shadow 505 of person 504).

[0078] 5(B), the bottom side of rectangular area 506 (feature point at the feet of person 504) is not within intrusion forbidden area 213 set on the ground, so control unit 130 determines that person 504 has not entered the intrusion forbidden area. In addition, the left side of rectangular area 506 is not within intrusion forbidden area 211, so control unit 130 determines that person 504 has not entered the intrusion forbidden area set around the exhibit. By using the invisible light image in this way, it can be confirmed that person 504 has not entered intrusion forbidden area 211 or intrusion forbidden area 213 as a final determination result. This makes it possible to prevent erroneous detection.

[0079] FIG. 5C shows an example of a case where an obstacle with high reflectivity to infrared light is present. The object 507 shown in FIG. 5C is an obstacle with high reflectivity placed around the person 504, and is not a person. In such a case, the analysis processing unit 160 may erroneously detect the object 507 as a subject (person), detect a detection area, and output this. The detection area in which the object 507 is detected may be a detection area 509 in which the object 507 is detected as an independent object, or a detection area 508 in which the person 504 and the object 507 are combined. As shown in FIG. 5C, the closer the distance between the person 504 and the object 507 is, the higher the possibility of erroneous detection.

[0080] Hereinafter, the flow of subject detection processing for avoiding this erroneous detection will be described with reference to FIG. 6. FIG. 6 is a flowchart for explaining the analysis processing operation of the imaging device of this embodiment. Each operation (process) shown in the flowchart of FIG. 6 is controlled by the control unit 130 of the imaging device 100 executing a program stored in the storage unit 150. Also, by adding an S to the beginning of each process (step), the process (step) will be omitted. Note that FIG. 6 performs the same process as the processing flow of FIG. 4, except that S605 is added. Therefore, detailed description of the same process as the process of FIG. 4 will be omitted.

[0081] In S601, the control unit 130 sets the imaging mode to a visible light image (first image) imaging mode, starts visible light imaging, and acquires a visible light image. Thereafter, the control unit 130 transmits the visible light image data to the analysis processing unit 160.

[0082] In S602, the analysis processing unit 160 detects a subject from the acquired visible light image data, and acquires the area in which the subject is detected as a detection area (first area). Thereafter, the control unit 130 determines whether or not the detection area of ​​the detected subject has entered the entry-forbidden area. If it is determined that the detection area of ​​the detected subject has not entered the entry-forbidden area (No in S602), the subject has not entered the entry-forbidden area, and the process proceeds to S608. On the other hand, if it is determined that the detection area of ​​the detected subject has entered the entry-forbidden area (Yes in S602), the process proceeds to S603.

[0083] In S603, the control unit 130 switches the shooting mode to a non-visible light image shooting mode and sends an instruction to the non-visible light illumination unit 80 to start irradiating infrared light. In S604, the control unit 130 starts shooting with non-visible light (infrared light shooting) and obtains a non-visible light image.

[0084] In S605, the analysis processing unit 160 compares the detection result (detection area) of the subject in the visible light image acquired by visible light photography in S601 with the detection result (detection area) of the subject in the non-visible light image acquired by non-visible light photography in S604. Then, the analysis processing unit 160 derives and acquires detection area 3 as a new detection area. The process of S605 will be described in detail later.

[0085] In S606, the control unit 130 judges whether the detection area 3 acquired in S605 has entered a predetermined area. In other words, it judges whether the detection area 3 has entered a preset intrusion-prohibited area (crossed the judgment boundary line). If it is judged that the detection area 3 has not entered the intrusion-prohibited area (has not crossed the judgment boundary line) (No in S606), it is determined that the subject has not entered the intrusion-prohibited area and the process proceeds to S608. On the other hand, if it is judged that the detection area 3 has entered the intrusion-prohibited area (has not crossed the judgment boundary line) (Yes in S606), the process proceeds to S607.

[0086] In S607, the control unit 130 determines that the detection area 3 has crossed the judgment boundary line (the subject has entered the entry-forbidden area), and notifies the client device 190 of a predetermined warning.

[0087] In S608, the control unit 130 judges whether or not to end the process. As an example, it judges whether a control signal (command) related to an operation to stop shooting of the imaging device 100 or the process related to this flow has been received from the client device 190, or whether an operation to turn off the power supply of the imaging device 100 has been performed. Then, if it is judged that the process is to be ended, the process in this flow is ended. On the other hand, if it is judged that the process is not to be ended, the process returns to S601, and the same process is repeated while continuing shooting.

[0088] Fig. 5(D) is a schematic diagram showing the process of S605. Fig. 5(D) shows a state in which detection area 508 detected in Fig. 5(C) is compared with detection area 510 detected in Fig. 5(A). Fig. 5(E) is a schematic diagram showing the process in which analysis processing unit 160 determines rectangular area 511, which is a rectangular area suitable for the size of person 501.

[0089] Detection area 510 is a rectangular area detected from the visible light image, and is detected as an area including person 501 and shadow 502 of person 501. Detection area 508 is a rectangular area detected from the non-visible light image, and is detected as a rectangular area including person 504 and object 507. Here, analysis processing unit 160 determines rectangular area 511, which is an overlapping area between rectangular area 508 and rectangular area 510, as a new detection area 3. Detection area 3 is an area smaller than detection area (first area) 510 and detection area (second area) 508.

[0090] The rectangular area 511 is detected as a rectangular area suitable for the size of the person 504. Then, the analysis processing unit 160 determines whether the determined rectangular area 511 encroaches on the intrusion forbidden areas 211 and 213, thereby making it possible to determine whether the subject (person 501) has intruded into the intrusion forbidden areas.

[0091] As described above, by performing the processing shown in the flowchart in FIG. 6 using the imaging device 100, it is possible to reduce erroneous detections and determine an area closer to the size of the person 504 as the detection area (rectangular area) for the subject.

[0092] As described above, the shadow cast on the subject changes when infrared illumination is applied. This reduces the effect of illumination in the installation environment, and improves the accuracy of subject detection (whether or not the subject has entered the intrusion-prohibited area) relative to the judgment boundary line. In addition, since invisible light illumination is not applied until the subject enters or approaches the intrusion-prohibited area, it is possible to reduce the power consumption of the imaging device 100.

[0093] Next, a further description will be given of the subject region detected by the analysis processing unit 160. Fig. 7 is a diagram showing an example of a process of detecting a rectangular region by performing object detection as the analysis process of the analysis processing unit 160, and a process of detecting the region shape of the object by segmentation.

[0094] The analysis processing unit 160 may specify the rectangular area using a trained model that has been trained to specify a rectangular area corresponding to an object such as a person or an animal when an image is input. Examples of the trained model include R-CNN (Region with Convolutional Neural Network), SSD (Single Shot Detector), and YOLO (You Only Look Once).

[0095] Alternatively, the subject may be identified by a trained model trained to extract the location and region of each individual object detection, such as instance segmentation.

[0096] The detection result 701 is a result of detecting the subject as a rectangular region. The detection results 704 to 705 are the results of the analysis processing unit 160 detecting the subject in the rectangular region of the detection result 701 by further combining a method such as difference detection to detect a region in which the subject exists on four sides of the rectangular region. For example, if a region such as the intrusion forbidden region 710 is set on the ground, when the rectangular region of the detection result 701 is judged, a part of the bottom side of the rectangular region is judged to have invaded the intrusion forbidden region. However, when the actual subject's foot region is judged as in the detection result 705, it is not judged to have invaded the intrusion forbidden region, and it is possible to prevent erroneous detection of an intrusion region. On the other hand, the detection result 702 is a result of detecting the shape of the subject as a region by instance segmentation or the like. As with the rectangular region, it is possible to judge an intrusion into the region by the detection result 702.

[0097] Detection results 731-739 in Fig. 7(B) show the relationship between each detected subject area and the set entry-restricted area 721. Detection results 734-739 are detection results including the shadow of a person. Detection results 737-739 are detection results obtained by analyzing the infrared photographed image with infrared lighting irradiated.

[0098] When a judgment is made on the entry-forbidden area based on the foot position of the detection result, the detection result 735 is judged to be a no-entry because the bottom end of the shadow is outside the entry-forbidden area. On the other hand, the detection result 738 obtained by analyzing the invisible light image result can be judged to be an entry into the entry-forbidden area because the position of the shadow changes due to the irradiation of infrared lighting. In other words, the possibility of false detection changes depending on the combination of the analysis model used for detection and the setting of the entry-forbidden area. In other words, regardless of which analysis process is selected, it is clear that irradiating invisible light lighting is effective because the influence of the shadow can be reduced by irradiating invisible light lighting.

[0099] Fig. 8 is a flowchart showing a process in which the analysis process is changed between a visible light image and a non-visible light image. Each operation (process) shown in the flowchart in Fig. 8 is controlled by the control unit 130 of the imaging device 100 executing a program stored in the storage unit 150. Moreover, each process (step) is represented by adding an S to the beginning, so that the notation of the process (step) is omitted. In Fig. 8, the same process is performed as in the process flow in Fig. 4, except that S802 and S806 are added. Therefore, a detailed description of the same process as in Fig. 4 will be omitted.

[0100] In S801, the control unit 130 sets the imaging mode to a visible light image (first image) imaging mode, starts visible light imaging, and acquires a visible light image. Thereafter, the control unit 130 transmits the visible light image data to the analysis processing unit 160.

[0101] In S802, the analysis processing unit 160 sets an analysis processing model for the visible light image acquired in S801. In this process, a pre-trained analysis processing model for object detection is selected and set in the analysis processing unit 160. The analysis processing model to be used and setting values ​​required for executing the analysis model are stored in advance in the storage unit 150. By setting data read from the storage unit 150 for each analysis processing function to be selected in the analysis processing unit 160, a plurality of different analysis processes can be executed.

[0102] In S803, the analysis processing unit 160 detects an object from the visible light image data using the analysis processing model set in S802, and obtains the area where the object is detected as a detection area (first area). Then, the control unit 130 judges whether the object has entered a predetermined area. In other words, the control unit 130 judges whether the detection area (first area) of the detected object has entered a previously set intrusion prohibited area. If it is judged that the detection area of ​​the detected object has not entered the intrusion prohibited area (No in S803), the object does not enter the intrusion prohibited area, and the process proceeds to S809. On the other hand, if it is judged that the detection area of ​​the detected object has entered the intrusion prohibited area (Yes in S803), the process proceeds to S804. Note that the object may be detected using analysis parameters for the visible light image acquired in S801, and the area where the object is detected may be obtained as a detection area (first area).

[0103] In S804, the control unit 130 switches the shooting mode to a non-visible light image shooting mode, sends an instruction to the non-visible light illumination unit 80, and starts irradiating non-visible light. In S805, the control unit 130 starts shooting with non-visible light (infrared light shooting) and obtains a non-visible light image.

[0104] In S806, the analysis processing unit 160 sets an analysis processing model for the non-visible light image acquired in S805. In this process, the analysis processing model learned from the non-visible light image is selected and set in the analysis processing unit 160.

[0105] In S807, the analysis processing unit 160 detects the subject from the non-visible light image using the analysis processing model set in S806, and specifies the area where the subject is detected as the detection area (second area). Then, the control unit 130 determines whether the subject has entered a predetermined area. In other words, the control unit 130 determines whether the detection area (second area) of the subject detected from the non-visible light image data has entered a previously set intrusion forbidden area. If it is determined that the detection area of ​​the subject has not entered the intrusion forbidden area (No in S807), the subject does not enter the intrusion forbidden area, and the process proceeds to S809. On the other hand, if it is determined that the detection area of ​​the subject has entered the intrusion forbidden area (Yes in S807), the process proceeds to S808. Note that the subject may be detected using the analysis parameters for the non-visible light image acquired in S805, and the area where the subject is detected may be acquired as the detection area (second area).

[0106] In S808, the control unit 130 notifies the client device 190 of a predetermined warning, since the detection area of ​​the subject detected from the non-visible light image in S805 has entered the entrance forbidden area (crossed the determination boundary line).

[0107] In S809, the control unit 130 judges whether or not to end the process. As an example, it judges whether a control signal (command) related to an operation to stop shooting of the imaging device 100 or the process related to this flow has been received from the client device 190, or whether an operation to turn off the power supply of the imaging device 100 has been performed. Then, if it is judged that the process is to be ended, the process in this flow is ended. On the other hand, if it is judged that the process is not to be ended, the process returns to S801, and the same process is repeated while continuing shooting.

[0108] Visible light images have color information, whereas invisible light images do not. In many cases, the trained model for the visible light environment is trained in advance using visible light images. For this reason, it is desirable that the analysis processing model for invisible light images selected and set in S806 is adjusted to match the invisible light images. For example, even if the same model as the analysis processing model for visible light images is used, it is possible to improve the detection accuracy by changing to parameters that have been re-trained in advance using infrared shooting image data groups. In addition, the image data itself is different, with the visible light image having a structure with three-dimensional color pixel information, while the invisible light image has a structure with only brightness pixel information. Therefore, as a pre-analysis process, the brightness value of the image data of the invisible light image is assigned to the three values ​​of RGB, and a process that seemingly matches the data structure is added, so that the same model can be used.

[0109] Furthermore, for example, the type of analysis process itself may be changed according to the lighting conditions in the usage environment, such as by using an object detection model for visible light images and detecting subjects through segmentation as analysis processing for invisible light images.

[0110] Next, a different configuration of the imaging device 100 in this embodiment will be described. Fig. 9 is a schematic diagram showing another configuration of the imaging section 110. Fig. 9(A) shows an example of a configuration in which the imaging section 110 includes two different optical sections. In this embodiment, the imaging section 110 is described as being configured with one optical section, but as shown in Fig. 9(A), it may be configured with two different optical sections.

[0111] 9(A), the visible light image that has passed through the optical unit 111 is received by the visible light imaging element 121 and output as image data, and the non-visible light that has passed through the optical unit 113 is received by the non-visible light imaging element 122 and output as image data. The image data output by the visible light imaging element 121 and the non-visible light imaging element 122 are each developed by the image processing unit 140. Here, by finding the correlation between the positions of the subject images formed on the visible light imaging element 121 and the non-visible light imaging element 122 and aligning them on the captured image after development, it becomes possible to accurately determine the detection position of the subject image even if the optical units are different.

[0112] FIG. 9B shows an example of the configuration of the imaging unit 110 including the IR cut filter 114 and its insertion / removal mechanism. The imaging element 123 is sensitive to both visible light and infrared light. When reading out a visible light image, the IR cut filter 114 is moved onto the optical path to cut out infrared wavelengths. When reading out an infrared light image, infrared illumination is applied and the IR cut filter 114 is removed from the optical path. The IR cut filter 114 may be an IR band pass filter. In this case, the insertion / removal relationship of the filter is reversed between reading out a visible light image signal and reading out an infrared light image signal. Alternatively, a combination of the IR cut filter 114 and an IR band pass filter may be used.

[0113] 9(C) shows an example in which the imaging unit 110 has an imaging element 124 equipped with IR pixels having high light sensitivity to non-visible light images (infrared images) in addition to R, G, and B pixels having high light sensitivity to visible light. The imaging element 124 is configured to be capable of simultaneously or selectively reading visible light image data and non-visible light image data. Under visible light illumination, visible light image data can be generated by developing only the signals of the RGB pixels, and under infrared light illumination, infrared light image data can be generated by developing only the signals of the IR pixels.

[0114] 10 and 11 are schematic diagrams showing the timing at which the visible light imaging element 121 and the non-visible light imaging element 122 output image data captured by the visible light imaging element 121 and the non-visible light imaging element 122, and the timing at which it is determined that a subject detected in an image has crossed the determination boundary line. T in Fig. 10 and Fig. 11 indicates timing.

[0115] The image sensor captures images at a predetermined frame rate, and the captured images developed by the image processing unit 140 are processed by the analysis processing unit for each frame rate. Image 1001 represents image data in which it has been determined, as a result of analyzing the visible light image data, that the detection area of ​​the detected subject does not exceed the judgment boundary line. Image 1002 represents image data in which it has been determined, as a result of analyzing the visible light image data, that the detection area of ​​the detected subject has exceeded the judgment boundary line. Image 1003 represents image data in which it has been determined, as a result of analyzing the non-visible light image data, that the detection area of ​​the detected subject does not exceed the judgment boundary line. Image 1004 represents image data in which it has been determined, as a result of analyzing the non-visible light image data, that the detection area of ​​the detected subject has exceeded the judgment boundary line.

[0116] 10A shows that only visible light shooting is performed during the period from T1 to T2. When it is detected at T2 that the detection area of ​​the subject detected from the visible light image has crossed the judgment boundary line, irradiation with non-visible light illumination is started at T3, and reading of non-visible light image data by non-visible light imaging element 122 is started.

[0117] If it is determined at T4 that the detection area of ​​the detected subject in the invisible light image has also exceeded the determination boundary line, it is determined that an intrusion of a subject into the intrusion prohibited area has been detected, and a warning is sent to the client device 190. Then, at T5, it is determined that the detection area of ​​the detected subject in the invisible light image has moved outside the intrusion prohibited area. Furthermore, if it is determined at T6 that the detection area of ​​the detected subject in the visible light image has also moved outside the intrusion prohibited area, the irradiation of invisible light is turned off.

[0118] 1 and the imaging units 110 shown in Figures 9(B) and 9(C), it is also possible to suspend the non-visible light imaging element 122 of the non-visible light image until it is determined that the determination boundary line has been crossed on the visible light image data. This not only stops the irradiation of non-visible light, but also stops the imaging element and further stops the development process, making it possible to further reduce power consumption.

[0119] In Figure 10(B), as in Figure 10(A), at timing T2, it is detected that the detection area of ​​the subject detected in the visible light image has crossed the judgment boundary line, and at T3, irradiation with non-visible light illumination begins, and reading of non-visible light image data by the non-visible light imaging element 122 begins.

[0120] T4 shows a case where the detection area of ​​the subject detected from the non-visible light image does not enter the intrusion prohibited area. Even though an intrusion into the intrusion prohibited area is detected in the visible light image, no intrusion is detected in the non-visible light image, so during that period, the two image pickup elements continue to read out pixel signals and the non-visible light illumination unit 80 also continues to drive, resulting in the state with the greatest power consumption.

[0121] Figures 11(A) and 11(B) show the case where visible light images and non-visible light images are read out alternately. The configuration of the imaging unit 110 in Figure 9(B) corresponds to the readout in Figure 11(A). Figure 11(B) shows the case where visible light images and non-visible light images are read out alternately. Furthermore, in Figure 11(B), the reading of visible light image data by the visible light imaging element 121 is stopped during the period from T4 to T5.

[0122] For example, when only a single subject is detected, the operation of the visible light imaging element 121 may be stopped in this manner. In the readout of Fig. 10(A), when it is determined that the subject detection area in the non-visible light image has crossed the judgment boundary line or entered the intrusion prohibited area in the section T4 to T5, the operation of the visible light imaging element 121 may be stopped as in Fig. 11(B). In this way, by stopping the operation of the visible light imaging element 121 and also stopping the development process of the visible light image, it is possible to further reduce power consumption.

[0123] On the other hand, when multiple subjects are near the judgment boundary line or the intrusion prohibited area, it is preferable to change the judgment conditions for each subject. Therefore, it is preferable to simultaneously read out the visible light image data by the visible light imaging element 121 and the non-visible light image data by the non-visible light imaging element 122, because this makes it possible to improve the detection accuracy.

[0124] As described above, even if the configuration of the imaging unit 110 in FIG. 1 is changed to the configuration shown in FIG. 9, it is possible to reduce the power consumption of the imaging device 100 while improving the detection accuracy of whether the subject detection area has entered the intrusion prohibited area.

[0125] 12 is a diagram showing a configuration in which an analysis processing unit is provided in an external information device such as a cloud server connected to the imaging device 100. In this configuration, the above-mentioned analysis processing performed by the imaging device 100 can be performed by an external information device communicably connected via a network 180. The following will describe this with reference to FIG.

[0126] The analysis processing unit 182 is an analysis processing unit provided in the cloud server 181. The analysis processing unit 196 is an analysis processing unit provided in the client device 190. In this case, the imaging device 100 transmits (outputs) the acquired visible light image and non-visible light image to an external information device, and receives the subject detection result (subject detection area) after analysis from the external information device.

[0127] The image capturing device 100 uses the received detection area (first area, second area) of the subject to determine whether the detection area has entered the intrusion forbidden area. If it is determined that the detection area of ​​the subject has entered the intrusion forbidden area (crossed the determination boundary line), the image capturing device 100 notifies the client device 190 of a predetermined warning.

[0128] Furthermore, the image capture device 100 may transmit setting information of the intrusion-prohibited area and the judgment boundary line to an external information device, and cause the external information device to carry out the judgment process of intrusion into the intrusion-prohibited area. In this case, the image capture device 100 only captures visible light images and invisible light images, controls the irradiation of invisible light (infrared light), and communicates via a network. However, it is possible to obtain the same effect as that described above in detecting whether the detection area of ​​the subject has intruded into the intrusion-prohibited area.

[0129] Although the external information device is connected via a network here, the image capturing apparatus 100 may also process the data in an external information device connected via an external connection I / F such as a USB provided in the image capturing apparatus 100 .

[0130] As described above, the imaging device 100 in this embodiment switches to imaging with invisible light only at the timing of detecting an intrusion into a set prohibited area, so that attributes such as the color of clothing can be distinguished during normal imaging. In particular, when it is necessary to determine that a person not wearing designated work clothes has intruded into a specific area, such as in a factory, erroneous detection can be reduced by performing analysis processing including color information. In addition, since the imaging device 100 does not need to keep the invisible light illumination on all the time, it is possible to reduce the power consumption of the imaging device 100. Furthermore, it is possible to improve detection accuracy by switching the analysis processing to a pre-learned model according to the light source.

[0131] Fig. 13 is a diagram showing an example of using the imaging device of this embodiment in a different installation environment. The example of Fig. 13 shows a situation in which a customer reaches out to a product displayed on a shelf in a small store such as a convenience store or a private shop.

[0132] The imaging device 252 is a small camera installed on a product shelf or the like in a store. The imaging device 253 is a camera installed on a ceiling or the like in a store. The functional configurations of the imaging devices 252 and 253 are similar to that of the imaging device 100 shown in FIG. 1, so a description thereof will be omitted.

[0133] A shadow 254 is generated on a customer 255 by lighting 251 in the store. In FIG. 13, a commodity detection area 256 is set as a contact determination area for determining contact with a commodity. The commodity detection area 256 may be set by a user operating the UI of the client device 190 as described above, or may be automatically set by a control unit in the imaging device 252 or the imaging device 253 detecting a commodity in an image. In this case, the control unit may set the commodity detection area 256 based on, for example, the position (coordinates) of one or more commodities in the image and the size (size) or position (coordinates) of the subject. The commodity detection area 256 shown here is an area similar to the entry prohibited area 321 and the entry prohibited area 323 described in FIG. 3, although the name is different.

[0134] The commodity detection area 256 is an area for detecting that the customer 255 picks up a commodity. Here, when the shadow 254 enters the commodity detection area 256 (when the shadow area overlaps with the commodity detection area), the action of the customer 255 picking up a commodity may be erroneously detected. Therefore, at the timing when the intrusion of the subject (customer 255) into the commodity detection area 256 is detected, either the imaging device 252 or the imaging device 253 irradiates invisible light and detects the intrusion into the commodity detection area 256 by the invisible light image. This makes it possible to prevent erroneous detection caused by the shadow 254. There are many similar use cases not only in stores but also in small spaces where shadows may occur, such as warehouses and the internal space of commodity shelves, and this example can be applied to any of them.

[0135] Therefore, by applying this example and detecting a subject with the illumination unit of the imaging device 252 or the imaging device 253 turned on, it is possible to prevent erroneous detection in the same way as the imaging device 100 described above.

[0136] Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist of the present invention described in the claims.

[0137] Furthermore, the functions of the above-described embodiments are not only realized by the computer reading and executing the program code, but also include an operating system (OS) running on the computer performing all or part of the actual processing based on the instructions of the program code.

[0138] Furthermore, the functions of the above-described embodiments may be realized by the following method: Program code read from a recording medium is written into a memory provided in a function expansion card inserted into a computer or a function expansion unit connected to a computer. Then, based on the instructions of the program code, a CPU or the like provided in the function expansion card or function expansion unit performs some or all of the actual processing.

[0139] When the present invention is applied to the above-mentioned recording medium, the recording medium stores program codes corresponding to the respective flowcharts described above.

[0140] The disclosure of this embodiment includes the following configuration, method, program, and system.

[0141] (Configuration 1) an imaging means for acquiring a first image captured at a predetermined angle of view using a first wavelength and a second image captured at the predetermined angle of view using a second wavelength different from the first wavelength; a specifying means for specifying an area in the first image where a subject is detected as a first area, and for specifying an area in the second image where the subject is detected as a second area; a determination means for determining whether or not at least a part of the first area or the second area has entered a preset detection area, The imaging device, characterized in that the determination means, when it determines that at least a portion of the first area has entered the detection area, determines whether at least a portion of the second area has entered the detection area.

[0142] (Configuration 2) an irradiation means for irradiating the light of the second wavelength, The imaging device according to configuration 1, wherein the irradiation means irradiates light of the second wavelength after the determination means determines that at least a portion of the first region has entered the detection region.

[0143] (Configuration 3) 3. The imaging device according to configuration 2, wherein the imaging means obtains the second image after the light having the second wavelength is irradiated by the irradiating means.

[0144] (Configuration 4) The imaging device described in any one of configurations 1 to 3, characterized in that when the determination means determines that at least a portion of the first area is within the detection area and when the determination means determines that at least a portion of the second area is not within the detection area, it outputs information indicating that this is a false detection.

[0145] (Configuration 5) The imaging device described in any one of configurations 1 to 4, characterized in that the determination means determines that a shadow of the subject has entered the detection area when it determines that at least a portion of the first area has entered the detection area and when it determines that at least a portion of the second area has not entered the detection area.

[0146] (Configuration 6) the determining means determines whether the subject has entered the detection area by comparing coordinate information of the detection area with coordinate information of the first area; The imaging device according to any one of configurations 1 to 5, characterized in that it determines whether the subject has entered the detection area by comparing coordinate information of the detection area with coordinate information of the second area.

[0147] (Configuration 7) 7. The imaging device according to any one of configurations 1 to 6, wherein the first wavelength includes a wavelength of visible light, and the first image is a visible light image including the subject.

[0148] (Configuration 8) 8. The imaging device according to configuration 7, wherein the second wavelength includes a wavelength of invisible light, and the second image is a invisible light image including the subject.

[0149] (Configuration 9) the determining means sets a position of the subject's feet in the first area and the second area as a feature point indicating the presence of the subject; An imaging device described in any one of configurations 1 to 8, characterized in that, when the detection area is set on the ground, it is determined whether the first area or the second area enters the detection area set on the ground using the feature points.

[0150] (Configuration 10) the identifying means identifies a third region, which is a region where the first region and the second region overlap, based on the first region and the second region; 10. The imaging device according to any one of configurations 1 to 9, wherein the determining means determines whether or not at least a part of the third region has entered the detection region.

[0151] (Configuration 11) The imaging device described in any one of configurations 1 to 10, characterized in that the identification means identifies the subject, the first region, and the second region from the first image and the second image, respectively, by analyzing the first image and the second image using different analytical models or analytical parameters.

[0152] (Configuration 12) the imaging means includes a first imaging means for acquiring the first image captured at a predetermined angle of view using the first wavelength, and a second imaging means for acquiring the second image captured at the predetermined angle of view using the second wavelength, a control means for stopping the operation of the other imaging means or setting the other imaging means to a power saving state when either one of the first imaging means or the second imaging means is operating; 12. The imaging device according to any one of configurations 1 to 11.

[0153] (Configuration 13) The imaging device described in configuration 2, wherein the irradiation means irradiates light of the second wavelength when the determination means determines that the first area has entered a pre-detection area that is set further outside the detection area.

[0154] (Configuration 14) 14. The imaging device according to any one of configurations 1 to 13, further comprising a notification means for notifying a predetermined warning when the determination means determines that at least a portion of the second area has entered the detection area.

[0155] (Configuration 15) the imaging device outputs the first image and the second image to an external device; The imaging device described in any one of configurations 1 to 14, characterized in that the determination means determines whether at least a portion of the first area or the second area of ​​the subject detected by the external device has entered the detection area.

[0156] (Configuration 16) 16. The imaging device according to any one of configurations 1 to 15, wherein the specifying means specifies the first area and the second area as rectangular frames having coordinate information.

[0157] (Configuration 17) A control method for an imaging device, comprising: an imaging step of acquiring a first image captured at a predetermined angle of view using a first wavelength and a second image captured at the predetermined angle of view using a second wavelength different from the first wavelength; a specifying step of specifying an area in the first image where a subject is detected as a first area, and specifying an area in the second image where the subject is detected as a second area; a determination step of determining whether or not at least a part of the first area or the second area has entered a preset detection area, A control method for an imaging device, characterized in that in the determination process, when it is determined that at least a portion of the first area has entered the detection area, it is determined whether at least a portion of the second area has entered the detection area.

[0158] (Configuration 18) A computer is caused to operate as the imaging device according to any one of configurations 1 to 16. Program for.

[0159] (Configuration 19) An imaging system including an imaging device and an information processing device communicably connected to the imaging device, The imaging device includes: an imaging means for acquiring a first image captured at a predetermined angle of view using a first wavelength and a second image captured at the predetermined angle of view using a second wavelength different from the first wavelength; a specifying means for specifying an area in the first image where a subject is detected as a first area, and for specifying an area in the second image where the subject is detected as a second area; a determination means for determining whether or not at least a part of the first area or the second area has entered a preset detection area, the determination means determines whether at least a portion of the second area has entered the detection area when it has determined that at least a portion of the first area has entered the detection area; The information processing device includes: A setting means for setting the detection area is provided. 1. An imaging system comprising: [Explanation of symbols]

[0160] 80 Invisible light illumination section 100 Imaging device 110 Imaging unit 130 Control section 150 Storage section 160 Analysis processing section 170 Communications Department

Claims

1. an imaging means for acquiring a first image captured at a predetermined angle of view using a first wavelength and a second image captured at the predetermined angle of view using a second wavelength different from the first wavelength; a specifying means for specifying an area in the first image where a subject is detected as a first area, and for specifying an area in the second image where the subject is detected as a second area; a determination means for determining whether or not at least a part of the first area or the second area has entered a preset detection area, The imaging device, characterized in that the determination means, when it determines that at least a portion of the first area has entered the detection area, determines whether at least a portion of the second area has entered the detection area.

2. an irradiation means for irradiating the light having the second wavelength, 2. The imaging device according to claim 1, wherein the irradiating means irradiates the light of the second wavelength after the determining means determines that at least a part of the first area has entered the detection area.

3. 3. The imaging device according to claim 2, wherein the imaging means obtains the second image after the light having the second wavelength is irradiated by the irradiating means.

4. The imaging device according to claim 1, characterized in that when the determination means determines that at least a portion of the first area is within the detection area and when the determination means determines that at least a portion of the second area is not within the detection area, it outputs information indicating that this is a false detection.

5. 2. The imaging device according to claim 1, wherein the determination means determines that a shadow of the subject has entered the detection area when it determines that at least a portion of the first area has entered the detection area and when it determines that at least a portion of the second area has not entered the detection area.

6. the determining means determines whether the subject has entered the detection area by comparing coordinate information of the detection area with coordinate information of the first area; 2. The imaging device according to claim 1, wherein the coordinate information of the detection area is compared with the coordinate information of the second area to determine whether the subject has entered the detection area.

7. 2. The imaging device according to claim 1, wherein the first wavelength includes a wavelength of visible light, and the first image is a visible light image including the subject.

8. 8. The imaging device according to claim 7, wherein the second wavelength includes a wavelength of invisible light, and the second image is a invisible light image including the subject.

9. the determining means sets a position of the subject's feet in the first area and the second area as a feature point indicating the presence of the subject; The imaging device according to claim 1, characterized in that, when the detection area is set on the ground, the feature points are used to determine whether the first area or the second area has entered the detection area set on the ground.

10. the identifying means identifies a third area, which is an area where the first area and the second area overlap, based on the first area and the second area; 2. The imaging device according to claim 1, wherein the determining means determines whether or not at least a part of the third area has entered the detection area.

11. 2. The imaging device according to claim 1, wherein the identification means identifies the subject, the first region, and the second region from the first image and the second image, respectively, by analyzing the first image and the second image using different analytical models or analytical parameters.

12. the imaging means includes a first imaging means for acquiring the first image captured at a predetermined angle of view using the first wavelength, and a second imaging means for acquiring the second image captured at the predetermined angle of view using the second wavelength, a control means for stopping the operation of the other imaging means or setting the other imaging means to a power saving state when either one of the first imaging means or the second imaging means is operating; 2. The imaging device according to claim 1 .

13. 3. The imaging device according to claim 2, wherein the irradiation means irradiates light of the second wavelength when the determination means determines that the first area has entered a pre-detection area that is further set outside the detection area.

14. 2. The imaging apparatus according to claim 1, further comprising a notification unit that issues a predetermined warning when the determination unit determines that at least a part of the second area has entered the detection area.

15. the imaging device outputs the first image and the second image to an external device; 2 . The imaging apparatus according to claim 1 , wherein the determining unit determines whether or not at least a part of the first area or the second area of ​​the subject detected by the external device has entered the detection area.

16. 2. The imaging device according to claim 1, wherein the specifying unit specifies the first area and the second area as rectangular frames having coordinate information.

17. A control method for an imaging device, comprising: an imaging step of acquiring a first image captured at a predetermined angle of view using a first wavelength and a second image captured at the predetermined angle of view using a second wavelength different from the first wavelength; a specifying step of specifying an area in the first image where a subject is detected as a first area, and specifying an area in the second image where the subject is detected as a second area; a determination step of determining whether or not at least a part of the first area or the second area has entered a preset detection area, A control method for an imaging device, characterized in that, in the determination process, when it is determined that at least a portion of the first area has entered the detection area, it is determined whether at least a portion of the second area has entered the detection area.

18. A computer is caused to operate as the imaging device according to any one of claims 1 to 16. Program for.

19. An imaging system including an imaging device and an information processing device communicably connected to the imaging device, The imaging device includes: an imaging means for acquiring a first image captured at a predetermined angle of view using a first wavelength and a second image captured at the predetermined angle of view using a second wavelength different from the first wavelength; a specifying means for specifying an area in the first image where a subject is detected as a first area, and detecting the subject from the second image and specifying an area in the second image where the subject is detected as a second area; a determination means for determining whether or not at least a part of the first area or the second area has entered a preset detection area, the determination means determines whether at least a portion of the second area has entered the detection area when it has determined that at least a portion of the first area has entered the detection area; The information processing device includes: A setting means for setting the detection area is provided.

1. An imaging system comprising:

Images