Surveillance cameras and surveillance systems

Abstract

To provide a monitoring camera which can prevent erroneous recognition and erroneous detection by suppressing the influence of a virtual image due to radiation heat by using a far-infrared sensor.SOLUTION: A monitoring camera having a far-infrared sensor comprises a calibration unit 16 which performs calibration processing that suppresses the influence of a virtual image. The calibration unit 16 calculates a feature amount of a thermal image region by specifying the thermal image region from live image data from an image processing unit 12, determines the specified thermal image region as a virtual image when the calculated feature amount of the thermal image region correlates with a feature amount of the virtual image by using an identification determination model for estimating the virtual image with respect to a real image based on the feature amount, and determines the thermal image region as the real image when the feature amounts do not correlate with each other.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a surveillance camera and imaging device that use far-infrared rays, and in particular to a surveillance camera and imaging device that can suppress the influence of virtual images caused by radiant heat, provide clear far-infrared images, and prevent erroneous recognition and detection. [Background technology]

[0002] Description of the Prior Art In recent years, there has been an increase in incidents of vandalism against parked cars overnight and vandalism against railway carriages, making it urgent to take measures such as quickly detecting the actions of trespassers and sounding alarms. In particular, for public infrastructure such as buses and trains, in addition to safety and security, smooth operation is required, so surveillance systems that operate 24 hours a day, day or night, are being built to monitor things like the intrusion of suspicious people into restricted areas and the situation at railroad crossings.

[0003] Conventional surveillance systems often use visible light sensors that capture images of visible light in the wavelength range of 380 to 780 nm. However, it has been pointed out that visible light sensors are vulnerable to fog and haze, and that lighting equipment is required as a light source for surveillance at night. In addition, images are blurred due to backlighting and reflected light from the train body, making accurate detection difficult.

[0004] For these reasons, when detecting vehicles or pedestrians moving in shadowed areas or through tunnels, night vision surveillance systems using far-infrared sensors that detect temperature (heat) are increasingly being used. Far-infrared rays have the advantage of being able to penetrate fog, and the use of far-infrared sensors has the advantage of being able to detect objects without the need for a light source, and regardless of the environment, such as darkness, smoke, or fog.

[0005] [Conventional surveillance system: Figure 7] The configuration of a conventional monitoring system will be described with reference to Fig. 7. Fig. 7 is a schematic diagram of a conventional monitoring system. As shown in FIG. 7, the conventional surveillance system is a night vision surveillance system, and includes a far-infrared camera 4 and a surveillance monitor 3. Further, the far-infrared camera 4 includes an imaging section 41 , an image processing section 42 , and an output section 43 .

[0006] The imaging unit 41 of the far-infrared camera 4 is a far-infrared sensor that detects far-infrared rays and outputs a signal (far-infrared image data) according to temperature. The image processing unit 42 performs image processing on the far-infrared image data from the imaging unit 22, and outputs image data for display (live image data). The output unit 43 converts the live image data into a format suitable for the surveillance monitor 3 and outputs it.

[0007] The surveillance monitor 3 displays an image based on the live image data from the surveillance camera 2 . The image captured by the far-infrared camera 4 is a black and white light and dark image based on the temperature difference, with high temperature areas displayed as high brightness (whitish) and low temperature areas displayed as low brightness (blackish).

[0008] In the conventional night vision surveillance system, the far-infrared image data captured by the imaging unit 41 of the far-infrared camera 4 is subjected to various image processing in the image processing unit 42 to generate live image data, and the live image data is output to the surveillance monitor 3 via the output unit 43, and a far-infrared surveillance image is displayed on the surveillance monitor 3. Although not shown, the far-infrared camera 4 is provided with a control unit, and when the control unit detects an abnormality (such as an intruder) based on the live image data, it issues an alarm to the surveillance monitor 3 or the like.

[0009] [Virtual image caused by radiant heat] Surveillance images captured by far-infrared sensors have lower resolution than those captured by visible light sensors, and are black and white light-dark images based on temperature differences, making it difficult to clearly identify objects that have no temperature difference from their surroundings. In addition, in the case of an object that has a high temperature relative to the surrounding temperature, the heat emitted by the heat generating body may cause a virtual image that does not actually exist to be reflected on the screen. For example, if a subject moves, heat remains in the place where the subject was before the movement, and this becomes a virtual image. The reflection of virtual images can lead to misrecognition or false detection.

[0010] [Related Technology] Incidentally, a conventional technique for a monitoring system using an infrared camera is disclosed in Japanese Patent Application Laid-Open No. 2017-97702, entitled "Monitoring system and its monitoring control device" (Patent Document 1). Patent Document 1 describes a monitoring system that acquires both visible image data and infrared image data and detects whether or not a temperature abnormality (fire) has occurred in a monitored object. [Prior art documents] [Patent documents]

[0011] [Patent Document 1] JP 2017-97702 A Summary of the Invention [Problem to be solved by the invention]

[0012] As described above, conventional surveillance cameras using far-infrared sensors had the problem that the radiation of heat from the heating element could cause virtual images that do not actually exist to be reflected on the screen, which could lead to false recognition or false detection.

[0013] Furthermore, Patent Document 1 does not mention that a feature amount of a thermal image area in live image data is calculated, and an identification and judgment model is used to estimate a virtual image versus a real image based on the feature amount, and if the calculated feature amount of the thermal image area correlates with the feature amount of a virtual image, the identified thermal image area is judged to be a virtual image.

[0014] The present invention has been made in consideration of the above-mentioned circumstances, and aims to provide a surveillance camera and imaging device that use a far-infrared sensor to suppress the effects of virtual images caused by radiant heat and prevent erroneous recognition and detection. [Means for solving the problem]

[0015] In order to solve the problems of the above-mentioned conventional example, the present invention provides a surveillance camera having an imaging unit that outputs captured far-infrared image data, and an image processing unit that performs image processing on the far-infrared image data and outputs live image data, By comparing the brightness and darkness intensity of the live image data with a preset threshold, Thermal image area from live image data detection death, Detected Thermal Image Area Based on a plurality of feature points set in advance, the center of gravity coordinates, the motion vector information, and the temperature information The present invention is characterized by having a calibration unit that calculates a feature amount of the thermal image area, and uses an identification and judgment model to estimate a virtual image versus a real image based on the feature amount, and judges the thermal image area to be a virtual image if the calculated feature amount correlates with the feature amount of the virtual image, and judges the thermal image area to be a real image if there is no correlation.

[0016] The present invention also provides a method for producing a semiconductor device comprising the steps of: The surveillance camera has an imaging unit that outputs captured far-infrared image data, and an image processing unit that performs image processing on the far-infrared image data and outputs live image data, and is provided with a calibration unit that identifies a thermal image area from the live image data, calculates a feature amount of the thermal image area, and uses an identification and judgment model for estimating a virtual image relative to a real image based on the feature amount to judge the thermal image area as a virtual image if the calculated feature amount correlates with the feature amount of the virtual image, and judges the thermal image area as a real image if there is no correlation, and the calibration unit generates correction data in which the feature is suppressed by lowering the gain of the thermal image area judged to be a virtual image, and is provided with an image synthesis unit that synthesizes the live image data from the image processing unit and the correction data. It is characterized by the following.

[0017] Further, in the surveillance camera according to the present invention, the calibration unit The gain of the thermal image area determined to be a real image is increased to generate correction data that emphasizes the characteristics, and the image synthesis unit synthesizes the live image data from the image processing unit and the correction data. It is characterized by the following.

[0018] The present invention also provides a method for producing a semiconductor device comprising the steps of: A surveillance camera having an imaging unit that outputs captured far-infrared image data, and an image processing unit that performs image processing on the far-infrared image data and outputs live image data, the surveillance camera further comprising a calibration unit that identifies a thermal image area from the live image data, calculates a feature amount of the thermal image area, and uses an identification and judgment model for estimating a virtual image relative to a real image based on the feature amount to judge the thermal image area as a virtual image if the calculated feature amount correlates with the feature amount of the virtual image, and judges the thermal image area as a real image if there is no correlation, The calibration section is When there are two adjacent thermal image areas in the live image data, the feature amount of the thermal image area with a higher temperature is set as a first feature amount, and the feature amount of the thermal image area with a lower temperature is set as a second feature amount, and the first feature amount is input to a discrimination judgment model as a real image. When the discrimination judgment model outputs a feature amount of a virtual image corresponding to the feature amount of the real image similar to the first feature amount, a correlation value between the second feature amount and the feature amount of the virtual image is calculated, and when the correlation value is equal to or greater than a first threshold value, the thermal image area with a lower temperature is judged to be a virtual image. It is characterized by the following.

[0019] Further, in the surveillance camera of the present invention, the calibration unit determines that the image is a virtual image. Even if the temperature in the thermal image area is equal to or higher than the second threshold, the thermal image is determined to be a real image. It is characterized by the following.

[0020] Furthermore, the present invention is characterized in that, in the above-mentioned surveillance camera, before operation, the calibration unit stores the first feature amount as a real image feature amount and the second feature amount as a virtual image feature amount in association with each other, and performs machine learning using the stored pair of the real image feature amount and the virtual image feature amount to generate an identification and determination model.

[0021] The present invention also provides any one of the above surveillance cameras. And security cameras The output from Live A video device that displays image data and performs image processing. A surveillance system equipped with And, The imaging device compares the brightness and darkness intensity of the image data with a preset threshold value to detect a thermal image area from the live image data, calculates feature amounts of center of gravity coordinates, motion vector information, and temperature information for the detected thermal image area based on a plurality of preset feature points, performs machine learning by associating the feature amounts with a determination result of a real image or a virtual image from the outside, and estimates a virtual image relative to a real image based on the feature amounts. identification judgement Model Generate a classification model The feature of this method is that it is set in the calibration section of the surveillance camera. Effect of the Invention

[0022] According to the present invention, there is provided a surveillance camera having an imaging unit that outputs captured far-infrared image data, and an image processing unit that performs image processing on the far-infrared image data and outputs live image data, By comparing the brightness and darkness intensity of the live image data with a preset threshold, Thermal image area from live image data detection death, Detected Thermal Image Area Based on a plurality of feature points set in advance, the center of gravity coordinates, the motion vector information, and the temperature information The surveillance camera is equipped with a calibration unit that calculates the feature values ​​of the thermal image area and uses an identification and judgment model to estimate whether the thermal image is a virtual image or a real image based on the feature values. If the calculated feature values ​​correlate with the feature values ​​of the virtual image, the thermal image area is judged to be a virtual image, and if they do not correlate, the thermal image area is judged to be a real image. This makes it possible to determine whether a thermal image area in live image data is a real image or a virtual image, and by applying image processing that makes the thermal image area of ​​a virtual image less noticeable, it is possible to prevent misrecognition and erroneous detection due to virtual images.

[0023] Further, according to the present invention, A surveillance camera having an imaging unit that outputs captured far-infrared image data, and an image processing unit that performs image processing on the far-infrared image data and outputs live image data, the surveillance camera further comprising a calibration unit that identifies a thermal image area from the live image data, calculates a feature amount of the thermal image area, and uses an identification and judgment model for estimating a virtual image relative to a real image based on the feature amount to judge the thermal image area as a virtual image if the calculated feature amount correlates with the feature amount of the virtual image, and judges the thermal image area as a real image if there is no correlation, The calibration unit generates correction data in which the gain of a thermal image area determined to be a virtual image is reduced to suppress the characteristics, and includes an image synthesis unit that synthesizes the live image data from the image processing unit and the correction data. Supervisor Since the image is captured by a virtual camera, the virtual image is less noticeable in the synthesized image, which has the effect of preventing misrecognition and false detection due to virtual images.

[0024] Furthermore, according to the present invention, the calibration unit generates correction data that emphasizes the features by increasing the gain of the thermal image area determined to be a real image, and the image synthesis unit is the above-mentioned surveillance camera that synthesizes the live image data from the image processing unit with the correction data, so that the real image is clearly displayed in the synthesized image, which has the effect of alerting the observer to the attention of the observer. [Brief description of the drawings]

[0025] [Figure 1] FIG. 1 is an explanatory diagram of the present monitoring system. [Diagram 2] FIG. 2 is an explanatory diagram showing an example of a shooting scene. [Diagram 3] 2 is an explanatory diagram showing feature amounts acquired by the moving object detection unit 21. FIG. [Figure 4] FIG. 4 is an explanatory diagram illustrating a feature amount table. [Diagram 5] 13 is a flowchart showing the process of the moving object detection unit 21 during operation. [Figure 6] 10 is a flowchart showing a calibration process of the control unit 22 during operation. [Figure 7] FIG. 1 is a schematic configuration diagram of a conventional monitoring system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described with reference to the drawings. [Outline of the embodiment] A surveillance camera according to an embodiment of the present invention (this surveillance camera) is a camera equipped with a far-infrared sensor, and is provided with a calibration unit that performs a calibration process to suppress the influence of virtual images. The calibration unit identifies a thermal image area from live image data from the image processing unit, calculates features of the thermal image area, and uses an identification and judgment model to estimate a virtual image relative to a real image based on the features, and if the features of the calculated thermal image area correlate with the features of the virtual image, it judges the identified thermal image area to be a virtual image, and if they do not correlate, it judges the thermal image area to be a real image. This makes it possible to determine whether a thermal image area in the live image data is a real image or a virtual image, and by performing image processing to make the virtual image less noticeable, it is possible to prevent erroneous recognition and erroneous detection due to virtual images.

[0027] In addition, in this surveillance camera, the calibration unit generates calibration data that suppresses the characteristics by lowering the gain of the thermal image area determined to be a virtual image, and the image synthesis unit synthesizes the live image data from the image processing unit with the correction data, so that it is possible to display an image that is masked so that the virtual image portion is not noticeable, reducing the reflection of the virtual image and preventing misrecognition and erroneous detection.

[0028] [Surveillance system equipped with this surveillance camera: Figure 1] A monitoring system equipped with this monitoring camera (this monitoring system) will be described with reference to Fig. 1. Fig. 1 is an explanatory diagram of this monitoring system. As shown in Fig. 1, this monitoring system, like the conventional monitoring system shown in Fig. 7, includes a far-infrared camera (this monitoring camera) 1 and a monitoring monitor 3. The monitoring monitor 3 is the same as in the conventional system.

[0029] [Configuration of a far-infrared camera: Figure 1] The configuration of the far-infrared camera 1 will be specifically described with reference to FIG. The far-infrared camera 1 includes an imaging unit 11, an image processing unit 12, a delay adjustment unit 13, an image synthesis unit 14, an output unit 15, and a calibration unit 16.

[0030] The imaging sensor 11 is a far-infrared sensor similar to a conventional sensor, which captures images using far-infrared rays and outputs far-infrared image data. The image processor 12 is similar to the conventional one, and performs various image processing on the input far-infrared image data to improve visibility and output live image data. Image processing includes noise removal, defect correction, gamma correction, etc.

[0031] The delay adjustment unit 13 receives one of the two pieces of live image data output from the image processing unit 12 and branched into two, and outputs the received image data with a delay corresponding to the time required for processing in the calibration unit 16 (described later). The image synthesis unit 14 synthesizes the calibration data (calibration data, correction data) from the calibration unit 16 and the delayed live image data from the delay adjustment unit 13, and outputs synthesized far-infrared image data. As will be described later, by synthesizing calibration data, it is possible to suppress the effects of virtual images and to make real images more noticeable and thus draw attention to the image. The output unit 15 adds a header and the like to the input synthetic far-infrared image data, converts it into a predetermined format, and outputs it to the surveillance monitor 3.

[0032] The calibration unit 16 is a characteristic feature of this surveillance camera, and generates calibration data to reduce the influence of a virtual image caused by radiant heat from the subject in the other branched live image data. Also, when a real image of a subject is captured rather than a virtual image, calibration data is generated so that the image is displayed more clearly. As shown in FIG. 1, the calibration unit 16 includes a moving object detection unit 21, a control unit 22, and a storage unit .

[0033] When the moving object detection unit 21 detects an area (thermal image area) having a higher temperature than the surroundings from the other live image data, the moving object detection unit 21 prompts the control unit 22 to issue an alarm. At the same time, the moving object detection unit 21 calculates multiple items that represent the characteristics of the thermal image area from the detected thermal image area, sets them as feature quantities, associates them one-to-one with the thermal image area, and outputs them to the control unit 22 together with an instruction to issue an alarm. When not one but multiple thermal image areas are detected, they are numbered in the order of detection, for example, thermal image area 1, thermal image area 2, and the feature amount of each is calculated.

[0034] When an instruction to issue an alarm is input from the moving object detection unit 21, the control unit 22 issues an alarm and notifies a predetermined notification destination of the abnormality. Another feature of this surveillance camera is that the control unit 22 is equipped with an identification and judgment model for determining whether a thermal image area detected in live image data is a real image or a virtual image, and inputs the feature amount input from the motion detection unit 21 into the identification and judgment model, and determines whether the corresponding thermal image area is a real image or a virtual image based on the output of the model.

[0035] The identification and judgment model is a trained model that has undergone machine learning based on the installation location and the characteristics of the subject. When this monitoring system is installed, as a preparation stage before operation, it is generated by inputting the features of the thermal image area and real / virtual image information for various anticipated scenes and learning from them. Specifically, using a video device that displays live image data, an observer looks at the live image data and determines whether the thermal image area displayed there is a real image or a virtual image, and sets the determination result in the control unit 22 in correspondence with the thermal image area. The generation of the discrimination model in the preparation stage before operation will be described later.

[0036] Furthermore, the control unit 22 performs a calibration process for calibrating the live image data so that the image displayed on the monitoring monitor 3 is improved depending on whether the thermal image area is a real image or a virtual image. Specifically, when the control unit 22 determines that a thermal image area is a virtual image, it generates and outputs calibration data that converts the light and dark intensity into a temperature range similar to the ambient temperature so that the virtual image is not noticeable. Conversely, when the thermal image area is determined to be a real image, the control unit 22 generates and outputs calibration data that is converted to be more conspicuous by increasing the light and dark intensity in order to encourage the observer to pay attention to it.

[0037] The storage unit 23 stores the identification judgment model and other programs and various data executed by the control unit 22, and in particular stores live image data used for machine learning and operation of the identification judgment model, and information such as feature amounts and real / virtual images, which will be described later. The information such as feature amounts and real / virtual images is stored in a feature amount table of the storage unit 23.

[0038] [Generating a classification model] When this surveillance camera is installed, a large amount of information is fed to the pre-trained model based on the environment of the installation location and anticipated abnormalities (suspicious people, etc.), and machine learning is performed to generate a trained identification and judgment model. First, when the system is set up, a subject playing the role of a suspicious person is made to perform the suspicious behavior patterns that have been previously compiled into a database, such as "wandering, stopping, and passing through," and this is photographed by the far-infrared camera 1. At that time, many far-infrared images are taken by changing various conditions such as the time of day, weather conditions, and the subject's behavior patterns.

[0039] Then, from the captured far-infrared image (live image data), thermal image areas are detected for each scene, and an observer present at the capture visually determines whether each thermal image area is a real image or a virtual image. The features of each thermal image area are associated with the judgment results and stored in a feature table in the memory unit 23. The control unit 22 then generates an identification and judgment model by subjecting the model to machine learning.

[0040] Although not shown in the figure, a monitor operates a video device connected to the output unit 15 of the far-infrared camera 1 to visually check the far-infrared image and input the judgment result to the surveillance camera. The video device is basically a computer that includes a control unit, a storage unit, an interface unit, an operation unit, and a display unit and is connected to a network.

[0041] Furthermore, machine learning of the identification and judgment model may be performed not inside the surveillance camera but in the imaging device, and the learned identification and judgment model may be set in the calibration unit 16 of the surveillance camera. By generating an identification and judgment model in the imaging device, the processing load of this surveillance camera can be significantly reduced.

[0042] [Example of a shooting scene: Figure 2] Next, an example of a shooting scene in generating an identification / determination model and reflection of a virtual image in a captured far-infrared image will be described with reference to Fig. 2. Fig. 2 is an explanatory diagram showing an example of a shooting scene. As shown in FIG. 2, a far-infrared camera 1 is installed at an operating location, and captures an image of a subject 51 acting as an intruder entering the facility. When the subject 51 moves, a virtual image 52 remains in the space where the subject 51 was before moving due to radiant heat from the subject 51. Therefore, in the far-infrared image displayed on the monitoring monitor 3, both the thermal image area 1, which is a real image, and the thermal image area 2, which is a virtual image, are displayed.

[0043] In this surveillance camera, the observer visually judges whether the thermal image area 1 and thermal image area 2 are real or virtual. Here, it is judged that the thermal image area 1 is a real image, and the thermal image area 2 is a virtual image corresponding to the thermal image area 1. When the observer inputs the result of the determination from the imaging device, the result of the determination is sent to the far-infrared camera 1, and the result of the determination is set in the control unit 22 together with the characteristic amount described later.

[0044] In addition, instead of a visual judgment by a surveillance person, a twin-lens camera equipped with a visible light sensor and a far-infrared sensor can be used to detect the features of both visible and far-infrared images taken at the same time and with the same angle of view, and by comparing both features, unnatural heating objects reflected in the far-infrared image can be judged to be virtual images, and the judgment result can be input to the control unit 22 of this surveillance camera.

[0045] Furthermore, when the generation of the identification judgment model is performed by a video device connected to the surveillance camera, in addition to the live image data, information on feature 1 and feature 2 is also output from the surveillance camera to the video device, and the video device stores the real / virtual image judgment result, and the identification judgment model is machine-learned based on the stored information.

[0046] [Feature acquisition: Figure 3] Next, the acquisition of feature amounts in the moving object detection unit 21 will be described with reference to Fig. 3. Fig. 3 is an explanatory diagram showing feature amounts acquired by the moving object detection unit 21. When live image data of each behavior scene in which a subject is photographed is input, the moving object detection unit 21 first detects a thermal image area. A threshold judgment method is used to detect the thermal image area, and an area in which the light and dark intensity is stronger than a preset threshold (appears brighter, has high luminance) is determined to be the thermal image area. A thermal image area with sufficient light and dark intensity is expected to be, for example, a real image of a person, so the thermal image area detected first is often a real image of a person. When multiple thermal image areas are detected within one screen, they are given identifiable names such as thermal image area 1, thermal image area 2, etc. in the order of detection.

[0047] Then, the moving object detection unit 21 detects a plurality of pre-defined feature points for the detected thermal image area. In the example of Fig. 3, feature points 1a, 1b, 1c, and 1d are detected for the thermal image area 1, and feature points 2a, 2b, 2c, and 2d are detected for the thermal image area 2.

[0048] Next, the moving object detection unit 21 calculates a feature amount from these feature points. The feature quantities calculated are three pieces of information: the "centre of gravity coordinates" calculated from the four feature points, "motion vector information" indicating the direction and amount of movement of the centre of gravity coordinates, and "temperature information" obtained by averaging the light and dark intensity of the area connecting the four feature points. The calculated feature amount is then stored in the storage unit 23 as the feature amount of each thermal image region. Here, the feature amount corresponding to thermal image region 1 is referred to as "feature amount 1", and the feature amount corresponding to thermal image region 2 is referred to as "feature amount 2". When only one thermal image area is detected, the moving object detection unit 21 calculates the feature amount of only the thermal image area 1 and stores it in the feature amount table of the storage unit 23.

[0049] When multiple thermal image areas are detected within the same screen, and one of the thermal image areas (here referred to as thermal image area 2) has a different light / dark intensity from the other (thermal image area 1), but is close to thermal image area 1 and moves in conjunction with the movement of thermal image area 1 (the motion vectors are approximately equal), it is estimated that thermal image area 2 is a virtual image of thermal image area 1.

[0050] In this surveillance camera, when a surveillance person visually determines that thermal image area 2 is a virtual image of thermal image area 1, the motion detection unit 21 associates feature amount 1 and feature amount 2 of the scene, labels feature amount 1 as a "real image" and feature amount 2 as a "virtual image," and stores them. The determination of whether an image is real or virtual may not be based on visual inspection, but may be based on the difference between adjacent frame images in a time series.

[0051] This causes information to be stored indicating that in the live image data of the scene, a real thermal image area 1 having feature amount 1 was accompanied by a virtual thermal image area 2 having feature amount 2. Incidentally, even if two thermal image regions are detected, if both are real images (such as when there are two intruders), there is no correlation between the arrangement and movement of the feature points.

[0052] Then, for many captured scenes, the features of thermal image area 1 and thermal image area 2 are calculated and labeled as real or virtual images, and the results are stored in a database. The correlation (trend) between feature 1 and feature 2 is then machine-learned.

[0053] In other words, the identification and judgment model learns the features of a virtual image that correspond to the features of thermal image area 1, which is a real image, and when in operation, when a feature (feature 1) of a real image of a heating element is input, it outputs feature 2 (model feature 2) of a virtual image that is likely to be caused by the heating element. In this surveillance camera, the control unit 22 of the calibration unit 16 uses the learned identification and judgment model to determine whether a thermal image area captured in the live image data is a real image or a virtual image.

[0054] [Generate a different classification model] Next, generation of another discrimination model will be described. Another identification and determination model includes a behavior pattern determination model and a virtual image determination model. As described above, the behavior pattern determination model learns feature 1 of multiple behavior patterns such as wandering, stopping, and passing through, and uses the output value (probability) to identify which behavior pattern the feature of the real image input during operation (feature 1) corresponds to.

[0055] The virtual image determination model is set up for each behavior pattern, learns the feature of the virtual image corresponding to the real image (feature 2), and outputs the probability that the input feature 2 is a virtual image of feature 1 used to identify the behavior pattern. During operation, if the probability value output from the virtual image determination model is equal to or greater than a specific threshold, the thermal image area having feature quantity 2 is determined to be a virtual image of the thermal image area having feature quantity 1.

[0056] When generating a separate identification and judgment model using a video device, the live image data captured by this surveillance camera and the information in the feature table described below (feature 1, feature 2, and real image / virtual image judgment result) are input to the AI ​​model through machine learning to generate a trained behavior pattern judgment model and a trained virtual image judgment model corresponding to each behavior pattern. The trained separate identification and judgment model includes all of these trained models. Then, the imaging device transfers (stores) the other already-learned identification and judgment model to the calibration unit 16 of the surveillance camera. This makes the other already-learned identification and judgment model available during operation.

[0057] [Feature table: Figure 4] The feature amount table provided in the storage unit 23 will be described with reference to Fig. 4. Fig. 4 is an explanatory diagram showing the feature amount table. As shown in FIG. 4, the feature table stores, for each captured behavior scene (behavior scene (1) to behavior scene (M)), a feature corresponding to thermal image area 1 (feature 1) and a feature corresponding to thermal image area 2 (feature 2) in one-to-one correspondence. Feature 1 and feature 2 are the center of gravity coordinates (horizontal / vertical) corresponding to each thermal image region, the movement vector (movement direction / movement amount), and temperature information.

[0058] Although not shown in the figure, information on whether the image is real or virtual as determined by the observer is also stored. Here, for simplicity, the thermal image area determined to be a real image is referred to as thermal image area 1, and the thermal image area determined to be a virtual image corresponding to thermal image area 1 is referred to as thermal image area 2, and their respective feature amounts are stored as feature amount 1 and feature amount 2. When the identification and judgment model is generated by the imaging device, the feature table is provided in the storage unit of the imaging device.

[0059] [Operational behavior: Figure 1] When an identification and judgment model is generated and operation is started, far-infrared image data captured by the imaging sensor 11 is subjected to image processing by the image processing unit 12, and live image data is output. The live image data is branched into two, one of which is delayed by a predetermined time in the delay adjustment unit 13 and output to the image synthesis unit 14.

[0060] The other live image data is input to the calibration unit 16, and when the moving object detection unit 21 of the calibration unit 16 identifies a thermal image area and detects a heat source, the control unit 22 issues an alarm.

[0061] In addition, in the moving object detection unit 21, the characteristic parameters of the thermal image area are calculated, and if the control unit 22 determines that it is a real image, calibration data is generated in which the light and dark intensity is strengthened and made clearer so that the thermal image area stands out, and this data is output to the image synthesis unit 14.

[0062] On the other hand, if the thermal image area is determined to be a virtual image, calibration data is generated in which the thermal image area is converted to a light-dark intensity equivalent to the surrounding temperature so that the thermal image area does not stand out, and is output to the image synthesis unit 14. In the image synthesis unit 14, the delayed live image data and the calibration data are synthesized, and in the output unit 15, the format is converted and the data is output to the monitoring monitor 3. As a result, the surveillance monitor 3 displays a far-infrared image with less noticeable virtual images and a clearer real image than before, thereby preventing misrecognition and erroneous detection by surveillance personnel.

[0063] [Processing by the motion detection unit 21: Figure 5] Next, the processing of the moving object detection unit 21 of the calibration unit 16 during operation will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the processing of the moving object detection unit 21 during operation. Note that, here, the processing when the moving object detection unit 21 detects two thermal image areas is shown. As shown in FIG. 5, when the moving object detection unit 21 detects a thermal image area a and a thermal image area b from the live image data (S11), it instructs the control unit 22 to issue an alarm as a heat source has been detected (S12).

[0064] Then, the moving object detection unit 21 calculates feature amounts for the thermal image area a and the thermal image area b (feature amount a, feature amount b). Furthermore, the moving object detection unit 21 compares the temperatures of the calculated feature quantities (S14), and designates the feature quantity corresponding to the higher temperature as thermal image area 1 (usually the actual image of the heating element) as feature quantity 1, and the feature quantity corresponding to the lower temperature as thermal image area 2 as feature quantity 2 (S15). Then, the moving object detection unit 21 outputs the feature amount 1 and the feature amount 2 to the control unit 22 (S16). In this manner, the processing of the moving object detection unit 21 during operation is carried out.

[0065] [Calibration process of the control unit 22: FIG. 6] Next, the calibration process of the control unit 22 of the calibration unit 16 during operation will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the calibration process of the control unit 22 during operation. Note that this shows the calibration process when two thermal image areas are detected in the live image data, and is performed following the process in Fig. 5. Also, when an instruction to issue an alarm is received from the moving object detection unit 21, the control unit 22 issues an alarm, but this is not shown in the calibration process in Fig. 6.

[0066] As shown in FIG. 6, when feature amount 1 and feature amount 2 are input from the moving object detection unit 21 (S21), the control unit 22 inputs feature amount 1 to the trained discrimination model (S22). Here, feature amount 1 is a feature amount of thermal image area 1, and is usually a feature amount of the actual image of a heat generating element.

[0067] In the identification and judgment model, the behavior scenes stored in the feature table are searched for, and a scene having a feature 1 (model feature 1) similar to the feature 1 of the input real image is identified. For example, in the identification and judgment model, the correlation between the feature amount 1 corresponding to each behavior scene and the feature amount 1 of the input real image is calculated, and the behavior scene with the highest correlation is identified. Then, the identification and judgment model outputs the feature amount 2 associated with the scene (or the feature amount 2 calculated based on the feature amount 2) as a model feature amount 2. The model feature amount 2 is a value that the identification and judgment model outputs as a feature amount of a virtual image that may appear in the scene.

[0068] Then, the control unit 22 compares the model feature 2 output from the identification and judgment model with the feature 2 from the moving object detection unit 21 to calculate a correlation value (S23), and determines whether the correlation value is greater than or equal to a first threshold (S24).If the correlation value is less than the first threshold (No), it determines that the thermal image area 2 in the live image data is unlikely to be a virtual image of the thermal image area 1, and that the thermal image area 2 is a real image (S28).

[0069] Furthermore, if the correlation value is equal to or greater than the first threshold in process S24 (YES), the control unit 22 further determines whether the temperature is lower than a second threshold based on the temperature information of feature 2 (S25), and if it is equal to or greater than the second threshold, proceeds to process S29 and determines that it is a real image. Typically, in dark environments where surveillance cameras and imaging devices using far-infrared rays are expected to be used, the surface temperature of a person's skin is higher than the temperature of the surrounding environment, so taking advantage of this property, the second threshold is set to a temperature slightly higher than the surrounding temperature.

[0070] Then, if the temperature information of feature 2 is lower than the second threshold in process S25 (Yes), the control unit 22 determines that the thermal image area 2 is a virtual image of the thermal image area 1 having the feature 1 input in process 11 (S26), reduces the gain of the thermal image area 2, and generates calibration data in which the light and dark intensity is converted to light and dark intensity in the same temperature range as the ambient temperature (S27), and proceeds to process S30. In other words, calibration data is generated to mask the thermal image area 2 determined to be a virtual image so as to make it less noticeable.

[0071] Furthermore, if the control unit 22 determines that the image is a real image in process S28, it generates calibration data to clarify the contrast with the surroundings and increase the light and dark intensity of the thermal image area 2 in order to attract the observer's attention (S29), and then proceeds to process S30. At that time, if there is a change such that the area of ​​the thermal image area 2 gradually increases, this means that an intruder is approaching the sensor, so in addition to generating calibration data that increases the light and dark intensity, an alarm is sent to notify the user of an emergency.

[0072] Then, when the configuration data is generated in steps S27 and S29, the control unit 22 proceeds to step S30 and outputs the calibration data to the image synthesis unit 14 (S30). In this manner, the calibration process in the control unit 22 of the calibration unit 16 is carried out.

[0073] [Judgment using a different discrimination model] In a judgment using another identification judgment model, when feature quantities 1 and 2 are input from the moving object detection unit 21, the control unit 22 first inputs feature quantity 1 to the behavior pattern judgment model to identify the behavior pattern. Then, the feature amount 2 input from the moving object detection unit 21 is input to the virtual image determination model corresponding to the identified behavior pattern, and a probability value is obtained from the virtual image determination model. The value of this probability is the probability that feature value 2 is a virtual image of a real image having feature value 1 used when identifying the behavior pattern.

[0074] Then, the control unit 22 determines that the thermal image area 2 is a virtual image of the thermal image area 1 if the probability value from the virtual image determination model is greater than or equal to a specific threshold, and determines that the thermal image area 2 is a real image if the probability value is less than the specific threshold. Generation and output of configured data after the determination is similar to that in FIG. In this manner, a judgment is made using a different discrimination judgment model.

[0075] [Effects of the embodiment] According to this surveillance camera, the camera is equipped with a far-infrared sensor and is provided with a calibration unit 16 that performs a calibration process to suppress the influence of virtual images. The calibration unit 16 identifies a thermal image area from live image data from the image processing unit 12, calculates features of the thermal image area, and uses an identification and judgment model to estimate a virtual image relative to a real image based on the features. If the features of the calculated thermal image area correlate with the features of the virtual image, the identified thermal image area is judged to be a virtual image, and if they do not correlate, the thermal image area is judged to be a real image. This has the effect of preventing erroneous recognition and erroneous detection due to virtual images by being able to determine whether a thermal image area in the live image data is a real image or a virtual image and by performing image processing to make the virtual image less noticeable.

[0076] Furthermore, according to this surveillance camera, the calibration unit 16 generates correction data that suppresses the characteristics by lowering the gain of the thermal image area determined to be a virtual image, and the image synthesis unit 14 synthesizes the live image data from the image processing unit 12 with the calibration data. This makes it possible to display an image that is masked so that the virtual image portion is not noticeable, thereby reducing the reflection of the virtual image on the surveillance monitor 3 and preventing erroneous recognition and detection.

[0077] Furthermore, according to the present imaging device, an unlearned model is trained to learn the features of the thermal image area through machine learning to generate an identification and judgment model, and the identification and judgment model is set in the calibration unit 16 of the present surveillance camera, which has the effect of significantly reducing the processing power of the surveillance camera. [Industrial Applicability]

[0078] The present invention is suitable for surveillance cameras and imaging devices that use far-infrared sensors to suppress the effects of virtual images caused by radiant heat and prevent erroneous recognition and detection. [Explanation of symbols]

[0079] 1,4...far-infrared camera, 3...monitor monitor, 11,41...imaging section, 12,42...image processing section, 13...delay adjustment section, 14...image synthesis section, 15,43...output section, 16...calibration section, 21...moving object detection section, 22...control section, 23...storage section, 51...subject, 52...virtual image

Claims

1. A surveillance camera having an imaging unit that outputs captured far-infrared image data, and an image processing unit that performs image processing on the far-infrared image data and outputs live image data, A surveillance camera characterized by comprising a calibration unit that compares the light and dark intensity of the live image data with a preset threshold to detect a thermal image area from the live image data, calculates features of the center of gravity coordinates, motion vector information, and temperature information for the detected thermal image area based on a plurality of preset feature points, and uses an identification and judgment model for estimating a virtual image versus a real image based on the features to determine that the thermal image area is a virtual image if the calculated features correlate with the features of a virtual image, and determines that the thermal image area is a real image if there is no correlation.

2. A surveillance camera having an imaging unit that outputs captured far-infrared image data, and an image processing unit that performs image processing on the far-infrared image data and outputs live image data, a calibration unit that identifies a thermal image area from the live image data, calculates a feature amount of the thermal image area, and uses an identification and judgment model for estimating a virtual image relative to a real image based on the feature amount to determine that the thermal image area is a virtual image if the calculated feature amount correlates with the feature amount of the virtual image, and determines that the thermal image area is a real image if the calculated feature amount does not correlate with the feature amount of the virtual image, the calibration unit generates correction data in which the gain of a thermal image region determined to be a virtual image is reduced to suppress the characteristics of the thermal image region; A surveillance camera comprising an image synthesis unit that synthesizes live image data from the image processing unit and the correction data.

3. the calibration unit generates correction data in which a gain of a thermal image area determined to be a real image is increased to emphasize the characteristics of the thermal image area, 3. The surveillance camera according to claim 2, wherein the image synthesis unit synthesizes the live image data from the image processing unit and the correction data.

4. A surveillance camera having an imaging unit that outputs captured far-infrared image data, and an image processing unit that performs image processing on the far-infrared image data and outputs live image data, a calibration unit that identifies a thermal image area from the live image data, calculates a feature amount of the thermal image area, and uses an identification and judgment model for estimating a virtual image relative to a real image based on the feature amount to determine that the thermal image area is a virtual image if the calculated feature amount correlates with the feature amount of the virtual image, and determines that the thermal image area is a real image if the calculated feature amount does not correlate with the feature amount of the virtual image, The calibration unit, when there are two adjacent thermal image areas in the live image data, sets the feature amount of the thermal image area with a higher temperature as a first feature amount, and sets the feature amount of the thermal image area with a lower temperature as a second feature amount, and inputs the first feature amount into a discrimination judgment model as a real image, and when a feature amount of a virtual image corresponding to the feature amount of a real image similar to the first feature amount is output from the discrimination judgment model, calculates a correlation value between the second feature amount and the feature amount of the virtual image, and if the correlation value is greater than or equal to a first threshold value, judges the thermal image area with the lower temperature to be a virtual image.

5. 5. The surveillance camera according to claim 4, wherein the calibration section determines the image to be a real image when the temperature in the thermal image area is equal to or higher than a second threshold value even if the image has been determined to be a virtual image.

6. The surveillance camera according to claim 4 or 5, characterized in that, before operation, the calibration unit stores the first feature amount as a feature amount of a real image and the second feature amount as a feature amount of a virtual image in correspondence with each other, and performs machine learning using the stored pairs of the feature amounts of the real image and the feature amounts of the virtual image to generate the identification and judgment model.

7. 6. A surveillance system comprising the surveillance camera according to claim 2, and a video device that displays live image data output from the surveillance camera and processes the images, The surveillance system is characterized in that the imaging device compares the light and dark intensity of the image data with a preset threshold to detect a thermal image area from the live image data, calculates features of the center of gravity coordinates, motion vector information, and temperature information for the detected thermal image area based on a plurality of preset feature points, performs machine learning by correlating the features with the judgment results of a real image or a virtual image from the outside, generates an identification judgment model for estimating a virtual image relative to a real image based on the features, and sets the identification judgment model in a calibration unit of the surveillance camera.

Images