Detection device, detection system, detection method, and model generation device

Abstract

A detection device according to the present invention determines whether or not a determination target image is a detection target image including a detection target, and includes: an image restorative generation unit that inputs data of a determination target image into a trained model, which receives data of a processing image as input data to output data of the detection target image including the detection target matching the processing image as output data, to generate a detection target image corresponding to the determination target image; a difference calculation unit that generates an image of difference between the determination target image and the detection target image generated by the image restorative generation unit; and a detection unit that determines whether or not the determination target image is the detection target image on the basis of the difference image generated by the difference calculation unit.

Description

Detection device, detection system, detection method, and model generation device

[0001] The present invention relates to a detection device, a detection system, a detection method, and a model generation device that determine whether or not a determination target image is a detection target image that includes a detection target object.

[0002] Patent document 1 describes a device that extracts change areas from images of a monitored area captured sequentially by an imaging device, and compares the feature values ​​of the monitored area based on the extracted change areas with predetermined set values ​​of the feature values ​​to determine whether an event has occurred.

[0003] Patent No. 5388827

[0004] The device described in Patent Document 1 may not be able to detect a detection target that exists within a monitored area if there is an obstacle such as a safety fence, a fence, a dust-proof or drip-proof cover, etc. between the monitored area and the imaging device. For this reason, there has been a demand for a technology that can accurately detect a detection target that exists within a monitored area even if there is an obstacle or an object with low transmittance between the monitored area and the imaging device.

[0005] The present invention has been made in consideration of the above-mentioned problems, and its object is to provide a detection device, a detection system, and a detection method that can accurately detect a detection object present in a monitored area even if an obstacle or an object with low transmittance exists between the monitored area and the imaging device. Another object of the present invention is to provide a model generation device that can generate a learning model that accurately detects a detection object present in a monitored area even if an obstacle or an object with low transmittance exists between the monitored area and the imaging device.

[0006] The detection device of the present invention is a detection device that determines whether an image to be determined is a detection target image that includes a detection target object, and is equipped with an image restoration generation unit that generates a detection target image corresponding to the image to be determined by inputting data of the image to be determined into a learning model that has data of a processed image as input data and data of a detection target image that includes the detection target object corresponding to the processed image as output data, a difference calculation unit that generates a difference image between the image to be determined and the detection target image generated by the image restoration generation unit, and a detection unit that determines whether the image to be determined is the detection target image based on the difference image generated by the difference calculation unit.

[0007] The image processing unit may be provided with an image processing unit that performs image processing on the difference image generated by the difference calculation unit to emphasize the features of the object to be detected, and the detection unit may determine whether the image to be determined is the image to be detected based on the difference image on which image processing has been performed by the image processing unit.

[0008] The detection section may determine whether the determination target image is the detection target image based on the number of pixels in the difference image whose luminance value is equal to or greater than a predetermined threshold value.

[0009] It is preferable to provide a model generation unit that generates the learning model by learning to reconstruct a detection target image that is identical to the learning source data or a detection target image that approximates the learning source data, using the data of the detection target image as learning source data.

[0010] The model generator may be an autoencoder.

[0011] The model generation unit may add data of a determination target image that the detection unit has determined to be a detection target image to the learning source data, and generate the learning model again.

[0012] The detection system of the present invention comprises the detection device of the present invention and an imaging unit that captures an image of the monitored area and outputs data of the captured image of the monitored area to the detection device as data of the image to be determined.

[0013] The detection method of the present invention is a detection method for determining whether an image to be determined is a detection target image containing a detection target object, and includes an image restoration generation step for generating a detection target image corresponding to the image to be determined by inputting data of the image to be determined into a learning model in which data of the processed image is input data and data of the detection target image containing the detection target object corresponding to the processed image is output data; a difference calculation step for generating a difference image between the image to be determined and the detection target image generated in the image restoration generation step; and a detection step for determining whether the image to be determined is the detection target image based on the difference image generated in the difference calculation step.

[0014] The model generation device of the present invention is a model generation device that generates a learning model for determining whether an image to be determined is a detection target image that includes a detection target object, and is equipped with a model generation unit that generates a learning model using data of a processed image as input data and data of a detection target image that includes the detection target object corresponding to the processed image as output data, and the model generation unit generates the learning model by learning to reconstruct a detection target image that is identical to the learning target data or a detection target image that approximates the learning target data, using the data of the detection target image as learning source data.

[0015] The detection device, detection system, and detection method according to the present invention can accurately detect a detection target that exists in a monitored area even if an obstacle or an object with low transmittance exists between the monitored area and the imaging device.Furthermore, the model generation device according to the present invention can generate a learning model that accurately detects a detection target that exists in a monitored area even if an obstacle or an object with low transmittance exists between the monitored area and the imaging device.

[0016] FIG. 1 is a block diagram showing the configuration of a detection system according to one embodiment of the present invention. FIG. 2 is a flowchart showing the flow of detection processing according to one embodiment of the present invention. FIG. 3 is a diagram for explaining differential image generation processing. FIG. 4 is a flowchart showing the flow of learning model generation processing according to one embodiment of the present invention. FIG. 5 is a diagram showing an example configuration of an autoencoder. FIG. 6 is a diagram for explaining detection processing according to an embodiment.

[0017] Hereinafter, the configuration and operation of a detection system according to one embodiment of the present invention will be described with reference to the drawings.

[0018] [Configuration] Fig. 1 is a block diagram showing the configuration of a detection system according to one embodiment of the present invention. As shown in Fig. 1, the detection system 1 according to one embodiment of the present invention is a system that determines whether a determination target image is a detection target image that includes a detection target object, and includes an imaging unit 2, a data acquisition unit 3, a model generation unit 4, a learning database 5, and a calculation unit 6. The data acquisition unit 3, the model generation unit 4, and the calculation unit 6 function as a detection device according to the present invention.

[0019] The imaging unit 2 is composed of an imaging device such as a camera or an industrial surveillance camera (ITV). The imaging unit 2 captures images of the monitored area and outputs the captured image data of the monitored area to the data acquisition unit 3. Specifically, the imaging unit 2 captures images of the monitored area before and after an event changes or before and after a movable object moves, and outputs the captured image data of the monitored area to the data acquisition unit 3.

[0020] The data acquisition unit 3 and the model generation unit 4 are functional blocks that are realized by a processing unit such as a CPU in the information processing device executing a computer program. The functions of the data acquisition unit 3 and the model generation unit 4 will be described later.

[0021] The learning database 5 is configured with a non-volatile storage device. The learning database 5 stores learning source data 5a and a learning model 5b. As will be described in detail later, the learning source data 5a is image data of the monitored area at the time when a detection target exists in the monitored area. The learning model 5b is a machine learning model that takes image data of the monitored area to be judged (data of the image to be judged) as input data and image data of the monitored area at the time when a detection target exists in the monitored area corresponding to the input data as output data. The model generation unit 4 and the learning database 5 may be configured as a system or device other than the detection system 1.

[0022] The calculation unit 6 is configured by an arithmetic processing device such as a CPU inside the information processing device. The calculation unit 6 functions as an image restoration generation unit 6a, a difference calculation unit 6b, an image processing unit 6c, a detection unit 6d, and an output unit 6e when the arithmetic processing device executes a computer program. The functions of each of these units will be described later. The information processing device that realizes each of these units and the information processing device that realizes the data acquisition unit 3 and the model generation unit 4 may be the same device or different devices. If they are different devices, the information processing devices are connected to each other via a telecommunications line.

[0023] The detection system 1 having such a configuration executes the detection process and learning process described below, thereby enabling accurate detection of a detection target object present in a monitored area, even if an obstacle or an object with low transmittance exists between the monitored area and the imaging unit 2. Below, the operation of the detection system 1 when executing the detection process and learning model generation process will be described with reference to the flowchart shown in FIG.

[0024] [Detection Process] First, with reference to FIG. 2, the operation of the detection system 1 when performing the detection process will be described.

[0025] 2 is a flowchart showing the flow of the detection process according to one embodiment of the present invention. The flowchart shown in FIG. 2 starts when an execution command for the detection process is input to the calculation unit 6, and the detection process proceeds to step S1.

[0026] In the process of step S1, the data acquisition unit 3 inputs the image data of the area to be monitored output from the imaging unit 2 as data of the image to be determined to the calculation unit 6. This completes the process of step S1, and the detection process proceeds to the process of step S2.

[0027] In the processing of step S2, the image restoration generation unit 6a inputs the data of the image to be determined input by the data acquisition unit 3 as input data to the learning model 5b, thereby generating image data (output image) of the area to be monitored at the time when the detection object exists in the area to be monitored, which reflects the features of the image to be determined. This completes the processing of step S2, and the detection processing proceeds to the processing of step S3.

[0028] In the process of step S3, the difference calculation unit 6b generates a difference image between the determination target image input by the data acquisition unit 3 and the image of the monitoring target area generated by the image restoration generation unit 6a. An image of the difference image generation process is shown in Figure 3. Details of Figure 3 will be described later. This completes the process of step S3, and the detection process proceeds to the process of step S4.

[0029] In the process of step S4, the image processing unit 6c performs image processing such as binarization, masking, edge detection, smoothing, filtering, correction, expansion, and contraction on the generated difference image in order to emphasize features such as the shape of the detection object and make it easier to detect the detection object. In the binarization process, the image processing unit 6c converts the difference image into an image containing only two values: white and black. This completes the process of step S4, and the detection process proceeds to the process of step S5.

[0030] In the processing of step S5, the detection unit 6d determines whether or not a feature such as the shape of the object to be detected (target shape) is present in the difference image after image processing, thereby determining whether or not the detection object is present in the difference image. Specifically, if the detection object is shown in the image to be determined, the difference image will be completely black, with nothing shown, as shown in Figure 3. On the other hand, if the detection object is not shown in the image to be determined, the difference image will show the feature of the detection object, such as the target shape, near the detection object.

[0031] Therefore, when the binarization process is performed in the process of step S4, the detection unit 6d counts the number of white pixels in the difference image and determines whether the count value is equal to or less than a threshold value preset depending on the image size and the purpose of detection. If the count value is equal to or less than the threshold value, the detection unit 6d determines that the detection target object is included in the determination target image. The number of threshold values ​​is not limited to one, and multiple threshold values ​​may be used to gradually determine changes in events or scenes included in the difference image. This completes the process of step S5, and the detection process proceeds to the process of step S6.

[0032] In the process of step S6, the output unit 6e outputs whether or not the detection target object exists in the determination target image according to the determination result of the detection unit 6d to an output device. Examples of the output device include a printing device, a display device, an audio output device, etc. This completes the process of step S6, and the series of detection processes ends.

[0033] [Learning Model Generation Process] Next, with reference to FIG. 4, the operation of the detection system 1 when executing the learning model generation process will be described.

[0034] 4 is a flowchart showing the flow of a learning model generation process according to one embodiment of the present invention. The flowchart shown in FIG. 4 starts when image data of a monitored area in which it has been determined that a detection target object exists is acquired, and the learning model generation process proceeds to step S11. The image data of the monitored area in which it has been determined that a detection target object exists may be obtained in the detection process described above, or may be separately collected and determined.

[0035] In the process of step S11, the model generation unit 4 adds the image data of the monitoring area in which it is determined that a detection target object exists to the learning source data 5a stored in the learning database 5. This completes the process of step S11, and the learning model generation process proceeds to the process of step S12.

[0036] In the process of step S12, the model generation unit 4 generates a learning model 5b using the learning source data 5a. As the learning model 5b, a learning model capable of reconstructing images identical or similar to the learning source data is used. An example of such a learning model is an autoencoder. An example of the configuration of an autoencoder is shown in FIG. 5. As shown in FIG. 5, an autoencoder is a learning model that learns features from training images, extracts features by compressing input images to reduce their dimensions, and reconstructs and outputs images that match or approximate the training content.

[0037] Therefore, when data of a target image is input, the learning model 5b, which has been trained using the learning source data 5a, outputs as output data image data of the target area at the time when a detected object is present in the target area corresponding to the target image. As described above, the autoencoder has the advantage that, due to its nature of learning parameters to restore a trained image, fewer adjustments are required to deal with unknown objects or disturbances, and a small amount of training data is also required. Therefore, even if the target image changes in response to changes in the surrounding environment of the imaging unit 2, the learning model 5b generates an output image corresponding to the target image, eliminating the need for adjustments according to various conditions. This completes the processing of step S12, and the learning process ends.

[0038] As is clear from the above description, the detection system 1 according to one embodiment of the present invention includes an image restoration generation unit 6a that generates a detection target image corresponding to the determination target image by inputting data of the determination target image into a learning model in which data of a processed image is input data and data of a detection target image including a detection target object corresponding to the processed image is output data; a difference calculation unit 6b that generates a difference image between the determination target image and the detection target image generated by the image restoration generation unit 6a; and a detection unit 6d that determines whether the determination target image is a detection target image based on the difference image generated by the difference calculation unit 6b. This allows for accurate detection of a detection target object present in the monitored area even if an obstacle or an object with low transmittance is present between the monitored area and the image capture unit 2. Furthermore, the model generation unit 4 and the learning database 5 allow for generation of a learning model that accurately detects a detection target object present in the monitored area even if an obstacle or an object with low transmittance is present between the monitored area and the image capture unit 2.

[0039] In this example, the present invention was applied to a process for determining whether a metal tube (whopper) supporting a metal product to prevent the inner diameter of the metal product from being crushed when it is wound during the manufacturing process of the metal product was used as the detection object. Furthermore, a wire mesh fence, dustproof and drip-proof cover, etc. were assumed as obstacles between the detection object and the imaging unit 2. In the learning model generation process, images of the monitored area with the whopper inserted in its designated position were captured, and 25 captured images were trained using an autoencoder to generate learning model 5b. The autoencoder had the following layer structure, and structural similarity indexes were used to evaluate changes in pixel values, contrast, and structure. This resulted in a learning model that generates images of the whopper inserted in its designated position.

[0040] ・Encoder: Compresses a 150x150 pixel image to 100 dimensions. ・Decoder: Restores the 100 dimensions to a 150x150 pixel image. ・Training times: 500 epochs. ・Loss function: SSIM (structural similarity).

[0041] When an image (regardless of whether or not a detection target object is present) is input into this learning model, an image of the whopper inserted in the designated position is output. The difference between the input image and the output image is then calculated. The difference image between the input image with the whopper inserted in the designated position and the output image with the whopper inserted in the designated position is a black image. Next, the difference image was binarized, and the areas other than the designated position in the difference image were masked to count the number of pixels with high brightness (white areas). The presence or absence of a detection target object in the input image was determined by determining whether the count value was equal to or less than a predetermined threshold. The threshold values ​​used were a first threshold (= 30) and a second threshold (= 100), which were determined in advance based on the image size of 150 x 150 pixels and the size of the area where the detection target object is present. If the count value was equal to or less than the first threshold, the scene was determined to have the whopper inserted in the designated position. If the count value was equal to or greater than the first threshold but less than the second threshold, the scene was determined to have the whopper not inserted in the designated position. Furthermore, if the count value is greater than the second threshold, it is determined that the obstacle is moving or that the scene is unknown. As a result, as shown in Figure 6, even if an obstacle or an object with low transmittance is present between the monitored area and the imaging unit, it is possible to accurately determine whether the detection target object is captured in the image to be determined.

[0042] Although the present invention has been described above as an embodiment, the present invention is not limited to the descriptions and drawings that form part of the disclosure of the present invention. In other words, other embodiments, examples, and operational techniques that can be made by those skilled in the art based on the present invention are all included in the scope of the present invention.

[0043] According to the present invention, it is possible to provide a detection device, a detection system, and a detection method that can accurately detect a detection object that exists in a monitored area even if an obstacle or an object with low transmittance exists between the monitored area and the imaging device. Also, according to the present invention, it is possible to provide a model generation device that can generate a learning model that accurately detects a detection object that exists in a monitored area even if an obstacle or an object with low transmittance exists between the monitored area and the imaging device.

[0044] REFERENCE SIGNS LIST 1 Detection system 2 Imaging unit 3 Data acquisition unit 4 Model generation unit 5 Learning database 5a Learning source data 5b Learning model 6 Calculation unit 6a Image restoration generation unit 6b Difference calculation unit 6c Image processing unit 6d Detection unit 6e Output unit

Claims

1. A detection device for determining whether a determination target image is a detection target image including a detection target object, comprising: An image restoration generation unit that inputs data of the determination target image to a learning model that uses data of a processed image as input data and data of a detection target image including the detection target object corresponding to the processed image as output data, thereby generating a detection target image corresponding to the determination target image; A difference calculation unit that generates a difference image between the determination target image and the detection target image generated by the image restoration generation unit; A detection unit that determines whether the determination target image is the detection target image based on the difference image generated by the difference calculation unit.

2. The detection device according to claim 1, further comprising an image processing unit that performs image processing for emphasizing features of the detection target object on the difference image generated by the difference calculation unit, wherein the detection unit determines whether the determination target image is the detection target image based on the difference image on which the image processing has been performed by the image processing unit.

3. The detection device according to claim 1 or 2, wherein the detection unit determines whether the determination target image is the detection target image based on the number of pixels in the difference image whose luminance value is equal to or greater than a predetermined threshold value.

4. The detection device according to any one of claims 1 to 3, further comprising a model generation unit that generates the learning model by learning to reconstruct the same detection target image as the learning source data or a detection target image approximating the learning source data using the data of the detection target image as the learning source data.

5. The detection device according to claim 4, wherein the model generation unit is an autoencoder.

6. The detection device according to claim 4 or 5, wherein the model generation unit adds data of the determination target image determined by the detection unit to be a detection target image to the learning source data and regenerates the learning model.

7. A detection system, comprising: The detection device according to any one of claims 1 to 6; An imaging unit that captures an image of a monitoring target area and outputs data of the captured image of the monitoring target area to the detection device as data of the determination target image.

8. A detection method for determining whether a determination target image is a detection target image including a detection target object, the method comprising: an image restoration generation step of inputting data of the determination target image to a learning model that uses data of a processed image as input data and data of a detection target image including the detection target object corresponding to the processed image as output data, thereby generating a detection target image corresponding to the determination target image; a difference calculation step of generating a difference image between the determination target image and the detection target image generated in the image restoration generation step; and a detection step of determining whether the determination target image is the detection target image based on the difference image generated in the difference calculation step.

9. A model generation device for generating a learning model for determining whether a determination target image is a detection target image including a detection target object, the device comprising a model generation unit that generates a learning model that uses data of a processed image as input data and data of a detection target image including the detection target object corresponding to the processed image as output data, wherein the model generation unit generates the learning model by learning to reconstruct the same detection target image as the learning source data or a detection target image approximating the learning source data using the data of the detection target image as the learning source data.

Images