Image generation system and smoke detection device

Abstract

To generate an image for a teacher for machine learning which can enhance detection accuracy even to smoke which does not provide satisfactory detection accuracy in a conventional smoke detection device like white smoke on a white background.SOLUTION: An image generation device 10 includes: a first image generation unit 26 for generating an image for a teacher to be used for machine learning of a smoke detection device for detecting a fire in a monitoring area, and generating a first image for a teacher 2600 by first processing for suppressing contrast of bright parts and highlighting contrast of dark parts to an original image for a teacher including the monitoring area and the smoke; and a second image generation unit 28 for generating a second image for a teacher 2800 by second processing for suppressing the contrast of the dark parts and highlighting the contrast of the bright parts to the original image for a teacher. Machine learning of the smoke detection device is performed by using the first image for a teacher 2600 and a second image for a teacher 2800.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image generation system that generates teacher images to be used in machine learning to detect fires in a monitored area, and a smoke detection device that detects fires through machine learning using the teacher images generated by the image generation system. [Background technology]

[0002] Conventionally, in a smoke detection device that detects fires in a monitored area by inputting an image (surveillance image) of the monitored area taken by a surveillance camera into a multi-layer neural network or the like, a learning phase involves generating a large number of teacher images of fire and non-fire situations and having the smoke detection device perform machine learning, and during monitoring as an inference phase, the surveillance images taken by the surveillance camera are input into the trained smoke detection device, and the output results are used to make a highly accurate estimation of whether a fire exists and to output an alarm (Patent Document 1).

[0003] A conventional image generation system for generating teacher images, for example, places a surveillance image of a monitored area where no fire is occurring in a virtual space, and places a smoke image generated using computer graphics (CG) or the like in the virtual space by adjusting its shape, color, direction, etc., and generates an image including the surveillance image and the smoke image as a teacher image.This makes it possible to easily generate teacher images in various patterns that accurately reproduce actual fire situations, and improves the efficiency of generating teacher images (Patent Document 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2018-088630 A [Patent Document 2] Patent Publication No. 2021-149378 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in smoke detection devices equipped with multi-layer neural networks that have undergone machine learning using training images generated by such conventional image generation systems, the detection accuracy is poor for surveillance images in which white smoke is present on a white background, and even for black or thin smoke, there are problems with detection accuracy depending on the background.

[0006] The present invention aims to provide an image generation system that generates training images for machine learning, which enables improved detection accuracy even for smoke for which conventional smoke detection devices have poor detection accuracy, such as white smoke against a white background, and a smoke detection device that has high detection accuracy and is trained by machine learning using the training images generated by the image generation system. [Means for solving the problem]

[0007] (Image Generation System) The present invention provides an image generation system for generating training images to be used in machine learning of a smoke detection device for detecting fires in a monitored area, comprising: a first image generating unit that generates a first teacher image by a first process that suppresses contrast in bright areas and emphasizes contrast in dark areas of a teacher original image that includes a monitoring area and smoke; a second image generating unit that generates a second teacher image by performing a second process for suppressing contrast in dark areas and enhancing contrast in bright areas of the teacher original image; Equipped with A feature of the present invention is that machine learning is performed on the smoke detection device using a first teacher image and a second teacher image.

[0008] (Gamma correction for the first process) As a first process, the first image generating unit performs gamma correction by setting a gamma value greater than 1 and raising the brightness of the teacher original image to the power of (1 / γ).

[0009] (γ value of the first process) In the gamma correction in the first process, the gamma value is set to a predetermined value of 2 or more and 3 or less.

[0010] (Gamma correction for the second process) As the second processing, the second image generating section performs gamma correction by setting the gamma value to a value less than 1 and raising the brightness of the teacher original image to the power of (1 / γ).

[0011] (γ value of the second treatment) In the gamma correction in the second process, the gamma value is set to a predetermined value of 0.5 or more.

[0012] (Generating teacher images that suppress contrast in medium brightness areas) The image processing unit is provided with a third image generating unit that generates a third teacher image by a third process that suppresses the contrast of the intermediate brightness areas of the original teacher image that are between the bright and dark areas.

[0013] (Smoke detection device) In another aspect of the present invention, there is provided a smoke detection apparatus comprising: A smoke detection unit that performs machine learning using a plurality of training images generated by the image generation system described above is provided, A feature of this system is that a surveillance image of a surveillance area is input to a smoke detection unit to detect a fire.

[0014] (Machine learning using the first and second training images) The smoke detection unit performs machine learning using a first teacher image and a second teacher image.

[0015] (Smoke detection unit trained by machine learning using different training images) The smoke detection unit is A first smoke detection unit that performs machine learning using a first teacher image and a second teacher image; A second smoke detection unit that performs machine learning using the first training image; a third smoke detection unit that performs machine learning using the second training image; Equipped with A surveillance image is input to each of the first smoke detection unit, the second smoke detection unit, and the third smoke detection unit, and smoke is detected based on the detection results output.

[0016] (Smoke detection using a segmented image divided into brightness regions) The smoke detection unit divides the surveillance image into a plurality of brightness regions having different predetermined brightness ranges, and inputs the divided images corresponding to each of the brightness regions to the first smoke detection unit, the second smoke detection unit, and the third smoke detection unit, and detects smoke based on the detection results output.

[0017] (Smoke detection using a segmented image divided into high-brightness and low-brightness areas) The smoke detection unit is The brightness area of ​​the surveillance image is divided into high brightness area and low brightness area, A divided image corresponding to a low brightness region is input to a first smoke detection unit and a third smoke detection unit; A divided image corresponding to a high luminance region is input to a first smoke detection unit and a second smoke detection unit.

[0018] (Smoke detection unit trained by machine learning using training images) The smoke detection unit further includes: A fourth smoke detection unit that performs machine learning using the original teacher image as a teacher image, A surveillance image is input to each of the first smoke detection unit, the second smoke detection unit, the third smoke detection unit, and the fourth smoke detection unit, and smoke is detected based on the detection results output. Effect of the Invention

[0019] (Effects of image generation system) The present invention is an image generation system for generating teacher images used in machine learning of a smoke detection device that detects fires in a monitored area, the system comprising: a first image generation unit that generates a first teacher image by a first process of suppressing the contrast of bright parts and emphasizing the contrast of dark parts of an original teacher image including the monitored area and smoke; and a second image generation unit that generates a second teacher image by a second process of suppressing the contrast of dark parts and emphasizing the contrast of bright parts of the original teacher image. In order to perform machine learning of the smoke detection device using the first teacher image and the second teacher image, a first teacher image with weakened contrast on the bright side and a second original teacher image with weakened contrast on the dark side are generated, thereby creating an image with low contrast against the background (small difference in brightness with the background) that resembles a state where smoke is just beginning to emerge, and by performing machine learning of the smoke detection device using the multiple teacher images (first teacher image and second teacher image) generated in this way, the detection accuracy of smoke close to the state where smoke that resembles the background color is just beginning to emerge is improved, thereby improving the detection accuracy of various smokes and making it possible to suppress and prevent erroneous detection of objects and phenomena other than smoke.

[0020] Furthermore, the first process emphasizes the contrast of dark areas, making it possible to perform machine learning using a first training image that emphasizes how a thin layer of smoke appears against a dark background with a clear pattern, such as a shelf, and how the background pattern changes due to the smoke.Similarly, the second process emphasizes the contrast of bright areas, making it possible to perform machine learning using a second training image that emphasizes how a thin layer of smoke appears against a white background with a clear pattern, such as a shelf, and how the background pattern changes due to the smoke.This is expected to have the effect of improving the detection accuracy of thin smoke against a background with a clear pattern, where detection accuracy was poor with conventional smoke detection devices.

[0021] (Effect of gamma correction in the first process) Furthermore, as a first process, the first image generation unit performs gamma correction by setting the gamma value to a value greater than 1 and raising the brightness of the original teacher image to the power of (1 / γ). More specifically, by setting the gamma value to a predetermined value between 2 and 3, machine learning is possible using the first teacher image that approximates a state in which white smoke is beginning to appear on a white background (light background) or a state in which thin smoke is appearing on a white background. This makes it possible to improve the accuracy of smoke detection, including the accuracy of detecting white smoke on a white background, which was poor with conventional smoke detection devices.

[0022] (Effect of gamma correction in the second process) In addition, as a second process, the second image generation unit performs gamma correction by raising the brightness of the original teacher image to the power of (1 / γ) with a gamma value less than 1. More specifically, by setting the gamma value to a predetermined value of 0.5 or more (less than 1), machine learning is possible using a second teacher image that approximates a state in which black smoke has begun to appear on a black background (dark background) or a state in which thin smoke is appearing on a black background. This makes it possible to improve the accuracy of smoke detection, including the accuracy of detecting black smoke, which was poor with conventional smoke detection devices.

[0023] (Effect of generating teacher images that suppress the contrast in medium brightness areas) In addition, the system is provided with a third image generation unit that generates a third teacher image by a third process that suppresses the contrast of the medium brightness areas of the original teacher image that are between the bright and dark areas.This makes it possible to perform machine learning using the third teacher image that resembles a state in which gray or thin smoke is beginning to emerge against a medium brightness background, thereby improving the accuracy of smoke detection.

[0024] (Effectiveness of smoke detectors) Another form of the present invention is a smoke detection device comprising a smoke detection unit that performs machine learning using a plurality of teacher images generated by the above-described image generation system, and a surveillance image of a monitored area is input to the smoke detection unit to detect a fire. Therefore, even when the contrast between the background and the smoke is low, it is possible to improve the smoke detection accuracy and correctly detect a fire.

[0025] (Effects of machine learning using the first and second teaching images) The smoke detection unit performs machine learning using the first teacher image and the second teacher image, so that high smoke detection accuracy can be achieved even in surveillance images where white smoke exists on a white background or black smoke exists on a black background, making it possible to correctly detect fires.

[0026] (Effect of smoke detection trained by machine learning using different training images) In addition, the smoke detection unit comprises a first smoke detection unit that has performed machine learning using a first teacher image and a second teacher image, a second smoke detection unit that has performed machine learning using the first teacher image, and a third smoke detection unit that has performed machine learning using the second teacher image, and a surveillance image is input into each of the first smoke detection unit, the second smoke detection unit, and the third smoke detection unit and smoke is detected based on the detection results output.Therefore, the first smoke detection unit can achieve high detection accuracy for surveillance images in which white smoke exists on a white background and surveillance images in which black smoke exists on a black background, the second smoke detection unit can achieve high detection accuracy for surveillance images in which white smoke exists on a white background, and further, the third smoke detection unit can achieve high detection accuracy for surveillance images in which black smoke exists on a black background, and by detecting smoke based on the detection results (detection values) output from the first to third smoke detection units, it is possible to improve the smoke detection accuracy.

[0027] (Effect of smoke detection using a segmented image divided into brightness regions) In addition, the smoke detection unit divides the surveillance image into a number of brightness regions having different predetermined brightness ranges, and inputs the divided images corresponding to each of the brightness regions into the first smoke detection unit, the second smoke detection unit and the third smoke detection unit, and detects smoke based on the detection results output.This makes it possible for the smoke detection unit to perform detection using machine learning training data from training images in which the contrast is suppressed or emphasized corresponding to the different brightness regions, thereby improving the accuracy of smoke detection.

[0028] (Effect of smoke detection using a segmented image divided into high-brightness and low-brightness areas) In addition, the smoke detection unit divides the brightness areas of the surveillance image into high-brightness areas and low-brightness areas, and inputs the divided image corresponding to the low-brightness areas to the first smoke detection unit and the third smoke detection unit, and inputs the divided image corresponding to the high-brightness areas to the first smoke detection unit and the second smoke detection unit.Therefore, by inputting the divided image of the high-brightness area, such as white smoke on a white background, to the first smoke detection unit and the second smoke detection unit, and the divided image of the low-brightness area, such as black smoke on a black background, to the first smoke detection unit and the third smoke detection unit, high smoke detection accuracy can be achieved and fires can be detected reliably.

[0029] (Effect of smoke detection using machine learning with training images) In addition, the smoke detection unit further includes a fourth smoke detection unit that has performed machine learning using the original teacher image as a teacher image, and detects smoke based on the detection results output when a surveillance image is input into each of the first smoke detection unit, the second smoke detection unit, the third smoke detection unit and the fourth smoke detection unit.Therefore, for surveillance images in which the smoke detection accuracy is reduced or insufficient when machine learning is performed using a teacher image (such as the first teacher image or the second original teacher image) in which the contrast of specific parts is suppressed or emphasized, for example, surveillance images in which the contrast of smoke against the background is sufficient and appropriate, the detection accuracy may be reduced or insufficient by the smoke detection unit that has performed machine learning using a teacher image that has been processed to suppress or emphasize the contrast.Therefore, by including a fourth smoke detection unit that has performed machine learning using an original teacher image that has not been processed to suppress or emphasize the contrast, it is possible to improve the smoke detection accuracy. [Brief description of the drawings]

[0030] [Figure 1] FIG. 2 is an explanatory diagram showing a functional configuration of the image generating system. [Diagram 2] FIG. 13 is an explanatory diagram showing an example of an original teacher image; [Diagram 3] 1 is an explanatory diagram showing gamma correction characteristics of input pixel values ​​and output pixel values ​​processed in accordance with a γ value; [Figure 4] 1 is an explanatory diagram showing a functional configuration for machine learning an image recognition model using a training image generated by an image generation device. FIG. [Diagram 5] This is an explanatory diagram showing, in list format, the smoke detection results obtained by a smoke detection device that has undergone machine learning using a teacher image generated by gamma correction. [Figure 6] 4 is a flowchart showing an image generation process by a control unit of the image generating device of FIG. [Figure 7] 1 is an explanatory diagram showing a first embodiment of a smoke detection device equipped with an image recognition model of "YOLO" trained using a first teacher image and a second teacher image. FIG. [Figure 8] FIG. 11 is an explanatory diagram showing an example of a display screen on which an output result of an image recognition model is displayed. [Figure 9] FIG. 11 is an explanatory diagram showing a second embodiment of a smoke detection device including a plurality of image recognition models trained using different training images. [Figure 10] FIG. 11 is an explanatory diagram showing a third embodiment of a smoke detection device equipped with a plurality of image recognition models that divide a monitoring image into regions according to brightness and recognize a smoke image. [Figure 11] 1 is an explanatory diagram showing a high brightness region divided image and a low brightness region divided image divided from a monitoring image; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An image generating system and a smoke detecting device according to an embodiment of the present invention will be described in detail below with reference to the accompanying drawings. It should be noted that the present invention is not limited to the following embodiment.

[0032] [Basic Concept of the Embodiment] First, a basic concept of the embodiment will be described. The embodiment generally relates to an image generation system that generates training images to be used in machine learning of a smoke detection device that detects fires in a monitored area.

[0033] Here, the term "monitoring area" refers to an area to be monitored, which is an outdoor or indoor space having a certain extent, and is a concept including areas such as rooms, corridors, and staircases of a building.

[0034] The "smoke detection device" is equipped with an image recognition model that is machine-learned using a teacher image to correctly detect fires. "Machine learning" means that a machine (computer) learns, and is, for example, an algorithm that inputs a teacher image containing smoke, which is given in advance and whose correct output for the input is known, into an image recognition model to learn the relationship between the output, and after learning, an unknown surveillance image is input to estimate the output. The "image recognition model" is a concept that includes known multi-layer (deep) neural networks (DNN), random forests, support vector machines (SVM), and the like. In the embodiment, a known image recognition algorithm known as "YOLO (You Only live Once)" that detects and identifies objects such as smoke images present in an image in parallel is used as the "image recognition model".

[0035] The image generating system of the embodiment includes a first image generating section and a second image generating section.

[0036] The "first image generating section" generates a first teacher image by performing a first process for suppressing contrast in bright areas and enhancing contrast in dark areas of an original teacher image.

[0037] Here, the "original teacher image" is an image that includes a surveillance area and smoke, and includes, for example, an image generated by placing a surveillance image capturing an image of a surveillance area in a virtual space and then arranging a smoke image generated using computer graphics (CG) or the like in the virtual space while changing the shape, color, direction, etc. of the smoke image.

[0038] "Contrast" refers to the difference between the light and dark parts of an image. The higher the contrast, the brighter the light parts are and the darker the dark parts are, making the difference between light and dark more obvious. Conversely, the lower the contrast, the smaller the difference between light and dark parts is, making the image appear flat.

[0039] Additionally, "contrast in bright areas" in the original teacher image refers to the "difference in brightness" in the bright areas of the image, while "contrast in dark areas" refers to the "difference in brightness" in the dark areas of the image. Therefore, "suppressing the contrast in bright areas" means reducing the difference in brightness (reducing the contrast) in the bright areas of the original teacher image, and in the white smoke image, reducing the difference in brightness between the "background and smoke" or "inside the smoke area," while "emphasizing the contrast in dark areas" means increasing the difference in brightness (increasing the contrast) in the dark areas of the original teacher image.

[0040] The "second image generating unit" generates a second teacher image by a second process that suppresses the contrast in dark areas and emphasizes the contrast in bright areas of the original teacher image. Here, "suppressing the contrast in dark areas" means reducing the difference in brightness (lowering the contrast) in the dark areas of the original teacher image, and in the case of a black smoke image, reducing the difference in brightness between the "background and smoke" or "inside the smoke area," and "emphasizing the contrast in bright areas" means increasing the difference in brightness (increasing the contrast) in the bright areas of the original teacher image.

[0041] This creates an image that resembles the state when smoke is just starting to emerge or when thin smoke is emerging, with a small difference in brightness between the smoke and the background image.By performing machine learning on the smoke detection device using multiple training images (first training image and second training image) generated in this way, it is possible to improve the detection accuracy of smoke close to the state when smoke is just starting to emerge and thin smoke that is similar to the background color, thereby improving the detection accuracy of various types of smoke and making it possible to suppress and prevent erroneous detection of smoke.

[0042] Furthermore, as a first process, the "first image generating unit" performs gamma correction, for example, by setting the gamma value to a value greater than 1 and raising the brightness of the original teacher image to the power of (1 / γ). Here, in gamma correction with a gamma value greater than 1, the larger the gamma value, the greater the degree to which the contrast of bright areas is suppressed and the contrast of dark areas is emphasized. The inventors of the present application evaluated an image recognition model trained by machine learning using teacher images generated by changing the gamma value under the condition that the gamma value is greater than 1, and found that high detection accuracy for white smoke was obtained when the gamma value was set to 2 or more and 3 or less.

[0043] Furthermore, as a second process, the "second image generating unit" performs gamma correction, for example, by raising the brightness of the original teacher image to the power of (1 / γ) with a γ value of less than 1. Here, in gamma correction with a γ value of less than 1, the smaller the γ value, the greater the degree to which the contrast of dark areas is suppressed and the contrast of bright areas is emphasized. The inventors of the present application evaluated an image recognition model trained by machine learning using teacher images generated by changing the γ value under the condition that the γ value is less than 1, and found that high detection accuracy for black smoke was obtained when the γ value was set to 0.5 or more.

[0044] The image generation system may further include a "third image generation unit" that generates a third teacher image by a third process that suppresses the contrast of the medium brightness area between the bright and dark areas of the original teacher image. By including the "third image generation unit," a third teacher image is generated that is corrected to suppress the contrast of the medium brightness area for an original teacher image in which gray or light smoke is present against a medium brightness background, and machine learning of the smoke detection device is performed on the third teacher image, thereby making it possible to improve the smoke detection accuracy even in a surveillance image in which gray or light smoke is present against a medium brightness background.

[0045] The embodiment also relates to a smoke detection device that is machine-learned using a plurality of training images generated by the image generation system described above.

[0046] The "smoke detection device" is equipped with a smoke detection unit that has undergone machine learning using multiple training images generated by the image generation system described above, and detects fires by inputting surveillance images into the smoke detection unit. This makes it possible to improve the smoke detection accuracy of the smoke detection unit and correctly detect fires even when the contrast between the background and the smoke is low.

[0047] In addition, a "surveillance image" is, for example, an image captured by an imaging device connected to a smoke detection device, and an "imaging device" is a device that is installed in a monitored area and captures images of the monitored area in which it is installed, and includes concepts such as television cameras that capture moving images, such as ITV cameras, and surveillance cameras.

[0048] In addition, the "smoke detection unit" is equipped with an image recognition model that uses the above-mentioned "YOLO" or the like, and machine learning of the image recognition model is performed using the first teacher image and the second teacher image generated by the above-mentioned image generation system. Therefore, the smoke detection unit that has completed the machine learning can achieve high smoke detection accuracy and reliably detect fires even in surveillance images where white smoke exists on a white background or black smoke exists on a black background.

[0049] In addition, the "smoke detection unit" may include a first smoke detection unit that has performed machine learning using a first teacher image and a second teacher image, a second smoke detection unit that has performed machine learning using the first teacher image, and a third smoke detection unit that has performed machine learning using the second teacher image, and may be configured to input a surveillance image to each of the first to third smoke detection units and detect smoke based on the detection results (detection values) output.

[0050] By providing first, second, and third smoke detection units, the first smoke detection unit can obtain high detection accuracy for surveillance images in which white smoke exists on a white background and surveillance images in which black smoke exists on a black background, the second smoke detection unit can obtain high detection accuracy for surveillance images in which white smoke exists on a white background, and further, the third smoke detection unit can obtain high detection accuracy for surveillance images in which black smoke exists on a black background, making it possible to further improve the accuracy of smoke detection.

[0051] The "smoke detection unit" may also be configured to divide the surveillance image into a plurality of brightness regions having different predetermined brightness ranges, input divided images corresponding to the respective brightness regions to the first smoke detection unit, the second smoke detection unit, and the third smoke detection unit, and detect smoke based on the detection results output. This enables smoke detection by the smoke detection unit that has been machine-trained using training images in which the contrast is suppressed or enhanced corresponding to the different brightness regions, thereby improving the accuracy of smoke detection.

[0052] The "smoke detection unit" divides the brightness area of ​​the surveillance image into high-luminance areas and low-luminance areas, inputs a divided image corresponding to the low-luminance areas to the first smoke detection unit and the third smoke detection unit, and inputs a divided image corresponding to the high-luminance areas to the first smoke detection unit and the second smoke detection unit. As a result, a divided image of a high-luminance area, such as white smoke on a white background, is input to the first smoke detection unit and the second smoke detection unit, and a divided image of a low-luminance area, such as black smoke on a black background, is input to the first smoke detection unit and the third smoke detection unit, thereby achieving high smoke detection accuracy.

[0053] The "smoke detection unit" may further include a fourth smoke detection unit that performs machine learning using the original teacher image as a teacher image, and detects smoke based on the detection results outputted by inputting the surveillance image to each of the first smoke detection unit, the second smoke detection unit, the third smoke detection unit, and the fourth smoke detection unit. As a result, for surveillance images in which the smoke detection accuracy is reduced or insufficient when machine learning is performed using a teacher image (such as the first teacher image or the second original teacher image) in which the contrast of a specific portion is suppressed or enhanced, for example, surveillance images in which the contrast of the smoke against the background is sufficient and appropriate, the detection accuracy may be reduced or insufficient when the smoke detection unit performs machine learning using the teacher image that has been processed to suppress or enhance the contrast. Therefore, by providing a fourth smoke detection unit that performs machine learning using the original teacher image that has not been processed to suppress or enhance the contrast, it is possible to improve the smoke detection accuracy.

[0054] Specific embodiments will be described below. In the specific embodiments described below, a case where the "monitoring area" is an "area inside a factory facility", specifically a "parts storage facility" will be described. Also, a case where "gamma correction" is performed as the "first process and second process" will be described. Also, an image recognition model provided in the smoke detection device will be described using "YOLO" as an example.

[0055] [Specific details of the embodiment] The imaging system and the smoke detection device will now be described in more detail, which will be divided into the following sections: a. Functional configuration of the image generation system b. Generation of original images for training c. Generation of teacher images using gamma correction d. Accuracy of smoke detection by a smoke detector trained by machine learning using training images d1. Image recognition model for smoke detection device d2. Machine learning for smoke detection devices d3. Smoke detection accuracy e. Generation of training images f. First embodiment of smoke detection device g. Second embodiment of smoke detection device h. Third embodiment of smoke detection device i. Variations of the Invention

[0056] [a. Functional configuration of the image generation system] The functional configuration of the image generation system will be described with reference to Fig. 1 showing the functional configuration of the image generation system.

[0057] As shown in Fig. 1, the image generation system according to the embodiment is configured with an image generation device 10. The image generation device 10 is a known computer device equipped with a CPU, a memory, and various input / output ports, and includes a control unit 12, an operation unit 14, a display unit 16, a communication unit 18, and a teacher image storage unit 20.

[0058] The operation unit 14 is provided with known input devices such as a keyboard, a mouse, etc. The display unit 16 displays images and the like required for image generation on a screen, and a known display or the like is used. The communication unit 18 is connected to an external device for communication and transmits and receives data including images, and is provided with known communication functions such as a local area network (LAN) or the Internet.

[0059] The teacher image storage unit 20 is a known storage device such as a memory or a hard disk, and has a storage capacity sufficient to store a large number of teacher images such as the first teacher image 2600 and the second teacher image 2800. The teacher image storage unit 20 may also include a removable external memory such as a USB memory, and may further be an external storage device connected via the communication unit 18.

[0060] The control unit 12 is provided with the functions of a teacher original image generating unit 22, a first image generating unit 26, and a second image generating unit 28, and these functions are realized by the execution of a program by the CPU.

[0061] [b. Generation of original images for teaching] Next, the generation of the original teacher image by the original teacher image generating unit provided in the control unit of the image generating device will be described with reference to Fig. 2 showing an example of the original teacher image.

[0062] The teacher original image generating section 22 generates an image that is the source of a teacher image including a monitoring area and smoke, that is, a so-called "teacher original image."

[0063] To generate a teacher original image, the teacher original image generating unit 22 prepares a surveillance image captured by a surveillance camera or the like of the surveillance area, and a smoke image that represents smoke generated by a fire. If the brightness of the surveillance area is constant, a surveillance image is prepared by storing in a memory or the like a single surveillance image captured at the installation position of the surveillance camera. If the brightness of the surveillance area varies depending on the time of day, surveillance images for each time period with different brightness are captured by the surveillance camera and stored in a memory or the like. The surveillance images captured and prepared by the surveillance camera are still images obtained on a frame-by-frame basis.

[0064] Any type of surveillance camera may be used to capture images of a surveillance area, but for example, the surveillance camera may acquire image data of RGB frame images of the surveillance area at 30 or 60 frames per second from a video consisting of a series of color frame images captured sequentially at a frame rate of, for example, 30 or 60 frames per second, and store one of the images in memory. Here, the size of the video captured by the surveillance camera may be any size, but may be, for example, 1280 x 720 pixels. The RGB frame images stored in memory or the like are converted, for example, into grayscale images or VHS frame images in the VHS color space in order to generate original images for teaching.

[0065] Furthermore, the teacher original image generating unit 22 generates a three-dimensional smoke image using computer graphics (CG) or the like, and stores it in a memory or the like to prepare it. Here, the smoke image stored in a memory or the like to prepare it is one that can be adjusted as necessary in terms of size, shape, color, brightness, position, or smoke density, rising direction, etc. For example, by adjusting the color of the smoke image, it is possible to deal with various types of smoke, such as white smoke, gray smoke, or black smoke, and it is possible to generate a smoke image in which the smoke changes stepwise from white to black.

[0066] The teacher original image generating unit 22 constructs a three-dimensional virtual space on a memory, arranges a two-dimensional surveillance image showing a surveillance area in the virtual space as a background image, arranges (composites) a smoke image in a place in the surveillance image (background image) where a fire is expected to break out, generates a teacher original image including the surveillance area and smoke, and stores it in a memory, etc. The storage device such as a memory that stores the teacher original image may be the same storage device as the teacher image storage unit 20, or may be a storage device separate from the teacher image storage unit 20.

[0067] Figure 2 is an example of a teacher original image 30 generated by the teacher original image generation unit 22, in which a surveillance image 32 captured by a surveillance camera is placed as the background image, with the surveillance area being a "parts storage facility" inside a factory facility, and a smoke image 34 is placed at any position in the surveillance image 32 where a fire is expected to break out, representing the fire source.

[0068] Here, the teacher original image generating unit 22 generates and stores in a memory or the like at least a teacher original image 30 in which the smoke image 34 is arranged in a bright part of the surveillance image 32, and a teacher original image 30 in which the smoke image 34 is arranged in a dark part of the surveillance image 32. In this way, a large number of teacher original images 30, for example about 10,000, which are the source of teacher images required for machine learning of the smoke detection device, are generated and stored in a memory or the like.

[0069] Here, the generation of the original teacher image 30 may be performed manually by a person in charge, or automatically by the control unit 12 of the image generating device 10. The method of automatic processing is arbitrary, but for example, points are manually or automatically designated on the screen of the surveillance image 32 as bright or dark areas that are likely to be fire locations, and smoke images are automatically positioned at each point to generate the original teacher image 30, which is stored in a memory or the like.

[0070] [c. Generating teacher images using gamma correction] Next, the generation of a teacher image by gamma correction of a teacher original image will be explained. In this explanation, reference will be made to Fig. 3, which shows the gamma correction characteristics of input pixel values ​​and output pixel values ​​processed according to the γ value. Note that the horizontal axis of Fig. 3 indicates the input pixel value x, and the vertical axis indicates the output pixel value y, which corresponds to, for example, the luminance value of a grayscale image or the V value (brightness) in the HVS color space.

[0071] 1, the control unit 12 is provided with a first image generating unit 26 and a second image generating unit 28. The first image generating unit 26 generates a first teacher image 2600 by performing a first process for suppressing the contrast of bright parts and emphasizing the contrast of dark parts for the teacher original image generated by the teacher original image generating unit 22, for example, in which a smoke image is arranged in a bright part of the monitoring area, and stores the generated first teacher image 2600 in the teacher image storage unit 20.

[0072] In addition, the second image generation unit 28 generates a second teacher image 2800 by performing a second process that suppresses the contrast of dark areas and emphasizes the contrast of bright areas for the teacher original image generated by the teacher original image generation unit 22, for example, in which a smoke image is placed in a dark part of the monitoring area, and stores the second teacher image 2800 in the teacher image memory unit 20.

[0073] The first process by the first image generating unit 26 to suppress the contrast in bright areas and emphasize the contrast in dark areas, and the process by the second image generating unit 28 to suppress the contrast in dark areas and emphasize the contrast in bright areas are optional, but in this embodiment, contrast suppression process is performed by gamma correction.

[0074] Here, gamma correction for the original teacher image refers to performing a nonlinear brightness correction process on the original teacher image, and converts the input pixel value x (the pixel value of the original teacher image) into the output pixel value y (the pixel value of the teacher image to be generated) according to the following formula. y=255×(x / 255) 1 / γ (Equation 1) As shown in FIG. 3, the gamma correction characteristic based on the above (Equation 1) is a linear characteristic in which, when the γ value is 1, the input pixel value x becomes the output pixel value y as it is.

[0075] For example, when the original teacher image has a smoke image placed in a bright part of the monitoring area, the first image generation unit 26 performs a first process on the original teacher image to suppress the contrast of the bright parts and emphasize the contrast of the dark parts, by setting the gamma value in (Equation 1) to a value greater than 1 and performing gamma correction by raising the brightness of the original teacher image to the power of (1 / γ).

[0076] In this way, when the gamma value is set to a value greater than 1, for example, as shown by the characteristic curves shown by the solid lines in Fig. 3 for gamma values ​​of "1.5", "2.0", "2.5", and "3.5", the larger the input pixel value x (the brighter the brightness), the smaller the difference in output pixel value y becomes for the same difference in input pixel value x, so that the first process suppresses contrast in areas with large pixel values ​​(brighter brightness) and emphasizes contrast in areas with small pixel values ​​(darker brightness), generating, for example, a teacher image approximating a state in which white smoke is beginning to appear on a white background (bright background) or an original teacher image approximating a state in which thin smoke is appearing on a white background. Also, the larger the gamma value, the greater the degree to which contrast in bright areas is suppressed and contrast in dark areas is emphasized.

[0077] On the other hand, when the original teacher image has, for example, a smoke image placed in a dark part of the monitoring area, the second image generation unit 28 performs a second process on the original teacher image to suppress the contrast in dark parts and emphasize the contrast in bright parts, by setting the gamma value in the above (Equation 1) to a value less than 1 and performing gamma correction by raising the brightness of the original teacher image to the power of (1 / γ).

[0078] In this way, when the gamma value is set to a value less than 1, for example, as shown by the characteristic curves shown by the dotted lines in Fig. 3 with gamma values ​​of "0.75", "0.66", "0.5", and "0.33", the smaller the input pixel value x (the darker the brightness), the smaller the difference in the output pixel value y for the same difference in input pixel value x, so that the second process suppresses contrast in areas with small pixel values ​​(dark brightness) and emphasizes contrast in areas with large pixel values ​​(bright brightness), generating, for example, a teacher image approximating a state in which black smoke is beginning to appear on a black background (dark background) or an original teacher image approximating a state in which thin smoke is appearing on a black background. Also, the smaller the gamma value, the greater the degree to which the contrast in dark areas is suppressed and the contrast in bright areas is emphasized.

[0079] [d. Smoke detection accuracy by a smoke detection device trained by machine learning using training images] Next, we will explain the smoke detection accuracy of a smoke detection device that has undergone machine learning using gamma-corrected training images.

[0080] (d1. Image recognition model for smoke detection device) We will explain the image recognition model provided in a smoke detection device that learns using a plurality of first teacher images 2600 generated by a first image generation unit 26 and a plurality of second teacher images 2800 generated by a second image generation unit 28.

[0081] This image recognition model is basically a convolutional neural network, but it is an image recognition model that uses "YOLO (You Only Look Once)", a well-known representative algorithm for object detection. "YOLO" performs the following steps (1) to (4).

[0082] (1) Division of S×S grid cells Divide the input image into S x S grid cells, for example 7 x 7 grid cells. If the center of an object is in a grid cell, that grid cell is trained to detect the object.

[0083] (2) Bounding box estimation Each grid cell has B bounding boxes, for example two bounding boxes, and predicts a confidence score for each bounding box. The confidence score indicates the probability that the bounding box contains an object rather than the background. The closer the confidence score is to 1, the more likely it is that an object is present. Each bounding box consists of five components: coordinate values ​​(x, y, w, h) and a confidence score (CS). The x and y coordinates represent the center of the box relative to the grid cell boundary. The width w and height h represent values ​​relative to the whole image. The confidence score is the difference between the predicted box and the ground truth box.

[0084] (3) Object Prediction Each grid cell predicts a conditional class probability P(class object) for C classes, e.g., 20 classes. By multiplying the calculated "conditional class probability" by the "confidence score of each bounding box", we can obtain a confidence score for the class of each bounding box, and based on this confidence score, we determine which bounding box detects the correct object.

[0085] (4) Selection The method is based on highly reliable bounding boxes and uses a method called NMS (Non-Maximum Suppression), which suppresses areas with high overlapping degrees using a threshold.

[0086] "YOLO" analyzes all areas contained in an image for learning, so it can identify and estimate with high accuracy while reducing false positives of the background. It integrates the data results of the grid cells and bounding boxes to identify what objects are present in the image. As mentioned above, the analysis of the grid cells is based on the class prediction probability of each cell, and the higher the class prediction probability, the higher the accuracy of the classification. The combination of this class prediction probability and the bounding box can accurately predict what objects are present in which areas of the image. In addition, image recognition models using "YOLO" are said to have faster processing speeds than those using other methods.

[0087] (d2. Machine learning for smoke detection devices) Next, machine learning of the smoke detection device (image recognition model) will be described. In this description, reference will be made to Fig. 4, which shows a functional configuration for machine learning the image recognition model using teacher images (first teacher image and second teacher image) generated by an image generating device.

[0088] As shown in FIG. 4, a learning control device 36 is provided for machine learning an image recognition model 38 using "YOLO" provided in the smoke detection device for the image generating device 10 shown in FIG.

[0089] The learning control device 36 is a known learning device, and performs machine learning of an image recognition model 38 using a plurality of first teacher images 2600 and a plurality of second teacher images 2800 stored in the teacher image storage unit 20.

[0090] Specifically, the learning control device 36 sets label data for the first teacher image 2600 and the second teacher image 2800 stored in the teacher image memory unit 20 indicating that the images should be learned as one of a number of classes, for example, as a "smoke class."

[0091] Next, learning is performed on the layers that make up the convolutional neural network, such as the convolutional layer, pooling layer, and fully connected layer of the image recognition model 38, according to a known backpropagation algorithm.

[0092] In the neural network of the image recognition model 38, convolution processing is performed using learned weights (filters) in the convolution layer, pooling (reduction) processing is performed in the pooling layer, and multiple units are provided in each fully connected layer and connected to multiple units in other layers, weights and bias values ​​are selected for each unit, and the vector product of multiple input values ​​and weights is calculated, the bias value is added to calculate the sum, and this is passed through a predetermined activation function and output to the units in the next layer, and forward propagation is performed in which values ​​are propagated until they reach the final layer. Note that the convolution layer and the pooling layer are alternately arranged multiple times on the input side of the fully connected layer.

[0093] In backpropagation learning, for example, a function of mean square error is used as the error for comparing the estimated value Y*, which is the result of forward propagation through the network, with the expected value Y, and the value is propagated from the rear of the network to the front while correcting the weights and biases. The corrected amount for each weight and bias is treated as a contribution to the error, and is calculated using the steepest descent method, and the value of the error function is minimized by changing the values ​​of the weights and biases.

[0094] (d3. Smoke detection accuracy) Next, a description will be given of the smoke detection accuracy by the trained image recognition model 38. In this description, reference will be made to Fig. 5 which shows, in a list format, the smoke detection results by the smoke detection device that has undergone machine learning using training images generated by gamma correction.

[0095] In the machine learning of the image recognition model 38 using "YOLO" shown in Fig. 4, a large number of first teacher images 2600 and second teacher images 2800 are divided into a plurality of blocks, a part of each block is used for learning, and the remaining blocks not used for learning are used for verification, and learning and verification are sequentially repeated for each block. Here, if the number of first teacher images 2600 and second teacher images 2800 is 10,000 each, for example, the first teacher images 2600 and second teacher images 2800 are divided into 10 blocks of 1000 each, and 900 images of each block are used for learning, and the remaining 100 images are used for verification.

[0096] The smoke detection results by the machine learning smoke detection device (image recognition model 38) for "white smoke on a white background" and "black smoke," for which conventional smoke detection devices had poor detection accuracy, are shown in Figure 5 for the cases where learning was performed using the first teacher image 2600, where learning was performed using the second teacher image 2800, and where learning was performed using both the first teacher image 2600 and the second teacher image 2800.

[0097] Also, "white smoke on a white background" is divided into "immediately after the fire breaks out" and "after a sufficient amount of time has passed," and "black smoke" is divided into "visibly thin smoke" and "visibly thick smoke." Figure 5 also shows the detection results when learning with a teacher image (original teacher image) that has not been gamma corrected, with a "circle" indicating that smoke was stably detected during the time period when smoke was generated, a "triangle" indicating that smoke was detected to some extent but also including cases where it was not detected, and a "cross" indicating that little or no smoke was detected. Note that in order for the smoke detection device to detect smoke, a threshold is set and evaluation is performed so that smoke is detected when the reliability score of smoke in processing by the image recognition model 38 is a predetermined value, for example, 0.5 or more.

[0098] First, when learning from a teacher image (original teacher image) that had not undergone gamma correction, for "white smoke on a white background," a "X" was obtained for both "immediately after the fire started" and "sufficient time had passed," and for "black smoke," a "triangle" was obtained for both "visibly thin smoke" and "visibly thick smoke."

[0099] In addition, when learning from the first teacher image 2600 (a teacher image gamma-corrected with a gamma value greater than 1), for "white smoke on a white background," a "triangle mark" is obtained "immediately after the fire breaks out," and a "circle mark" is obtained "after a sufficient amount of time has passed," and for "black smoke," both "visibly thin smoke" and "visibly thick smoke" are obtained as "X marks."

[0100] Furthermore, when learning using the second teacher image 2800 (a teacher image gamma-corrected with a gamma value less than 1), for "white smoke on a white background," a "X" was obtained for both "immediately after the fire started" and "sufficient time had passed," and for "black smoke," a "Circle" was obtained for both "visibly thin smoke" and "visibly thick smoke."

[0101] In addition, when learning using the first teacher image 2600 and the second teacher image 2800, for "white smoke on a white background," a "circle mark" was obtained for "immediately after the fire started" and a "triangle mark" was obtained for "after a sufficient amount of time had passed," and for "black smoke," a "circle mark" was obtained for "visibly thin smoke" and a "triangle mark" was obtained for "visibly thick smoke."

[0102] From the above detection results, it can be said that the smoke detection device trained with the first teacher image 2600 has excellent detection accuracy for "white smoke on a white background", the smoke detection device trained with the second teacher image 2800 also has excellent detection accuracy for "black smoke", and the smoke detection device trained with both the first teacher image 2600 and the second teacher image 2800 has excellent detection accuracy for both "white smoke on a white background" and "black smoke".

[0103] In addition, higher detection accuracy was obtained when the gamma value used in gamma correction to generate the first teacher image 2600 was set to a predetermined value of 2 or more and 3 or less, and the gamma value used in gamma correction to generate the second teacher image 2800 was set to a predetermined value of 0.5 or more and less than 1.

[0104] [e. Generation of teacher images] Next, the process of generating a teacher image by the control unit of the image generating device will be described with reference to Fig. 6, which is a flow chart showing the image generating process by the control unit 12 of the image generating device 10 in Fig. 1.

[0105] 6, the control unit 12 of the image generating device 10 starts operation in response to the start-up operation of the device, and first performs a predetermined initial setting for the image generating process (step S1). The initial setting is arbitrary, but for example, a virtual space for generating a teacher image is formed in a memory, and an image of the virtual space is displayed on the display unit 14.

[0106] Next, a teacher original image is generated by a predetermined operation (step S2). This teacher original image is generated by selecting one of a plurality of surveillance images stored in advance in a memory or the like and selecting one of a plurality of smoke images stored in advance in a memory or the like, arranging the selected surveillance image as a background image in the virtual space, and arranging the selected smoke image at a predetermined location on the surveillance image as the fire source position, generating a teacher original image and storing it in a memory or the like.

[0107] The teacher original image is converted to an image format used for detection in the smoke detection device and stored. For example, if the image format used for detection in the smoke detection device is a grayscale image, the teacher original image is converted to a grayscale image and stored. If the image format used for detection in the smoke detection device and the teacher original image are the same (for example, both are RGB format), image format conversion processing is not required.

[0108] Next, gamma correction is performed on the original teacher image stored in a memory or the like, with a gamma value greater than 1, according to the above (Equation 1), to generate a first teacher image 2600 in which the contrast of bright areas is suppressed and the contrast of dark areas is emphasized, and the first teacher image 2600 is stored in the teacher image memory unit 20 (step S3).

[0109] Next, gamma correction is performed on the same original teacher image stored in a memory or the like, with the gamma value set to a value less than 1 according to the above (Equation 1), to generate a second teacher image 2800 in which the contrast of dark areas is suppressed and the contrast of bright areas is emphasized, and the second teacher image 2800 is stored in the teacher image memory unit 20 (step S4).

[0110] Next, if the specified ending operation is not identified (No in step S5), the process returns to the generation of the original teacher image (step S2), and, for example, the selected surveillance image is left as is, and the already placed smoke image is moved to another fire source position, or the color of the smoke image is gradually changed from white smoke to black smoke, to generate a new original teacher image, and then gamma correction is similarly performed on the original teacher image to generate new first teacher image 2600 and second teacher image 2800 and store them in the teacher image memory unit 20 (steps S3 and S4), repeating this process until the specified ending operation is identified and the process is terminated (Yes in step S5).

[0111] [f. First embodiment of smoke detection device] Next, a first embodiment of a smoke detection device equipped with an image recognition model trained using a teacher image generated by an image generation device will be described. In this description, reference will be made to Fig. 7 showing a first embodiment of a smoke detection device equipped with an image recognition model of "YOLO" trained using a first teacher image and a second teacher image generated by the image generation device of Fig. 1, and Fig. 8 showing an example of a display screen displaying the output result of the image recognition model.

[0112] 7, a smoke detection device 40 (40-1) of the first embodiment is composed of an image input unit 44, an image recognition model 38, and a fire judgment unit 46. Surveillance images captured by a surveillance camera 42 installed in a monitored area are input to the image input unit 44. As described above, the surveillance camera 42 transmits to the image input unit 44 a video consisting of a succession of color RGB frame images captured in sequence of the monitored area at a frame rate of, for example, 30 frames per second or 60 frames per second.

[0113] The image input unit 44 acquires image data of RGB frame images of the monitoring area at 30 frames per second or 60 frames per second from the monitoring camera 42 and stores the image data in a frame memory. The size of the moving images captured by the monitoring camera 42 is arbitrary, but may be, for example, 1280 x 720 pixels. The RGB frame images stored in the frame memory by the image input unit 44 are converted, for example, into grayscale images so as to be input to the image recognition model 38 for image recognition.

[0114] The image recognition model 38 using "YOLO" is subjected to machine learning by the learning control device 36 shown in FIG. 4 using both a large number of first teacher images 2600 and a large number of second teacher images 2800 generated by the image generation device 10 and stored in the teacher image storage unit 20. The image recognition model 38 receives a surveillance image (frame image) thinned out at a predetermined interval from the image input unit 44, and outputs a smoke detection result if a smoke image is present in the surveillance image. The thinning interval of the surveillance image corresponds to the processing time per image of the image recognition model 38.

[0115] The smoke detection result output by the image recognition model 38 includes, for example, a detection frame including the detected smoke image and a reliability score, with the reliability score being closer to 1, indicating a higher reliability. In addition, the detection frame and reliability score output by the image recognition model 38 can be displayed together with a surveillance image displayed on a display unit such as a display provided in the smoke detection device 40 (40-1) in which the image recognition model 38 is implemented or a display connected to the smoke detection device.

[0116] Figure 8 is an example of a display screen showing the output results of the image recognition model 38, in which detection and estimation is performed by the image recognition model 38 on a smoke image 62 present in a surveillance image 60, and a detection frame 64 surrounding the smoke image 62 is displayed. In addition, a confidence score 66 is displayed at the top of the detection frame 64 as "KEMURI 0.80" together with the class name.

[0117] The fire determination unit 46 determines that there is a fire when the reliability score indicating the smoke detection result output by the image recognition model 38 satisfies a predetermined fire determination condition, and transmits a fire detection signal to an external device such as a receiver of the fire alarm system to output a fire alarm. The fire determination condition of the fire determination unit 46 is arbitrary, but for example, a fire is determined when the reliability score output by the image recognition model 388 as the smoke detection result exceeds a predetermined threshold (e.g., 0.5) or exceeds the predetermined threshold for a predetermined accumulation time.

[0118] [g. Second embodiment of the smoke detection device] Next, a second embodiment of a smoke detection device including an image recognition model trained using a teacher image generated by an image generation device will be described. In this description, reference will be made to Fig. 9 showing the second embodiment of the smoke detection device including a plurality of image recognition models trained using different teacher images.

[0119] As shown in Figure 9, the smoke detection device 40 (40-2) of the second embodiment is composed of an image distribution unit 45, a first image recognition model 50, a second image recognition model 52, a third image recognition model 54, a fourth image recognition model 56, and a fire judgment unit 58.

[0120] The image distribution unit 45 is obtained by adding to the image input unit 44 of the first embodiment shown in FIG. 7 a function of inputting the same surveillance image (frame image) sequentially or in parallel to the first image recognition model 50, the second image recognition model 52, the third image recognition model 54, and the fourth image recognition model 56.

[0121] The first image recognition model 50 is the same as the image recognition model 38 of the first embodiment shown in Figure 7, and machine learning is performed using both the first teacher image 2600 and the second teacher image 2800 generated by the image generating device 10.

[0122] In addition, the second image recognition model 52 is trained by machine learning using a first teacher image 2600 generated by the image generating device 10, the third image recognition model 54 is trained by machine learning using a second teacher image 2800 generated by the image generating device 10, and the fourth image recognition model 56 is trained by machine learning using a teacher image that has not been gamma corrected, i.e., an original teacher image, generated by the image generating device 10.

[0123] The fire determination unit 58 determines that a fire has occurred if the four reliability scores CS1 to CS4 output when the same surveillance image is input to each of the first image recognition model 50, the second image recognition model 52, the third image recognition model 54, and the fourth image recognition model 56 satisfy a predetermined fire determination condition. The fire determination condition for this purpose is arbitrary, but for example, the average value or sum of the four reliability scores CS1 to CS4 is calculated, and a fire is determined if it exceeds a predetermined threshold value or exceeds the predetermined threshold value for a predetermined accumulation time.

[0124] Here, the first image recognition model 50, which has been trained using both the first teacher image 2600 and the second teacher image 2800, is capable of accurately detecting smoke in a surveillance image of "white smoke on a white background" and a surveillance image of "black smoke", as shown in FIG. 5, and therefore obtains a high confidence score CS1 when a surveillance image of "white smoke on a white background" or a surveillance image of "black smoke" is input.

[0125] In addition, the second image recognition model 52, which has been trained using the first teacher image, is able to accurately detect smoke in a surveillance image of "white smoke against a white background", as shown in FIG. 5, and therefore obtains a high confidence score CS2 when a surveillance image of "white smoke against a white background" is input, and the third image recognition model 54, which has been trained using the second teacher image, is able to accurately detect smoke in a surveillance image of "black smoke", as shown in FIG. 5, and therefore obtains a high confidence score CS3 when a surveillance image of "black smoke" is input.

[0126] In addition, the fourth image recognition model 56, which is trained using an original teacher image that has not been gamma corrected, obtains a high confidence score CS4 when a surveillance image containing smoke is input, for which the detection accuracy is insufficient when training using a teacher image that has been gamma corrected.

[0127] For this reason, smoke images are recognized by each of the first image recognition model 50, second image recognition model 52, and third image recognition model 54, which were generated based on the same original teacher image but were trained with teacher images in which the parts where contrast is suppressed or emphasized are different, and the fourth image recognition model 56, which was trained with an original teacher image in which contrast is not suppressed or emphasized, and a comprehensive fire judgment is made based on each smoke detection result (confidence score), thereby making it possible to achieve higher fire detection accuracy.

[0128] [h. Third embodiment of smoke detection device] Next, a third embodiment of a smoke detection device equipped with an image recognition model that has been trained using a teacher image generated by an image generation device will be described. In this description, reference will be made to Fig. 10 showing the third embodiment of a smoke detection device equipped with a plurality of image recognition models that divide a surveillance image into regions according to brightness and recognize a smoke image, and Fig. 11 showing a high-luminance region divided image and a low-luminance region divided image divided from a surveillance image. Fig. 11(A) shows the surveillance image before division, Fig. 11(B) shows the divided high-luminance region divided image, and Fig. 11(C) shows the divided low-luminance region divided image.

[0129] As shown in FIG. 10, the third embodiment of the smoke detection device 40 (40-3) is composed of an image segmentation unit 48, a first image recognition model 50, a second image recognition model 52, a third image recognition model 54, a fourth image recognition model 56, and a fire judgment unit 58, and is similar to the second embodiment shown in FIG. 9 except for the image segmentation unit 48.

[0130] The image division unit 48 divides the surveillance image (frame image) received from the surveillance camera 42 into a plurality of brightness regions having different brightness ranges, and inputs the divided images corresponding to the respective brightness regions sequentially or in parallel to the first image recognition model 50, the second image recognition model 52, the third image recognition model 54, and the fourth image recognition model 56 to recognize the smoke image. The image division by the image division unit 48 according to the brightness in the surveillance image is arbitrary, but for example, the image is divided into a high brightness region divided image in which bright regions in the surveillance image are extracted, and a low brightness region divided image in which dark regions are extracted.

[0131] 11(A) shows a surveillance image 70 of a surveillance area captured by a surveillance camera 10. In the surveillance image 70, the area including the parts shelf and desk located in the center is bright, and the outer area surrounding it is dark. For this reason, the image division unit 48 divides the surveillance image 70 into two brightness areas, a high brightness area and a low brightness area, and the division of the brightness areas is performed based on, for example, a "binarization method." This "binarization method" is arbitrary, but a well-known method known as the "Otsu's method" is used, for example.

[0132] In the binarization by "Otsu's method," a threshold value Pth for optimal binarization is determined based on a histogram of pixel values ​​in an image.

[0133] In addition, in "Otsu's method," the pixel values ​​of an image are divided into two classes, and the binarization threshold Pth is determined by minimizing the variance within the class, and the process is carried out in the following manner.

[0134] (1) The histogram of the input image is obtained. (2) The minimum, maximum, and average pixel values ​​are calculated from the histogram. (3) Select a threshold value within the range between the minimum and maximum values. (4) Divide the histogram into two classes, a first class and a second class, using a selected threshold. (5) Calculate the variance, mean, and number of pixels for the first class. (6) Calculate the variance, mean, and number of pixels for the second class. (7) Calculate the within-class variance and between-class variance. (8) Calculate the degree of separation from the within-class variance and the between-class variance. (9) The next new threshold is selected and the steps (4) to (8) are repeated until the degree of separation is calculated. (10) The threshold value that results in the maximum degree of separation is determined as the binarization threshold value Pth.

[0135] Fig. 11(B) shows a high-brightness region divided image 72 generated by dividing pixels equal to or greater than the binarization threshold Pth in the monitoring image 70 in Fig. 11(A) into high-brightness regions 7210. In the high-brightness region divided image 72, pixels having pixel values ​​equal to or greater than the binarization threshold Pth in the monitoring image 70 in Fig. 11(A) are treated as valid pixels to generate the high-brightness region 7210, and pixels having pixel values ​​less than the binarization threshold Pth are treated as invalid pixels, for example with a minimum pixel value of 0 (black pixels), to generate a pixel invalid region 7220 where no monitoring image exists.

[0136] Furthermore, with regard to the merging of the boundary portions between the high-brightness areas 7210 and the invalid pixel areas 7220 and the small other areas (hole areas) that occur within each area, well-known morphological processing or the like may be used to merge the areas and fill in the holes to reduce and remove image components that act as noise.

[0137] Fig. 11(C) shows a low brightness region divided image 74 generated by dividing the pixels in the monitoring image 70 in Fig. 11(A) that are less than the binarization threshold Pth into low brightness regions 7410. In the low brightness region divided image 74, the pixels in the monitoring image 70 in Fig. 11(A) that have pixel values ​​less than the binarization threshold Pth are treated as valid pixels to generate the low brightness region 7410, and the pixels that have pixel values ​​equal to or greater than the binarization threshold Pth are treated as invalid pixels, for example with a minimum pixel value of 0 (black pixels), to generate a pixel invalid region 7420 in which no monitoring image exists.

[0138] Furthermore, with regard to the merging of the boundary portions between the low-brightness areas 7410 and the invalid pixel areas 7420 and the small other areas (hole areas) that occur within each area, the areas may be merged or holes may be filled using known morphological processing or the like to reduce or remove image components that act as noise.

[0139] The high-brightness region divided image 72 generated by the image dividing unit 48 is input to each of the first image recognition model 50, the second image recognition model 52, the third image recognition model 54, and the fourth image recognition model 56, and image recognition (smoke detection) is performed on the image present in the high-brightness region 7210 which is a valid pixel, and the invalid pixel region 7220 is excluded from the target of image recognition, making it possible to detect smoke images with high accuracy in the high-brightness region 7210, and the accuracy of fire judgment is improved by a comprehensive judgment by the fire judgment unit 58.

[0140] In addition, the low-brightness area divided image 74 generated by the image dividing unit 48 is input to each of the first image recognition model 50, the second image recognition model 52, the third image recognition model 54, and the fourth image recognition model 56, and image recognition (smoke detection) is performed on the image present in the low-brightness area 7410 which is a valid pixel, and the invalid pixel area 7420 is excluded from the target of image recognition, making it possible to detect smoke images with high accuracy in the low-brightness area 7410, and the accuracy of fire judgment is improved by a comprehensive judgment by the fire judgment unit 58.

[0141] In FIG. 11, the monitoring image 70 is divided into a high-brightness region divided image 72 and a low-brightness region divided image 74. However, if the histogram of pixel values ​​of the high-brightness region 7210 indicates a distribution that can be further divided, a divided image may be generated in which the high-brightness region 7210 (high-brightness region divided image 72) is further divided into a high-brightness region and a medium-brightness region by “Otsu's method”.

[0142] Furthermore, the high-brightness region divided image 72 and the low-brightness region divided image 74 divided and generated by the image dividing unit 48 do not necessarily need to be input to all image recognition models. For example, the high-brightness region divided image 72 may be input to each of the first image recognition model 50, the second image recognition model 52, and the fourth image recognition model 56, and the low-brightness region divided image 74 may be input to each of the first image recognition model 50, the third image recognition model 54, and the fourth image recognition model 56.

[0143] The reason why the high-brightness region segmentation image 72 and the low-brightness region segmentation image 74 are not input to the second image recognition model 52 is that machine learning is performed on the second image recognition model 52 using the first teacher image 2600, and as shown in FIG. 5, while it has high smoke detection accuracy for a surveillance image of "white smoke with a white background", it does not have good detection accuracy for "black smoke". The surveillance image of "white smoke with a white background", for which the second image recognition model 52 has high smoke detection accuracy, corresponds to the high-brightness region segmentation image 72, while "black smoke", for which it does not have good detection accuracy, corresponds to the low-brightness region segmentation image 74.

[0144] Furthermore, the reason why the low-brightness region segmentation image 74 and not the high-brightness region segmentation image 72 are input to the third image recognition model 54 is that machine learning is performed on the third image recognition model 54 using the second teacher image 2800, and as shown in FIG. 5, although it does not have good detection accuracy for a surveillance image of "white smoke with a white background", it has high smoke detection accuracy for "black smoke", and the surveillance image of "black smoke", for which the third image recognition model 54 has high smoke detection accuracy, corresponds to the low-brightness region segmentation image 74, while "white smoke with a white background", for which it does not have good detection accuracy, corresponds to the high-brightness region segmentation image 72.

[0145] [i. Modifications of the present invention] Modifications of the image generating system and smoke detecting device according to the present invention will now be described In addition to the above-described embodiment, the image generating system and smoke detecting device according to the present invention include the following modifications.

[0146] (Image recognition model) The above embodiment takes an image recognition model using the "YOLO" algorithm as an example, but any image recognition model such as a support vector machine (SVM) or random forest may be used. In addition, it may be applied to a recognition model in which feature quantities such as smoke flow and time changes in pixel values ​​are calculated from multiple temporally consecutive frames, and the feature quantities are learned by an SVM.

[0147] (Generating teacher images that suppress contrast in medium brightness areas) In addition, in the above embodiment, a teacher image that suppresses the contrast of a bright part or a dark part of the teacher original image is generated and machine learning is performed, but a third image generation unit that generates a third teacher image by a third process that suppresses the contrast of a medium brightness part between a bright part and a dark part of the teacher original image may be provided. The relationship between the input pixel value x and the output pixel value y of the gamma correction by the third process may be an S-curve or an inverse S-curve with the linear characteristic with the gamma value of 1.0 shown in FIG. 3 as the center line.

[0148] By generating a third teacher image through a third process that suppresses the contrast in such medium-brightness areas and performing machine learning on the learning recognition model, it is possible to improve the accuracy of smoke detection even in surveillance images with low contrast, such as gray or thin smoke against a medium-brightness background.

[0149] (Processing that only suppresses contrast, Processing that only enhances contrast) In addition, in the above embodiment, as the most desirable form, the first process suppresses the contrast of the bright parts and emphasizes the contrast of the dark parts, and the second process suppresses the contrast of the dark parts and emphasizes the contrast of the bright parts, but the embodiment is not limited to this. The first process may suppress the contrast of the bright parts, and the second process may suppress the contrast of the dark parts. In this case, it is possible to improve the detection accuracy of various smokes by improving the detection accuracy of smoke at a stage close to the beginning of the generation of smoke that is similar to the background color in a situation where smoke has just started to be generated.

[0150] Alternatively, the first process may enhance the contrast of dark areas, and the second process may enhance the contrast of bright areas, improving the accuracy of detecting thin smoke against a background with a clear pattern, a condition for which conventional smoke detectors have poor detection accuracy.

[0151] (Surveillance camera) In the above embodiment, a surveillance camera that outputs color video was used, so the color frame images were converted to grayscale images in the image input unit 44. However, if a surveillance camera that outputs monochrome video is used, grayscale conversion in the image input unit is not necessary.

[0152] (others) Furthermore, the present invention is not limited to the above-described embodiment, but includes appropriate modifications that do not impair the objects and advantages of the present invention, and is not limited to the numerical values ​​shown in the above-described embodiment. [Explanation of symbols]

[0153] 10: Image generation device 12: Control unit 14:Operation unit 16: Display section 18: Communications Department 20: Image storage section for teachers 22: Teacher's original image generation unit 26: First image generating unit 2600: First teacher image 28: Second image generating unit 2800: Second teacher image 30: Original image for teacher 32,60,70: Surveillance image 34,62: Smoke image 36: Learning control device 38: Image recognition model 40(40-1)~40(40-3): Smoke detection device 42: Surveillance camera 44: Image input unit 45: Image distribution section 46,58:Fire judgment department 48: Image division section 50: First image recognition model 52: Second image recognition model 54: Third image recognition model 56: 4th image recognition model 64: Detection frame 66: Confidence score 72: High-brightness region segmentation image 7210: High brightness area 7220, 7420: Pixel invalid area 74: Low brightness segmented image 7410: Low brightness area

Claims

1. An image generation system for generating training images to be used in machine learning of a smoke detection device for detecting a fire in a monitored area, comprising: a first image generating unit that generates a first teacher image by a first process that suppresses contrast in a bright part and emphasizes contrast in a dark part of the teacher original image including the monitoring area and the smoke; a second image generating unit that generates a second teacher image by performing a second process on the original teacher image to suppress contrast in a dark portion and emphasize contrast in a bright portion; Equipped with An image generation system characterized by performing machine learning on a smoke detection device using the first teacher image and the second teacher image.

2. 2. The image generating system of claim 1, The image generation system according to the present invention, wherein the first image generation unit performs gamma correction as the first processing by setting a gamma value greater than 1 and raising the brightness of the original teacher image to the power of (1 / γ).

3. 3. The image generating system according to claim 2, The image generating system according to claim 1, wherein the gamma correction in the first process sets the gamma value to a predetermined value of 2 or more and 3 or less.

4. 2. The image generating system of claim 1, The image generation system is characterized in that, as the second processing, the second image generation unit performs gamma correction by raising the brightness of the original teacher image to the power of (1 / γ) with a gamma value less than 1.

5. 5. The image generating system according to claim 4, The image generating system according to claim 1, wherein the gamma correction in the second process sets the gamma value to a predetermined value of 0.5 or more.

6. 2. The image generating system of claim 1, further comprising: An image generation system comprising a third image generation unit that generates a third teacher image by a third process that suppresses the contrast of the intermediate brightness areas of the original teacher image that are between the bright areas and the dark areas.

7. A smoke detection unit that performs machine learning using a plurality of training images generated by the image generation system according to any one of claims 1 to 6, A smoke detection device, comprising: a monitoring image of the monitoring area inputted to the smoke detection unit to detect a fire.

8. 9. The smoke detection device according to claim 8, The smoke detection device, characterized in that the smoke detection unit performs machine learning using the first teacher image and the second teacher image.

9. 9. The smoke detection device according to claim 8, The smoke detection unit is A first smoke detection unit that performs machine learning using the first teacher image and the second teacher image; A second smoke detection unit that performs machine learning using the first teacher image; A third smoke detection unit that performs machine learning using the second teacher image; and Equipped with A smoke detection device characterized in that the surveillance image is input to each of the first smoke detection unit, the second smoke detection unit, and the third smoke detection unit, and smoke is detected based on the detection results output.

10. 10. The smoke detection device according to claim 9, The smoke detection unit divides the surveillance image into a plurality of brightness areas having different predetermined brightness ranges, and inputs divided images corresponding to each of the brightness areas into the first smoke detection unit, the second smoke detection unit, and the third smoke detection unit, and detects smoke based on the detection results outputted.

11. 11. The smoke detection device according to claim 10, The smoke detection unit is Dividing the brightness area of ​​the monitoring image into a high brightness area and a low brightness area; A divided image corresponding to the low brightness area is input to the first smoke detection unit and the third smoke detection unit; A smoke detection device comprising: a first smoke detection unit and a second smoke detection unit, the first smoke detection unit receiving a divided image corresponding to the high luminance region;

12. 10. The smoke detection device according to claim 9, The smoke detection unit further includes: a fourth smoke detection unit that performs machine learning using the original teacher image as the teacher image; A smoke detection device characterized in that the surveillance image is input to each of the first smoke detection unit, the second smoke detection unit, the third smoke detection unit and the fourth smoke detection unit, and smoke is detected based on the detection results output.

Images