Method for detecting noise in images of a surrounding area, and method for training a machine learning algorithm

Abstract

The invention relates to a method for detecting noise in images of a surrounding area, having the steps of: providing (220) a plurality of grayscale images (221, 222, 223) of the surrounding area; removing (230) the background of the surrounding area from the plurality of grayscale images in order to obtain a plurality of noise images (241, 242, 243); and determining (240) a region (251) with noise in the surrounding area on the basis of the plurality of noise images.

Description

[0001] Description

[0002] title

[0003] Method for detecting smoke in images of an environment and method for training a machine learning algorithm

[0004] The present invention relates to a method for detecting smoke in images of an environment and a method for training a machine learning algorithm as well as a computing unit and a computer program for carrying out the same.

[0005] Background of the invention

[0006] Detecting fire or smoke in an environment is often a safety-relevant task. Video-based systems that detect smoke in images of the environment can be used for this purpose. For example, DE 10 2014219 829 A1 describes a method for detecting smoke based on a sequence of images of an environment.

[0007] Disclosure of the invention

[0008] According to the invention, a method for detecting smoke in images of an environment and a method for training a machine learning algorithm, as well as a computing unit and a computer program for implementing them, are proposed, having the features of the independent patent claims. Advantageous embodiments are the subject of the dependent claims and the following description.

[0009] Both single-frame-based methods and methods with temporal processing can be used to detect smoke in video data or images (contained in video data). Temporal information reflected in the images has proven to be advantageous in this case. One option for detecting smoke can, for example, be based largely on motion estimation using optical flow. Sufficient smoke density can be important for successful motion estimation. The continuous movement of smoke can therefore be used, for example, as a key feature in the detection of smoke in images. Motion estimation is usually carried out by calculating optical flow. This then detects the typically upward movement of smoke. However, in order to detect movement using optical flow, the smoke must be sufficiently dense.

[0010] Further verification of potential smoke candidates, i.e., areas in images that potentially show smoke, can be achieved using color features. Here, too, certain requirements must generally be met for reliable detection.

[0011] Smoke can be understood, in particular, as a mixture of a solid phase and a gaseous phase. Smoke can also be understood as an aerosol produced by combustion in a finely dispersed form consisting of particles such as dust or ash particles and / or liquid droplets (water, oil vapors, acid vapors, liquid combustion residues), particularly those contained in exhaust gases.

[0012] As has been shown, preprocessing of data (i.e. the underlying images) can be an important step in the development of a machine learning-based algorithm. The aim here is to retain information that is important for classification and to remove information that potentially has less characteristic features if necessary. The second aspect is particularly relevant when the data set is small, as otherwise so-called overfitting often occurs. This is the term used to describe a phenomenon in which an algorithm makes decisions based on features in the training data that are not general features of the class to be recognized. A method for preprocessing data presented within the scope of the present invention fulfills precisely this purpose by reducing the appearance of smoke in video sections (i.e. images) to characteristic information.

[0013] As mentioned, previous investigations of detection algorithms have mostly been limited to clearly visible smoke. In a typical application, the visibility of smoke is limited by the density of the smoke and various environmental influences, such as lighting. In addition, uniformly moving objects such as conveyor belts, cranes, or lifting platforms can lead to false alarms if they meet certain color criteria. Furthermore, color information in the area of ​​thin smoke is usually overlaid with color information from the background. This specific property of smoke significantly complicates successful detection based on color characteristics.

[0014] Previous algorithms for visual smoke detection are therefore only applicable with significant limitations. One goal of the present invention, however, is to significantly reduce the minimum requirements for successful detection and significantly increase reliability. To this end, pattern recognition of moving smoke edges is used.

[0015] The present invention provides a way to visualize moving smoke edges using simple mathematical operations and use them to detect smoke. These can then be used as input for machine learning algorithms, particularly artificial neural networks. Compared to the use of individual images, these have proven to be significantly more suitable and, in experiments, have delivered significantly better results on previously unseen data.

[0016] Convolutional neural networks, in particular, have been shown to be one of the most promising approaches in automated image processing. Their functionality is based on the recognition of learned patterns. While clearly visible, dense smoke exhibits typical patterns even in individual images, thin, poorly lit smoke is difficult or almost impossible to recognize in individual images. However, it can typically still be recognized based on its characteristic movement. The patterns to be recognized in the area of ​​very thin smoke are therefore typically primarily those of the background.

[0017] Within the scope of the present invention, it is proposed that a plurality of grayscale images of the surroundings, i.e. the surroundings which are, for example, being monitored and in which potentially present smoke is to be detected, are provided. Typically, images (e.g. from a video) are obtained in color, so that, for example, a plurality of images, in particular color images, of the surroundings are initially provided, from which the plurality of grayscale images can then be obtained by conversion, in particular by reducing the color information in the plurality of images. If grayscale images are already provided directly, e.g. from a camera or surveillance camera, the conversion step can be omitted. Ultimately, stacked grayscale images or grayscale images are then available. The grayscale images can, for example, be those with 256 gray levels or.Grayscale images can be used, but in principle other types of grayscale images can also be used, for example with more or fewer grayscale levels.

[0018] A background of the surroundings is then removed from the multiple grayscale images, resulting in multiple smoke images (a smoke image is thus created by removing the background from a grayscale image). Various methods can be used to extract or remove the background model. A continuously updated background model would be conceivable, for example, but simple subtraction of a reference image, obtained, for example, by averaging the multiple images or grayscale images, is also conceivable. For this purpose, reference is also made to DE 10 2014 219 829 A1.

[0019] Based on the multiple smoke images, an area with smoke in the surrounding area is then determined. As already mentioned, this is particularly usefully done using a machine learning algorithm. The machine learning algorithm receives the smoke images as input values ​​and outputs the area with smoke as the output value. As also mentioned, the machine learning algorithm comprises, in particular, an artificial neural network, more particularly a convolutional neural network, which is particularly suitable for image analysis.

[0020] Preferably, determining the area with smoke in the environment based on the plurality of smoke images comprises first determining edges of smoke in the smoke images. Based on the edges of the smoke, the area with smoke in the environment is then determined. As has been shown, the images always exhibit similar characteristic patterns for the smoke. These patterns correspond to the moving edges of the smoke.

[0021] Furthermore, this form of representation with the grayscale images, which can be obtained in particular by means of special preprocessing, as mentioned, has proven to be a particularly suitable input format for a deep learning-based algorithm (machine learning algorithm for smoke detection). Emerging patterns of rising, spreading, and sometimes turbulent smoke movement have proven particularly characteristic for distinguishing smoke from other moving objects.

[0022] To train such a machine learning algorithm or artificial neural network, training data can be provided which comprises a plurality of smoke images, each of one or more different environments, wherein the plurality of smoke images have each been obtained from a plurality of grayscale images of the environment after removal of a background. As mentioned, the aforementioned preprocessing (conversion) may again be necessary here. The training data can in particular also comprise annotated smoke images or images, i.e. images for which it is noted whether or not smoke is present. The machine learning algorithm is then adapted based on the training data such that the machine learning algorithm determines an area with smoke in the respective environment based on the respective plurality of smoke images. This is therefore in particular a "supervised learning" approach.The training thus refers to a process consisting of a feed forward step and a backpropagation step. A computing unit according to the invention, e.g., a computer or a control unit of a surveillance camera, is configured, particularly in terms of programming, to carry out a method according to the invention.

[0023] The implementation of a method according to the invention in the form of a computer program or computer program product with program code for carrying out all method steps is also advantageous, since this entails particularly low costs, in particular if an executing control unit is also used for other tasks and is therefore already present. Finally, a machine-readable storage medium is provided with a computer program stored thereon, as described above. Suitable storage media or data carriers for providing the computer program are, in particular, magnetic, optical, and electrical memories, such as hard disks, flash memories, EEPROMs, DVDs, and others. Downloading a program via computer networks (Internet, intranet, etc.) is also possible. Such a download can be wired or cable-based or wireless (e.g., via a WLAN network, a 3G, 4G, 5G, or 6G connection, etc.).

[0024] Further advantages and embodiments of the invention will become apparent from the description and the accompanying drawings.

[0025] The invention is illustrated schematically in the drawing using an embodiment and is described below with reference to the drawing.

[0026] Short description of the drawings

[0027] Figure 1 schematically shows an environment in which a method according to the invention can be carried out.

[0028] Figure 2 schematically shows a sequence of a method according to the invention in a preferred embodiment. Figure 3 schematically shows a sequence of a method according to the invention in another preferred embodiment.

[0029] Embodiment(s) of the invention

[0030] Figure 1 schematically shows an environment 100 in which a method according to the invention can be carried out. Two objects 110 and 112 are shown in the environment, as well as smoke 114, which may develop in the environment 100, for example, due to a fire.

[0031] Furthermore, a surveillance camera 120 with a control or computing unit 122 is shown by way of example, by means of which the environment 100 can be monitored. Also shown by way of example is a remote server or computer 130 (this can represent, for example, a so-called cloud), which can be connected to the surveillance camera 120 or the control or computing unit 122 therein for data transmission, e.g., wirelessly (as indicated) or otherwise.

[0032] While monitoring the surrounding area, the surveillance camera 120 generates video data, which in turn also includes individual images captured sequentially. By analyzing these images, potential smoke in the surrounding area, as indicated by 114, is to be detected.

[0033] Figure 2 schematically illustrates a preferred embodiment of a method according to the invention. Such a method can run entirely or partially, e.g., in the control or processing unit 122 of the surveillance camera 120 or in the server or computer 130, or split between both.

[0034] In a step 200, first, a plurality of images 201, 202, 203, in particular color images, of the surroundings are provided. These may have been captured by the surveillance camera; in particular, these images may also have been extracted from video data of the surveillance camera. Although three images 201, 202, 203 are shown here as an example, more than three images or possibly just two images may be used. In a step 210, these plurality of images 201, 202, 203 are converted into a plurality of (corresponding) grayscale images 221, 222, 223; this may be done, for example, by reducing color information in the plurality of images 201, 202, 203. These plurality of grayscale images 221, 222, 223 are then provided in step 220.

[0035] In a step 230, a background 231 is then removed from each of the multiple grayscale images 221, 222, 223. Such a background (or a background image) can be obtained, for example, from the multiple images 201, 202, 203 or from the multiple grayscale images 221, 222, 223, for example by averaging or the like. By removing the background, a plurality of smoke images 241, 242, 243 are obtained from the multiple grayscale images 221, 222, 223. These smoke images 241, 242, 243 then only contain or can be seen the smoke, since this does not belong to the background and therefore nothing has been removed.

[0036] In a step 250, an area 251 with smoke in the surroundings is then determined based on the smoke images 241, 242, 243. For this purpose, for example, edges of smoke in the smoke images 241, 242, 243 can be detected using pattern recognition and tracked across the smoke images. An edge is designated 245 in smoke image 243 as an example. In this way, the area 251 with smoke can be determined based on the edges of smoke. If an area with smoke has been determined - and smoke has thus been detected - an alarm can be triggered, for example, or another measure can be initiated.

[0037] As mentioned, it is particularly preferred to use a machine learning algorithm for determining the area with smoke in the environment, particularly preferably a convolutional neural network. Such a machine learning algorithm is shown schematically in Figure 2 as an example and is designated 248. The machine learning algorithm 248 receives the smoke images 241, 242, 243 as input values, recognizes the area with smoke based thereon, and outputs this area 251 as the output value. Within the scope of the present invention, it is therefore possible to highlight smoke edge patterns, in particular with the aid of special preprocessing - the conversion to grayscale images. As input for the conversion, for example, several images from a stationary video or surveillance camera taken at a time interval are used. Then, if necessary, the color information of the input images is reduced. This is followed by, for example,Simple background subtraction. It was particularly recognized that a targeted reduction of input information (only grayscale images instead of color images) enables improved detection results for neural networks on unseen data.

[0038] While based on RGB images (i.e. color images), for example, a detection rate of approximately 50% with a false detection rate of approximately 25% can be achieved, with the proposed grayscale images a detection rate of approximately 75% with a false detection rate of only approximately 2% can be achieved.

[0039] Figure 3 schematically shows a sequence of a method according to the invention in a further preferred embodiment, namely a training of a machine learning algorithm, as it can be used - in particular after training - to detect smoke in images of an environment, as was explained in more detail with reference to Figure 2, for example.

[0040] For this purpose, training data 302 is first provided in step 300. The training data 302 comprises a plurality of smoke images, each of one or more different environments (i.e., if there are multiple environments, a plurality of smoke images should be used for each environment). The plurality of smoke images, in turn, are each obtained from a plurality of grayscale images of the environment after removing a background, i.e., preprocessed, for example, as explained above.

[0041] These grayscale images can, for example, be of the type explained and shown with reference to Figure 2; however, it is understood that a significantly larger number of such grayscale images, particularly of different environments, is expedient for training. Furthermore, the training data 302 can include appropriately annotated images.

[0042] In a step 310, a machine learning algorithm 312 is then adapted based on the training data 302. This occurs in such a way that the machine learning algorithm 312 determines an area with smoke in the respective environment based on the respective multiple smoke images. For example, weights or so-called neurons (or general parameters) of the machine learning algorithm are adjusted, i.e., changed if necessary. In this way, a trained or machine learning algorithm 322 is obtained, which can be made available for further use in a step 320.

[0043] The training of the machine learning algorithm can be carried out, for example, on a suitable computer, e.g. on the server or computer 130 as shown in Figure 1.

Claims

Claims 1. A method for detecting smoke in images of an environment, comprising: Providing (220) a plurality of grayscale images (221, 222, 223) of the environment; Removing (230) a background of the environment from the plurality of grayscale images to obtain a plurality of smoke images (241, 242, 243); and determining (240) an area (251) with smoke (114) in the environment based on the plurality of smoke images.

2. The method of claim 1, further comprising: Providing (200) several images (201, 202, 203), in particular color images, of the environment; and Converting (210) the plurality of images into the plurality of grayscale images, in particular by reducing color information in the plurality of images to provide the plurality of grayscale images.

3. The method of claim 1 or 2, wherein determining (240) the area with smoke in the environment based on the plurality of smoke images is performed using a machine learning algorithm (248), wherein the machine learning algorithm receives the smoke images as input values ​​and outputs the area with smoke as output value.

4. The method according to claim 3, wherein the machine learning algorithm (248) comprises an artificial neural network, in particular a convolutional neural network.

5. The method according to one of claims 3 or 4, wherein the machine learning algorithm (248) is provided with training data (302) comprising a plurality of smoke images each of one or more different environments, wherein the plurality of smoke images each consist of a plurality of grayscale images of the environment obtained after removing a background, trained and adapted to it.

6. The method according to any one of the preceding claims, wherein determining the area with smoke in the environment based on the plurality of smoke images comprises: Determining edges (245) of smoke in the smoke images; and Determine the area of ​​smoke in the environment based on the edges of smoke.

7. Method according to one of the preceding claims, wherein the plurality of grayscale images of the environment correspond to images captured at different times, in particular according to a predetermined time schedule.

8. A method for training a machine learning algorithm (312) for detecting smoke in images of an environment, comprising: Providing (300) training data (302) comprising a plurality of smoke images each of one or more different environments, wherein the plurality of smoke images each have been obtained from a plurality of grayscale images of the environment after removal of a background; Adapting (310) the machine learning algorithm (312) based on the training data (302) such that the machine learning algorithm determines an area with smoke in the respective environment based on the respective plurality of smoke images; and Providing (320) the trained machine learning algorithm (322).

9. A computing unit (122, 130) configured to carry out all method steps of a method according to any one of the preceding claims.

10. A computer program which causes a computing unit (122, 130) to carry out all method steps of a method according to one of claims 1 to 8 when it is executed on the computing unit (122, 130).

11. A machine-readable storage medium having a computer program according to claim 10 stored thereon.

Images