Method for detecting noise and / or a flame in an image of a surrounding area, and method for training a machine learning algorithm

Abstract

The invention relates to a method for detecting noise and / or a flame in an image of a surrounding area using a machine learning algorithm, having the steps of: providing (300) an image (301) of the surrounding area; for each of a plurality of image regions (302, 303) of the image, determining (310) whether noise and / or a flame is detected in the image region using the machine learning algorithm (312), which obtains the image or the image regions as an input; and for each image region, providing (320), as the output of the machine learning algorithm, information (321) as to whether noise and / or a flame was detected therein.

Description

[0001] Description

[0002] title

[0003] Method for detecting smoke and / or flames in an image of an urn and method for training a

[0004] The present invention relates to a method for detecting smoke and / or flames in an image of an environment and a method for training a machine learning algorithm as well as a computing unit and a computer program for carrying out the same.

[0005] Background of the invention

[0006] Detecting fire, smoke, and / or flames in an environment is often a safety-relevant task. Video-based systems can be used for this purpose, for example, to detect smoke and / or flames in images of the environment. DE 102019 207 711 A1, for example, describes a method for detecting smoke based on a sequence of images of an environment.

[0007] Disclosure of the invention

[0008] According to the invention, a method for detecting smoke and / or flames in an image of an environment and a method for training a machine learning algorithm, as well as a computing unit and a computer program for implementing the method, are proposed with the features of the independent patent claims. Advantageous embodiments are the subject of the dependent claims and the following description.

[0009] Automated localization of objects in images can be achieved, for example, using so-called anchors or anchor points. Such anchors typically contain information about the position and extent of the object in the image. This information can be represented, for example, using rectangles around the object to be localized. An alternative to anchor-based methods are so-called anchor-free methods such as FCOS, as described in "Tian, ​​Zhi, et al. "Feos: Fully convolutional one-stage object detection." Proceedings of the IEEE / CVF international conference on computer vision. 2019."

[0010] However, the detection of smoke or flames differs significantly from the detection of other objects due to the non-trivial localization of smoke or flames. As a semi-transparent object, smoke generally lacks clear edges, making it often impossible to determine where in the image the actual smoke is located. Accordingly, exact localization of smoke is rarely possible. Furthermore, smoke cannot always be described using a rectangle, which means that the smoke only covers a small area of ​​the rectangle (e.g., in the anchor-based method mentioned above). This also applies to flames. The detection of smoke and flames is therefore generally difficult.

[0011] Smoke can be understood, in particular, as a mixture of a solid phase and a gaseous phase. Smoke can also be understood as an aerosol produced by combustion in a finely dispersed form consisting of particles such as dust or ash particles and / or liquid droplets (water, oil vapors, acid vapors, liquid combustion residues), particularly those contained in exhaust gases.

[0012] Within the scope of the present invention, a line-by-line (or area-by-area) localization of smoke and / or flames (this can also be referred to as fire detection) is proposed, using a machine learning algorithm such as an artificial neural network, in particular a convolutional neural network. In particular, the machine learning algorithm can be based on deep learning. This is therefore in particular a deep learning-based, video-based video fire detection algorithm. Here, an image of an environment that is to be monitored, for example, is first provided. For example, the image can be provided by a camera or surveillance camera. Such an image can also be part of a video or video sequence.It goes without saying that multiple images can also be provided, for which the procedure described below can then be applied separately - this then allows, for example, continuous monitoring for smoke and / or flames.

[0013] Then, for each of several image regions of the image, it is determined whether smoke and / or flames are detected there, using the machine learning algorithm, which receives the image or image regions as input. Conveniently, the machine learning algorithm receives the image as input and then, when applied, divides the image into several image regions (instead of image regions, cells can also be spoken of). However, it is also conceivable that the image is divided into image regions in advance, so that the image regions are passed directly to the machine learning algorithm as input. The image can, for example, be divided into nx times ny image regions or cells, where nx stands for a number of image regions in the x-direction (e.g. horizontal) and ny for a number of image regions in the y-direction (e.g. vertical).These numbers and also their arrangement (preferably not overlapping but preferably adjacent to each other) can be determined in advance.

[0014] The detection of smoke or flames in individual image areas can then be based on image processing or image analysis, for example, based on motion estimation using optical flow. This then detects, for example, the typically upward-directed movement behavior of smoke.

[0015] The machine learning algorithm then outputs information for each image region, indicating whether smoke and / or flames have been detected within it. This information can then be made available for further use. For example, if smoke or flames are detected, an alarm can be triggered or other measures can be initiated. Preferably, the machine learning algorithm has multiple processing layers and / or is configured to detect smoke and / or flames in the image and, in doing so, divide the image into several separate image regions in which smoke and / or flames have been detected.

[0016] The image is divided into several image regions, and a prediction is made for each of these image regions, indicating whether smoke and / or flames are present in the given image region. The advantages of this method are its simplicity compared to anchor-based methods (no hyperparameters and no so-called "non-maximum suppression" are required), as well as the visually better localization, similar to coarse segmentation.

[0017] Fundamentally, this can be a typical structure of a deep learning application. From an input (e.g., an image, possibly from a video sequence, of a possible fire scenario), a machine learning algorithm or network, preferably with multiple layers in which the input is processed, determines whether smoke or flames are detected. The machine learning algorithm or network includes, for example, a combination of convolutional, activation, normalization, and pooling layers, or even just some of them.

[0018] The last of these multiple layers has, in particular, a dimension or output dimension, which corresponds to a product of a number of horizontal image regions (cf. nx above) of the image, a number of vertical image regions (cf. ny above) of the image, and a number of classes. The number of classes is at least two. These classes are selected, for example, from: smoke detected, flames detected, and neither smoke nor flames detected. In the case of combined smoke and flame detection, the number of classes is, for example, three. In the case of simple smoke detection, however, the number of classes is two. An activation function of this last layer can, for example, be a so-called softmax function, whereby the outputs can be interpreted as probabilities for the respective class. The output can therefore be interpreted as a probability estimate for the presence of smoke or flames in the respective image regions. It can then, for example,If the probability of the presence of, for example, smoke is higher than a certain, predetermined threshold, the presence of smoke can be concluded.

[0019] Training data can be provided to train such a machine learning algorithm or artificial neural network. The training data includes multiple images, each with multiple image regions, as well as multiple labels (so-called annotations) that indicate whether smoke and / or flames are detectable for at least some, preferably all, image regions. The multiple images preferably originate from different environments and / or situations, thus, in particular, also showing different situations related to smoke and / or flames (e.g., with regard to their extent).

[0020] The machine learning algorithm is then adapted based on the training data so that the machine learning algorithm determines information for each image area as to whether smoke and / or flames have been detected in it.

[0021] The multiple labels can be generated from existing labels, each of which indicates whether smoke and / or flames can be detected in an image. This is achieved by transferring information from the existing labels, particularly automatically, to the image regions. For example, existing anchor-based labels can be transferred to the individual cells or image regions using a so-called "intersection-over-union heuristic." For this purpose, a grid can be imagined, for example, with a rectangular bounding box (which is or will be defined by a user, for example). To train the grid-based approach, labels in grid format (instead of a box) are then used. To generate a label in grid format, an overlap with the bounding box can be calculated (or between each cell and the bounding box, "intersection-over-union"). In this way, each cell is assigned a value ("intersection-over-union" with the bounding box).Cells with a certain overlap can then be defined as "positive" (i.e., cells with smoke). Cells with too little overlap are entered into the training as "negative" (i.e., cells without smoke). However, the labels for the image regions can also be generated manually.

[0022] Cross entropy can be used as a loss function for training (especially supervised learning) a machine learning algorithm or network. However, other loss functions are also possible.

[0023] Anchor-based methods differ from the proposed approach primarily in the form of their output. Anchor-based methods typically expect a single box for an object. In the approach described or proposed here, the output potentially includes multiple cells or image regions per object.

[0024] A computing unit according to the invention, e.g. a computer or a control unit of a surveillance camera, is configured, in particular in terms of programming, to carry out a method according to the invention.

[0025] The implementation of a method according to the invention in the form of a computer program or computer program product with program code for carrying out all method steps is also advantageous, since this entails particularly low costs, in particular if an executing control unit is also used for other tasks and is therefore already present. Finally, a machine-readable storage medium is provided with a computer program stored thereon, as described above. Suitable storage media or data carriers for providing the computer program are, in particular, magnetic, optical, and electrical memories, such as hard disks, flash memories, EEPROMs, DVDs, and others. Downloading a program via computer networks (Internet, intranet, etc.) is also possible. Such a download can be wired or cable-based or wireless (e.g., via a WLAN network, a 3G, 4G, 5G, or 6G connection, etc.).Further advantages and embodiments of the invention will become apparent from the description and the accompanying drawings.

[0026] The invention is illustrated schematically in the drawing using an embodiment and is described below with reference to the drawing.

[0027] Short description of the drawings

[0028] Figure 1 schematically shows an environment in which a method according to the invention can be carried out.

[0029] Figure 2 shows schematically a result of a method according to the invention in a preferred embodiment.

[0030] Figure 3 shows schematically a sequence of a method according to the invention in a preferred embodiment.

[0031] Figure 4 shows schematically a sequence of a method according to the invention in a further preferred embodiment.

[0032] Embodiment(s) of the invention

[0033] Figure 1 schematically shows an environment 100 in which a method according to the invention can be carried out. Two objects 110 and 112 are shown in the environment, as well as smoke 114, which can develop, for example, due to a fire in the environment 100.

[0034] Furthermore, a surveillance camera 120 with a control or computing unit 122 is shown by way of example, by means of which the environment 100 can be monitored. Also shown by way of example is a remote server or computer 130 (this can represent, for example, a so-called cloud), which can be connected to the surveillance camera 120 or the control or computing unit 122 therein for data transmission, e.g., wirelessly (as indicated) or otherwise.

[0035] While monitoring the surrounding area, the surveillance camera 120 generates video data, which in turn also includes images. By analyzing these images, or one of these images, potential smoke in the surrounding area, as indicated by 114, or flames, can be detected.

[0036] Figure 2 schematically shows two images: image 200 at the top and image 210 at the bottom. Both images essentially depict the same situation of an environment with smoke. In image 200, the smoke is detected using an anchor-based method and represented or indicated by a single rectangle 201. It is clearly visible here that the smoke only fills a small portion of the rectangle, meaning detection is rather difficult.

[0037] In image 210, the smoke is detected by a method according to the invention in a preferred embodiment. A border 211 of the smoke is significantly more detailed and precise here; this is based on cell- or image-area-wise detection of the smoke, as mentioned above and will be explained in more detail below, and is represented or indicated by a single rectangle 201.

[0038] Figure 3 schematically illustrates a preferred embodiment of a method according to the invention. Such a method can run entirely or partially, e.g., in the control or processing unit 122 of the surveillance camera 120 or in the server or computer 130, or split between both.

[0039] In a step 300, an image 300 of the surroundings is first provided. This image may have been captured by the surveillance camera; in particular, this image may also have been extracted from video data or a video sequence from the surveillance camera. Although the method is explained here only for one image, the method can also be repeatedly performed for a new or subsequent image (e.g., from the video data) in order to achieve continuous monitoring or detection.

[0040] In a step 310, for each of several image regions of the image 300, it is determined whether smoke and / or flames are detected therein, using a machine learning algorithm 312 that receives the image or image regions as input. Two image regions 302, 303 are shown as examples later in a subsequent step.

[0041] It can be seen that the machine learning algorithm 312, e.g., a neural network, has several processing layers, the final layer of which is designated 314. These can be convolutional, activation, normalization, and pooling layers, or a combination thereof. Depending on the type of network, only some of these layers may be used.

[0042] In particular, the last layer 314 has a dimension that corresponds to the product of a number of horizontal image regions of the image, a number of vertical image regions of the image, and a number of classes. The number of classes is, for example, three and includes the classes: smoke detected, flames detected, and neither smoke nor flames detected (i.e., no detection).

[0043] As an output of the machine learning algorithm 312, information 321 is determined for each image area (two image areas 302, 303 are indicated here as an example), indicating whether smoke and / or flames have been detected therein. In step 320, this information can be made available for further use; for example, if smoke or flames are detected, an alarm can be triggered or another action can be initiated.

[0044] An activation function of this final layer 314 can, for example, be a so-called softmax function, whereby the outputs can be interpreted as probabilities for the respective class. The output can thus be interpreted as a probability estimate for the presence of smoke or flames in the respective image areas. It can then be concluded that smoke is present, for example, if the probability for the presence of smoke is higher than a certain, predefined threshold.

[0045] Figure 4 schematically shows a sequence of a method according to the invention in a further preferred embodiment, namely a training of a machine learning algorithm, as it can be used - in particular after training - to detect smoke and / or flames in an image of an environment, as was explained in more detail with reference to Figure 3, for example.

[0046] For this purpose, training data 402 is first provided in step 400. The training data 402 comprises a plurality of images 404, each with a plurality of image regions, and a plurality of labels 406 that indicate, for at least some, preferably all, image regions, whether smoke and / or flames can be detected therein. The images or image regions can, for example, be of the type also explained and shown with reference to Figure 3; however, it is understood that a significantly larger number of such images or image regions, and in particular also from different environments, is expedient for training.

[0047] In a step 410, a machine learning algorithm 412 is then adapted based on the training data 402. This occurs in such a way that the machine learning algorithm 412 determines information for each image region, whether smoke and / or flames have been detected therein. For example, weights or so-called neurons (or general parameters) of the machine learning algorithm are adapted, i.e., changed if necessary. In this way, a trained or machine learning algorithm 422 is obtained, which can be made available for further use in a step 420.

[0048] The training of the machine learning algorithm can be carried out, for example, on a suitable computer, e.g. on the server or computer 130 as shown in Figure 1.

Claims

Claims 1. A method for detecting smoke and / or flames in an image of an environment using a machine learning algorithm, comprising: Providing (300) an image (301) of the environment; Determining (310) for each of a plurality of image regions (302, 303) of the image whether smoke and / or flames are detected therein, using the machine learning algorithm (312) receiving the image or image regions as input; and Providing (320), as an output of the machine learning algorithm, information (321) for each image area as to whether smoke and / or flames have been detected therein.

2. The method according to claim 1, wherein the machine learning algorithm (312) has a plurality of processing layers (314), and / or the machine learning algorithm (312) is designed to detect smoke and / or flames in the image (301) and to divide the image (301) into a plurality of separate image areas (302, 303) in which smoke and / or flames have been detected.

3. The method of claim 2, wherein a last of the plurality of processing layers has a dimension equal to a product of a number of horizontal image regions of the image, a number of vertical image regions of the image, and a number of classes, and wherein the number of classes is at least two.

4. The method according to claim 3, wherein the classes are selected from: Smoke detected, flames detected, and - no smoke or flames detected.

5. Method according to one of the preceding claims, wherein the machine learning algorithm comprises an artificial neural network, in particular a convolutional neural network, and / or is based on deep learning.

6. The method according to any one of the preceding claims, wherein the machine learning algorithm receives the image (301) as input and, when applied, divides the image into the plurality of image regions.

7. A method for training a machine learning algorithm to detect smoke and / or flames in an image of an environment, comprising: Providing (400) training data (402) comprising a plurality of images each having a plurality of image regions, and a plurality of labels indicating for at least some, preferably all, image regions whether smoke and / or flames can be recognized therein; Adapting (410) the machine learning algorithm (412) based on the training data such that the machine learning algorithm determines information for each image area as to whether smoke and / or flames have been detected therein; and Providing (420) the trained machine learning algorithm (422).

8. The method according to claim 7, wherein the plurality of labels are generated from existing labels which each indicate for an image whether smoke and / or flames can be seen therein, by transferring information from the existing labels, in particular in an automated manner, to the image areas.

9. A computing unit (122, 130) configured to carry out all method steps of a method according to any one of the preceding claims.

10. A computer program which causes a computing unit (122, 130) to carry out all method steps of a method according to one of claims 1 to 8 when it is executed on the computing unit (122, 130).

11. A machine-readable storage medium having a computer program according to claim 10 stored thereon.

Images