Detection of unstructured moving objects using difference images

Abstract

A sensor arrangement (2) for potential moving objects (6) includes a camera (8) for images of a surveillance area (4), a difference module (14) to create difference images (16a-c) of the images (10a-d), and a neural network (18) fed with the difference images (16a-d) to assign and provide classifiers (20a-c) for potentially detected objects (6) to the difference images (16a-c). A network arrangement (31) with the sensor arrangement (2) includes a training environment (32) for the neural network (18) to capture or receive positive images (34) of the monitoring area (4) known to contain objects (6) and negative images (36) known to contain no objects (6), to form difference images (16a-c) and to assign or receive known correct classifiers (20a-c) to these, and to train the neural network (18) on the basis of the difference images (16a-c) and the known correct classifiers (20a-c).

Description

State of the art

[0001] A smoke detection method is known from CN 111461076 A. Disclosure of the invention

[0002] The invention relates to a sensor arrangement according to patent claim 1. Preferred or advantageous embodiments of the invention and other categories of invention emerge from the further claims, the following description and the attached figures.

[0003] This serves or is designed to detect potentially moving objects. These objects may occur in a surveillance area, which is then examined for the potential presence of such objects. In other words, the surveillance area is monitored for the potential occurrence of such objects. These objects are, in particular, "unstructured" objects, such as smoke, steam, flames, flowing or moving water / oil, or other materials.

[0004] “For detection” means that the sensor array assists in detection, but detection does not need to be completed within the sensor array.

[0005] The sensor array contains at least one camera. The camera is configured to capture images of the surveillance area during operation. The images are captured at different times. In other words, at least two images exist during operation, each captured at different times.

[0006] The sensor array contains a difference module. The difference module is configured to create difference images from two images each, using two images taken at different times. The difference is thus created over time. The camera is, in particular, fixed in position, at least with respect to the two images. Image content that does not change over time, e.g., a landscape, a part of a building, etc., is thus "deleted" from the difference image.

[0007] The sensor array contains a neural network. During operation of the network or sensor array, it is fed with the difference images and processes them as input variables. The neural network is configured to assign a classifier to each difference image. The classifier is correlated with an object potentially detected or contained in the difference image. In other words, the neural network is configured to examine the difference images for potential objects and classify them. The "assignment of a classifier" can also mean that no classifier is assigned if necessary, for example, if no potential object is detected in the difference image. Alternatively, a classifier can also be assigned in this case, which then contains the information that no potential object was detected in the difference image in question.The classifier describes a classified potential object, and possibly also the fact that no potential object was detected. The classifier can contain, for example, information / data about the location in the difference image where the potential object is suspected / detected, the type of object, its size, characteristics, etc.

[0008] The neural network is configured to provide the classifier. This enables the detection of potential objects in the surveillance area based on or with the help of the classifier.

[0009] The sensor arrangement thus provides classifiers that are correlated with the detection of potential objects in the surveillance area or enable such detection.

[0010] The neural network is, in particular, a deep neural network with a layered structure and at least five layers. Such networks are particularly suitable for image processing.

[0011] The invention has the advantage that a smaller number of sensors, for example only one camera, is required to monitor the surveillance area for such objects, even covering large surveillance areas. If the objects are smoke, for example, a different approach would require a large number of smoke detectors in the surveillance area. The detection and classification of objects, particularly in the form of smoke, is faster than with conventional smoke detectors. This results in additional benefits / functionality for surveillance cameras. This is particularly true if they are used as a camera in the sensor arrangement, but the cameras also serve, for example, as regular surveillance cameras and their images are observed by operating personnel. The invention results in a new input modality for neural networks, namely difference images instead of complete images.

[0012] In a preferred embodiment, the sensor arrangement further includes a hypothesis module. This module is configured to create a hypothesis about a potentially detected object in the surveillance area for specific difference images based on at least one of the classifiers assigned to them. The difference images in question are those that lie within a specific time window or period.

[0013] The sensor array also contains a verification module. This module is designed to test the hypothesis. For this purpose, all classifiers assigned to the respective difference images in the time window and the respective time points underlying the difference images (their constituent images) in the respective time window are used ("accumulated"). The verification module is also designed to provide an output variable based on the result of this test. The output variable indicates the detection of an object if the hypothesis was evaluated as correct according to the corresponding test. If the hypothesis is tested / evaluated as incorrect according to the test, either a corresponding statement is provided as an output variable or no output variable is provided.

[0014] Thus, the sensor array becomes a complete detector for an object in the surveillance area and not only provides the classifiers needed for this purpose. The sensor array no longer merely represents a basis for enabling corresponding detection, but rather offers actual detection by providing a corresponding output variable that indicates whether or not detection has occurred.

[0015] The output variable can also include, for example, an alarm indicating that an object (fire, smoke) has been detected.

[0016] In a preferred embodiment, the object is a fire characteristic, i.e., a characteristic of a fire or other damage characteristic. In particular, the fire or damage is present in the surveillance area. Such a fire characteristic is, for example, smoke, open fire, flames, or similar, which is recognizable or manifests itself in the camera images. Another damage characteristic is, for example, spreading puddles of water on a floor, where the water is accompanied by movement, for example, an expanding puddle or flowing water. The sensor arrangement can thus detect corresponding fires or damage.

[0017] In a preferred embodiment, the classifier contains location data of the potential object in the difference image. Such location data is represented, for example, by a bounding box in the difference image. A corresponding bounding box could also be displayed to a user of the sensor array, e.g., using an image underlying the respective difference image. This allows the user to intuitively identify the location of a potentially detected object.

[0018] In a preferred embodiment, the images are continuous images of a video stream, with the video stream being recorded by the camera. The times of the images are thus cyclically constant, continuous recording times of the individual images or frames of the video stream. This makes it particularly easy to record images at different recording times.

[0019] In a preferred embodiment, the difference images are formed from images / image pairs with a constant time difference between their recording times. In other words, each of the difference images is based on two images that were recorded at a fixed time interval from one another. Preferably, the images and / or the formed difference image are color images. In one variant, the images and / or the formed difference image are a grayscale image. To form the difference image, the grayscale values ​​and / or the color values, in particular the RGB color values, of the corresponding pixels of the image pairs are subtracted from one another and the difference value is preferably normalized (magnitude formation). This normalized difference image as a difference grayscale image and / or difference color value image is fed to the neural network for training and / or for detecting potentially moving objects.This is particularly easy to implement in conjunction with the video stream mentioned above, for example, by processing consecutive images into a difference image. However, images separated by greater distances in time can also be processed into difference images, for example, only every second, fifth, tenth, etc., image of the video stream.

[0020] In a preferred embodiment, the neural network is a convolutional neural network (CNN), in particular a so-called "Faster R-CNN." Such networks are particularly suitable for object recognition, especially in images from video streams. However, other neural networks (in particular so-called "detection" networks) can also be used.

[0021] A further subject matter of the invention is a network arrangement according to claim 8. This contains the sensor arrangement according to the invention. Furthermore, the network arrangement contains a training environment for the neural network. The training environment is configured to record or receive (or use, if available) positive images of the surveillance area with the aid of a camera (directed at the surveillance area, in particular the camera of the sensor arrangement), on the one hand, positive images of the surveillance area that are known to contain objects to be detected; on the other hand, negative images of the surveillance area that are known not to contain objects to be detected. The training environment is also configured to form difference images and assign known correct classifiers to them, or to receive such difference images and / or classifiers if they are already available (the underlying positive / negative images may then not be required)."Known correct" means that the classifier actually correctly indicates whether and what type of object, etc., is present in a difference image. The training environment is also set up to train the neural network based on the difference images and the known correct classifiers.

[0022] This provides a fully functional system that can train the neural network and also use it in operation.

[0023] The invention is based on the following findings, observations, and considerations and also includes the following preferred embodiments. These embodiments are sometimes referred to as "the invention" for simplicity. The embodiments may also contain parts or combinations of the above-mentioned embodiments or correspond to them and / or may also include previously unmentioned embodiments.

[0024] The invention is based on the following observation: The desired goal is to detect unstructured, moving objects such as smoke / fire in warehouses or puddles of water on the floor. This has potential applications in surveillance and security.

[0025] In practice, common computer vision methods such as color, motion, and texture analysis are used to detect unstructured objects, such as smoke.

[0026] Video-based smoke detection using optical flow features is also conceivable. Detection is performed by searching for directed motion behavior in the video that resembles typical smoke movement. To detect this type of movement, the smoke must be dense enough and the detection time should be long enough. Other features such as color and contrast are used for further verification.

[0027] Machine learning algorithms, which learn smoke characteristics from statistical properties of the data, can also be used in smoke detection. Object detection through machine learning can, for example, be based on convolutional neural networks (CNNs).

[0028] The deep neural network, which can be used for the detection of unstructured moving objects (events), has two modes: training and operation (test): In training mode, one or more cameras are used to capture RGB image sequences (images) of positive scenarios (surveillance area with object) that contain the object to be detected (e.g. smoke); and of negative scenarios (surveillance area without object) that do not contain the object to be detected. The positive sequences are labeled with bounding boxes (classifiers) that indicate the position of the object in each image. A neural network for object detection (Faster R-CNN) is trained by using the difference (difference image) between each two images with a certain time interval (time difference) as the image source (input for the neural network) and the positions of the bounding boxes (known correct classifiers).

[0029] In operational mode, a camera is used to capture a sequence of images of the monitored area (surveillance zone). The difference between two images is calculated (difference image) and used to detect smoke (object) with the best-trained model (neural network). The detections are accumulated both spatially and temporally (in the hypothesis module) to form a hypothesis for a smoke alarm. Each hypothesis is evaluated based on the reliability of its detection list (classifiers) and its age (verification module). The hypotheses are classified (output) as to whether or not a smoke alarm is present.

[0030] Using difference images has the advantage of suppressing background details while retaining sufficient texture for the object to be trained. Consequently, the amount of data required for training is smaller than that required for RGB data.

[0031] According to the invention, a differential image itself is fed into the neural network. Alarm verification takes place in the verification module by accumulating detection results across multiple images. The invention is particularly suitable for use indoors, i.e., when the monitored area is the interior of a building, etc. The invention is particularly suitable for detecting smoke as an object, but can also be used, for example, to detect a puddle of water as an object.

[0032] The invention further comprises a method for detecting potentially moving objects in a surveillance area, in particular for detecting a fire feature, wherein difference images are created from two images of the surveillance area taken at different times, wherein a neural network is fed with the difference images, wherein the neural network assigns and provides a classifier correlated with potentially detected objects to each difference image. Preferably, the neural network is a CNN and / or the neural network was trained with difference images. In particular, the difference images used for training are designed as positive images of the surveillance area known to contain objects and / or as negative images known to contain no objects. The neural network is designed as a neural network for detecting and localizing objects.Preferably, the neural network is an RCNN (Region Based Convolutional Neural Network), in particular a Faster RCNN. Alternatively or additionally, the neural network is a YOLO neural network. Preferably, labeled difference images, in particular those labeled with the position of objects, are used for training in the form of supervised training.

[0033] Furthermore, the invention includes a computer program configured to execute all steps of the described method. The invention further includes a machine-readable storage medium, in particular a non-volatile machine-readable storage medium, on which the computer program is stored, and an electronic control unit configured to execute all steps of the described method.

[0034] Further features, effects, and advantages of the invention will become apparent from the following description of a preferred embodiment of the invention and the accompanying figures. Each of these figures shows a schematic diagram: Fig. 1 a sensor arrangement in operation, Fig. 2 the neural network Fig. 1 in a training mode.

[0035] Fig. 1 shows a sensor arrangement 2. It is used to monitor a surveillance area 4, in this case a warehouse, in order to be able to detect potentially occurring unstructured moving objects 6, in this case smoke from a fire.

[0036] The sensor arrangement 2 contains a camera 8, which in operation B (of the camera 8 and thus of the sensor arrangement 2) takes pictures 10 of or from the monitoring area 4. Examples are shown in Fig. 1 shows four images 10a-d. Each of the images 10a-d was recorded at a recording time ta-d, whereby the recording times are different. The camera 8 here is a video camera that generates a continuous video stream 12, symbolically indicated here by an arrow. The images 10a-d are continuous individual images of the video stream 12, and the times ta-d are the respective cyclically continuous recording times of the individual frames of the video stream 12. In other words, the video stream 12 represents an image sequence of the images 10.

[0037] The sensor arrangement 2 contains a difference module 14, indicated here only symbolically by an arrow. This generates a respective difference image 16a-c from two of the images 10a-d from different recording times ta-d. In this case, the difference image 16a is the result of subtracting the image 10b from the image 10a. The difference image 16b is formed from the difference between the image 10c and the image 10b; the difference image 16c is formed from the difference between the images 10d and 10c. The difference module 14 thus calculates an "image difference."

[0038] The sensor arrangement 2 also contains a neural network 18, which, in operation B, is fed with the difference images 16a-c. The neural network 18 assigns a classifier 20a-c to each of the difference images 16a-c. The classifier 20a-c indicates whether a potential object 6 was detected in the respective difference image 16a-c and, if so, at which location in the image. To identify the location, the classifiers 20a-c are designed here as selection frames. The classifiers 20a-c thus contain location data O of the potential object 6 in the difference image 16a-c.

[0039] To provide a viewer with a realistic picture of the situation in surveillance area 4, in the example, classifiers 20a-c are entered into each of the images 10a-c, each of which contributed to the formation of the difference images 16a-c as minuends. For example, classifier 10a from the difference image 16a is entered into image 10a. In other words, single-frame detection is performed by the neural network 18.

[0040] The sensor arrangement 2 further contains a hypothesis module 22. This module examines or evaluates respective difference images 16 that lie within a specific time window T. In this case, the time window T is selected such that it contains exactly three temporally consecutive difference images 16, here the difference images 16a-c. For the sake of clarity, these are represented in the figure by their corresponding constitutive minuend images 10a-c. The hypothesis module 22 evaluates all difference images 16a-c in the time window T, associated classifiers 20a-c, and the respective recording times ta-c underlying the difference images 16a-c. Based on this data, it creates a hypothesis 26 about a potentially detected object 6 in the surveillance area 4.

[0041] The sensor arrangement 2 further contains a verification module 28. This checks the hypothesis 26 based on the underlying classifiers 20a-c and the underlying times ta-c, for example, with regard to matching location, the same type of identified object 6 (always smoke), etc. Based on this check, the verification module 28 provides an output variable 30 that indicates the detected object 6, since the hypothesis 26 was evaluated as correct. The output variable 30 here includes an alarm signal (red flashing selection frame in the image) and, based on the selection frame, indicates the potentially detected object 6 or its position in the image.

[0042] Object 6 here is therefore a fire characteristic, namely smoke. The time difference D between two recording times ta-d of the images 10a-d processed into difference images 16a-c is therefore constant or equal in each case and is the time interval between two frames of the video stream 12. The neural network 18 here is a Faster R-CNN.

[0043] Fig. 1 also shows symbolically indicated a network arrangement 31. In addition to the sensor arrangement 2, this also contains a training environment 32 for the neural network 18, here symbolically indicated by an arrow.

[0044] Fig. Figure 2 shows the operation of the training environment 32, namely the training of the neural network 18 prior to its actual operation B. Here, too, images 10 of the surveillance area 4 are recorded with the camera 8, including positive images 34 that knowingly contain an object 6, as well as negative images 36 that knowingly do not contain any of the objects 6. From all of these images 10, difference images 16 are again generated, as described above (indicated by an arrow). Based on the knowledge of the objects, the respective knowingly correct classifiers 20 for the positive images 34 and negative images 36, and therefore for the resulting difference images 16, are known (indicated by an arrow). Based on these difference images 16 with known classifiers 20, the neural network 18 is trained (indicated by arrows), resulting in a trained neural network 18 (indicated by arrow 38).This can then be used for company B in accordance with . Fig. 1 can be used.

[0045] Fig. 1 thus represents the operating mode or operation B of the sensor arrangement 2. Fig. 2, on the other hand, represents the training mode of the sensor arrangement 2 or network arrangement 31. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] CN 111461076 A

[0001]

Claims

[1] Sensor arrangement (2) for the detection of potential moving objects (6) in a surveillance area (4), - with at least one camera (8) which is designed to record images (10a-d) of the surveillance area (4) at different recording times (ta-d) during operation (B), - with a difference module (14) which is designed to create difference images (16a-c) of two images (10a-d) at different times (ta-d), - with a neural network (18) fed with the difference images (16a-d) during operation (B), which is designed to assign and provide a classifier (20a-c) correlated with potentially recognized objects (6) to a respective difference image (16a-c). [2] Sensor arrangement (2) according to claim 1, characterized by that the sensor arrangement (2) further contains - a hypothesis module (22) which is designed to create a hypothesis (26) about a potentially detected object (6) for difference images (16a-c) which lie within a certain time window (T) using at least one of its classifiers (20a-c), - a verification module (28) which is configured to check the hypothesis (26) on the basis of all classifiers (20a-c) and the points in time (ta-c) in the time window (T) on which the difference images (16a-c) are based, and to provide an output variable (30) on the basis of this check, which indicates the detection of an object (6) if the hypothesis (26) is checked as correct. [3] Sensor arrangement (2) according to one of the preceding claims, characterized by that the object (6) is a fire feature or other damage feature. [4] Sensor arrangement (2) according to one of the preceding claims, characterized bythat the classifier (20a-c) contains location data (O) of the potential object (6) in the difference image (16a-c). [5] Sensor arrangement (2) according to one of the preceding claims, characterized by that the images (10a-d) are continuous images of a video stream (12) recorded by the camera (8). [6] Sensor arrangement (2) according to one of the preceding claims, characterized by that the difference images (16a-c) are formed from images (10a-d) with a constant time difference (D) of their recording times (ta-d). [7] Sensor arrangement (2) according to one of the preceding claims, characterized by that the neural network (18) is a CNN and / or the neural network (18) was trained with difference images (16a-c). [8] Network arrangement (31), - with the sensor arrangement (2) according to one of the preceding claims, - with a training environment (32) for the neural network (18), which is designed to - to record or receive positive images (34) of the surveillance area (4) known to contain objects (6) and negative images (36) known to contain no objects (6), - to form difference images (16a-c) and to assign known correct classifiers (20a-c) to them or to receive such, - to train the neural network (18) on the basis of the difference images (16a-c) and the known correct classifiers (20a-c). [9] Method for detecting potentially moving objects (6) in a surveillance area (4), in particular for detecting a fire feature, wherein difference images (16a-c) of two images (10a-d) of different times (ta-d) of the surveillance area (4) are created, wherein a neural network (18) is fed with the difference images (16a-d), wherein the neural network (18) assigns and provides a classifier (20a-c) correlated with potentially detected objects (6) to a respective difference image (16a-c). [10] Method according to claim 9, characterized by that the neural network (18) is a CNN and / or the neural network (18) was trained with difference images (16a-c). [11] Method according to claim 10, characterized bythat the difference images (16a-c) used for training are formed as positive images (34) of the surveillance area (4) known to contain objects (6) and / or as negative images (36) known to contain no objects (6). [12] Computer program which is arranged to carry out all steps of the method according to one of claims 9 to 11. [13] Machine-readable storage medium, in particular non-volatile machine-readable storage medium, on which the computer program according to claim 12 is stored. [14] Electronic control unit arranged to carry out all the steps of the method according to any one of claims 9 to 11.

Images