Detection of unstructured moving objects with the aid of difference images

Abstract

Sensor arrangement 2 for detecting unstructured moving objects 6 in a monitoring area 4, comprising: at least one camera 8, that records images 10a-d at different times ta-d; a difference module 14 that creates difference images 16a-c from two images of different times; and a neural network 18 trained to classify 20a-c objects in the difference image. The object may be a fire feature, smoke or other damage feature such as moving water. The difference images are formed from images with a constant time difference (D) of their acquisition times (ta-d). The neural network may be a CNN trained with difference images. The classifier may contain object location data (O). A hypothesis module 22 may generate a hypothesis 26 about objects in difference images in a time window (T) using at least one classification result (20a-c). A verification module 28 may test the hypothesis using all classification results and the time points (ta-c) in the time window (T) and provides an output indicating if the hypothesis is correct. A training environment 32 may be provided to the neural network.

Description

Title Detection of unstructured moving objects with the aid of difference images Prior art A smoke detection method is known from CN 111461076 A. Disclosure of the invention The subject of the invention is a sensor arrangement according to patent claim 1. Preferred or advantageous embodiments of the invention and other categories of the invention are shown in the further claims, the following description and the attached figures. This serves or is set up for the detection of potential moving objects, wherein the objects can occur in a monitoring area and this is examined for the objects potentially present there. In other words, the surveillance area is monitored for the potential occurrence of such objects. The objects are in particular “unstructured” objects, such as smoke, steam, flames, flowing or moving water / oil or other materials. “For detection” means that the sensor arrangement supports detection, but this does not have to be completed within the sensor arrangement. The sensor arrangement contains at least one camera. The camera is set up to record images of the surveillance area during operation. The images are recorded at different times. In other words, there are at least two images in operation that were taken at different times. The sensor arrangement contains a differential module. The difference module is set up to create difference images of two images each, wherein in each case two images that were taken at different times are used. The difference is therefore formed over time. In particular, the camera is stationary, at least in relation to the two images. Image content that does not change over time, e.g. a landscape, part of a building, etc., is thus “deleted” in the difference image. The sensor arrangement contains a neural network. When the network or sensor arrangement is in operation, it is fed with the differential images, i.e. it processes them as input variables. The neural network is set up to assign a classifier to a respective difference image. The classifier is correlated with an object potentially recognised / contained in the difference image. In other words, the neural network is set up to examine and classify the difference images for potential objects. The “assignment of a classifier” can also be understood to mean that no classifier is assigned, for example if no potential object is recognised in the difference image. Alternatively, a classifier can also be assigned in this case, which then contains the information that no potential object was recognised in the relevant difference image. The classifier therefore describes a classified potential object or, if applicable, that no potential object has been detected. The classifier can, for example, contain information / data about the location in the difference image where the potential object is suspected / recognised, the type of object, its size, characteristics, etc. The neural network is set up to provide the classifier. This enables the detection of potential objects in the monitored area by means of or with the aid of the classifier. The sensor arrangement thus provides classifiers that are correlated with the detection of potential objects in the monitored area or enable such detection. The neural network is in particular a deep neural network with a layered structure and at least 5 layers. Such networks are particularly suitable for image processing. According to the invention, the advantage is that a smaller number of sensors, for example only one camera, is required to monitor the surveillance area for such objects in order to cover even large surveillance areas. If the objects are smoke, for example, a different approach would require a large number of smoke detectors in the monitoring area. The detection or classification of objects, especially in the form of smoke, is faster than that of conventional smoke detectors. This results in additional benefits / functionality for surveillance cameras. This applies in particular if they are also used as cameras in the sensor arrangement, but the cameras also serve as regular surveillance cameras, for example, and their images are monitored by operating personnel. The invention provides a new input modality for neural networks, namely difference images instead of complete images. In a preferred embodiment, the sensor arrangement also includes a hypothesis module. This is set up to create a hypothesis about a potentially detected object in the surveillance area for certain difference images using at least one of the classifiers assigned to them. The difference images in question are those that he within a specific time window or period. The sensor arrangement also contains a verification module. This is set up to test the hypothesis. For this purpose, all classifiers assigned to the relevant difference images in the time window and the respective points in time underlying the difference images (their constituent images) in the relevant time window are used (“accumulated”). The verification module is also set up to provide an output vanable based on the result of this check. The output variable indicates the detection of an object if the hypothesis was evaluated as correct according to the corresponding test. If, according to the test, the hypothesis is tested / evaluated as not applicable, either a corresponding statement is provided as an output variable or no output variable is provided. The sensor arrangement thus becomes a complete detector for an object in the monitored area and not only supplies the classifiers used for this purpose. The sensor arrangement is therefore no longer just a basis for enabling corresponding detection, but provides actual detection by providing a corresponding output variable that indicates detection or not. The output variable can also include an alarm, for example, which indicates that an object (fire, smoke) has been detected. In a preferred embodiment, the object is a fire feature, i.e. a feature of a fire or other damage feature. In particular, the fire or damage is present in the monitoring area. Such a fire feature is, for example, smoke, open fire, flames or similar, which is recognizable or manifests itself in the camera images. Another sign of damage is, for example, spreading puddles of water on a floor where the water is accompanied by movement, such as an enlarging puddle or flowing water. The sensor arrangement can therefore detect fires or damage. In a preferred embodiment, the classifier contains location data of the potential object in the difference image. Such location data is represented, for example, by means of a selection frame / bounding box in the difference image. A corresponding selection frame could also be displayed to a user of the sensor arrangement, e.g. using an image on which the relevant difference image is based. This allows the user to intuitively recognise the location of a potentially detected object. In a preferred embodiment, the images are continuous images of a video stream, wherein the video stream is recorded by the camera. Tire times of the images are therefore cyclically constant recording times of the individual images or frames of the video stream. This makes it particularly easy to take pictures at different times. In a preferred embodiment, the difference images are formed by images / image pairs with a constant time difference between their respective recording times. In other words, each of the difference images is based on two images taken at a fixed time interval. Preferably, the images and / or the difference image formed are colour images. In one variant, the images and / or the difference image formed are a grey value image. To form the difference image, the grey values and / or the colour values, in particular the RGB colour values, of the corresponding pixels of the image pairs are subtracted from each other and the difference value is preferably normalised (amount formation). This normalised difference image as a difference grey value image and / or difference colour value image is fed to the neural network for training and / or detection of potential moving objects. This is particularly easy to implement in conjunction with the above-mentioned video stream, for example by processing successive images into a difference image. However, images that are further apart in time can also be processed into difference images, for example only every second, fifth, tenth etc. image of the video stream. In a preferred embodiment, the neural network is a convolutional neural network (CNN), in particular a so-called “Faster R-CNN”. Such networks are particularly suitable for object recognition, especially in images of video streams. However, other neural networks (especially so-called “detection” networks) can also be used. A further object of the invention is a network arrangement according to claim 8. This contains the sensor arrangement according to the invention. The network arrangement also contains a training environment for the neural network. The training environment is set up to use a camera (directed at the monitoring area, in particular the camera of the sensor arrangement) to record or receive positive images of the monitoring area (or to use them if they are available) that are known to contain objects to be detected; on the other hand, negative images of the monitoring area are also known to contain no objects to be detected. The training environment is also set up to create difference images and to assign correct classifiers to them or to receive such difference images and / or classifiers if they are already available (the underlying positive / negative images may then not be required). “Known correct” means that the classifier actually correctly indicates whether and what type of object etc. is in a difference image or not. The training environment is also set up to train the neural network on the basis of the difference images and the known correct classifiers. This provides a fully-fledged system that can train the neural network and also use it in operation. The invention is based on the following findings, observations or considerations and also has the following preferred embodiments. These embodiments are sometimes also referred to as “the invention” for the sake of simplicity. The embodiments can also include or correspond to parts or combinations of the above-mentioned embodiments and / or in a given case also include embodiments not previously mentioned. The invention is based on the following observation: Desired is the detection of unstructured, moving objects such as smoke / fire in warehouses, puddles of water on the floor. This has potential applications in the areas of surveillance and security. In practice, the usual computer vision methods such as colour, motion and texture analysis are used to detect unstructured objects such as smoke. Video-based smoke detection using the optical flow characteristics is also conceivable. Detection takes place by searching for a directional movement behaviour in the video that resembles the typical movement of smoke. To detect this type of movement, the smoke must be dense enough and the detection time should be long enough. Other features such as colour and contrast are used for further verification. Machine learning algorithms, in which the characteristics of smoke are learned from statistical properties of the data, can also be used for smoke detection. The recognition of objects by machine learning can be based on Convolutional Neural Networks (CNNs), for example. The (deep) neural network, which can be used for the recognition of unstructured moving objects (events), has two modes: Training and operation (test): In training mode, one or more cameras are used to record RGB image sequences (images) of positive scenarios (monitoring area with object) that contain the object to be detected (e.g. smoke); and of negative scenarios (monitoring area without object) that do not contain the object to be detected. The positive sequences are marked with bounding boxes (classifier) that indicate the position of the object in each image. A neural network for object recognition (Faster R-CNN) is trained by using the difference (difference image) between two images with a certain time interval (time difference) as the image source (input for the neural network) and the positions of the bounding boxes (known correct classifiers). In operating mode, a camera is used to record an image sequence of the monitored area (monitoring area). The difference between two images is calculated (difference image) and used to detect smoke (object) with the best trained model (neural network). The detections are accumulated both spatially and temporally (in the hypothesis module) in order to form a hypothesis for a smoke alarm. Each hypothesis is evaluated based on the reliability of its recognition list (classifiers) and its age (verification module). The hypotheses are classified (output variable) as to whether it is a smoke alarm or not. The use of difference images has the advantage that the background details are suppressed and at the same time sufficient texture is retained for the object to be learned. Consequently, the amount of data required for training is less than that of the RGB data. According to the invention, a difference image is therefore fed into the neural network itself. Alarm verification is carried out in the verification module by accumulating detection results over several images. The invention is particularly suitable for use indoors, i.e. the monitoring area is an interior of a building, etc. The invention is particularly suitable for detecting smoke as an object, but can also be used, for example, to detect a puddle of water as an object. The invention further comprises a method for recognising potential moving objects in a surveillance area, in particular for recognising a fire feature, wherein difference images of two images each of different points in time of the surveillance area are created, wherein a neural network is fed with the difference images, wherein the neural network assigns and provides a classifier correlated with potentially recognised objects to a respective difference image. Preferably, the neural network is a CNN and / or the neural network has been trained with difference images. In particular, the differential images used for training are known to be positive images of the monitoring area with objects and / or known to be negative images without objects. The neural network is designed as a neural network for recognising and localising objects. Preferably, the neural network is an RCNN (Region Based Convolutional Neural Network), in particular a Faster RCNN. Alternatively or additionally, the neural network is a YOLO neural network. Preferably, labelled difference images, in particular those labelled with the position of objects, are used for training in the form of supervised training. Furthermore, the inventor comprises a computer program which is set up to carry out all the steps of the method described. The invention further comprises a machine-readable storage medium, in particular a non-volatile machine-readable storage medium, on which the computer program is stored, and an electronic control unit which is set up to carry out all the steps of the method described. Further features, effects and advantages of the invention are shown in the following description of a preferred embodiment example of the invention and in the accompanying figures. In each case, a schematic sketch of the principle is shown: Figure 1 sensor arrangement in operation, Figure 2 the neural network from Figure 1 in a training mode. Figure 1 shows a sensor arrangement 2. It is used to monitor a surveillance area 4, in this case a warehouse, in order to be able to detect potentially occurring unstructured moving objects 6, in this case smoke from a fire. The sensor arrangement 2 contains a camera 8 which, in operation B (of the camera 8 and thus of the sensor arrangement 2), records images 10 of or from the monitoring area 4. As an example, four images lOa-d are shown in Figure 1. Each of the images lOa-d is recorded at a recording time ta-d, wherein the recording times are different. The camera 8 here is a video camera that generates a continuous video stream 12, symbolically indicated here by an arrow. The images lOa-d are continuous individual images of the video stream 12, the times ta-d are the respective cyclically continuous recording times of the individual frames of the video stream 12. In other words, the video stream 12 represents an image sequence of the images 10. The sensor arrangement 2 contains a differential module 14, indicated here only symbolically by an arrow. This generates a respective difference image 16a-c of two of the images lOa-d of different recording times ta-d In the present case, the difference image 16a is the result of a subtraction of image 10b from image 10a. The difference image 16b is formed from a difference of the image 10c minus the image 10b; the difference image 16c is formed from a difference of the images lOd and 10c. The difference module 14 therefore calculates an “image difference”. The sensor arrangement 2 also contains a neural network 18, which is fed with the difference images 16a-c in operation B. The neural network 18 assigns a classifier 20a-c to each of the difference images 16a-c. The classifier 20a-c indicates whether a potential object 6 was detected in the respective difference image 16a-c and, if so, at which location in the image. The classifiers 20a-c are designed here as selection frames to indicate the location. The classifiers 20a-c thus contain location data O of the potential object 6 in the difference image 16a-c. In order to give a viewer a realistic picture of the situation in the monitoring area 4, the classifiers 20a-c are entered in one of the images lOa-c in the example, each of which has contributed to the formation of the difference images 16a-c as minuends. As an example, the classifier 10a from the difference image 16a is entered in the image 10a. In other words, the neural network 18 performs single frame detection. The sensor arrangement 2 also contains a hypothesis module 22. This looks at or evaluates the respective difference images 16 that lie within a certain time window T. In the present case, the time window T is selected so that it contains exactly three consecutive difference images 16, in this case the difference images 16a-c. For the sake of clarity, these are shown in the figure by their corresponding constituent minuend images lOa-c. The hypothesis module 22 evaluates all difference images 16a-c in the time window T, associated classifiers 20a-c and the respective associated acquisition times ta-c on which the difference images 16a-c are based. Based on this data, it creates a hypothesis 26 about a potentially detected object 6 in the monitoring area 4. The sensor arrangement 2 also includes a verification module 28. This verifies the hypothesis 26 using the underlying classifiers 20a-c and the underlying times ta-c, for example with regard to matching location, the same type of identified object 6 (always smoke), etc. On the basis of this check, the verification module 28 provides an output variable 30 which indicates the detected object 6, since the hypothesis 26 was evaluated as correct. The output variable 30 contains an alarm signal (red flashing selection frame in the image) and uses the selection frame to indicate the potentially detected object 6 or its position in the image. Object 6 is therefore a fire feature here, namely smoke. The time difference D between two recording times ta-d of the images lOa-d processed into difference images 16a-c is therefore constant or the same in each case and is the time interval between two frames of the video stream 12. The neural network 18 here is a Faster R-CNN. Figure 1 also shows a symbolic network arrangement 31. In addition to the sensor arrangement 2, this also contains a training environment 32 for the neural network 18, symbolically indicated here by an arrow. Figure 2 shows the mode of operation of the training environment 32, namely the training of the neural network 18 before its actual operation B. Here too, images 10 of the monitoring area 4 are recorded with the camera 8, wherein positive images 34 are included, which knowingly contain an object 6; as well as negative images 36, which knowingly do not contain any of the objects 6. As described above, difference images 16 are generated from all these images 10 (indicated by an arrow). Based on the knowledge of the objects, the respective knowingly correct classifiers 20 are known for the positive images 34 and negative images 36 and therefore the resulting difference images 16 (indicated by an arrow). Based on these difference images 16 with known classifiers 20, the neural network 18 is trained (indicated by arrows) so that the result (indicated by the arrow 38) is a trained neural network 18. This can then be used for operation B as shown in Figure 1. Figure 1 thus shows the operating mode or operation B of the sensor arrangement 2. Figure 2, on the other hand, shows the training mode of the sensor arrangement 2 or network arrangement 31.

Claims

1. Sensor arrangement (2) for detecting potential moving objects (6) in a monitoring area (4),- with at least one camera (8), which is set up to record images (lOa-d) of the monitoring area (4) at different recording times (ta-d) during operation (B),- with a difference module (14) which is set up to create difference images (16a-c) of two images (lOa-d) of different times (ta-d),- with a neural network (18) fed with the difference images (16a-d) during operation (B), which is set up to assign and provide a classifier (20a-c) correlated with potentially recognised objects (6) to a respective difference image (16a-c).

2. Sensor arrangement (2) according to claim 1, characterised in thatthe sensor arrangement (2) further comprises- a hypothesis module (22) which is set up to generate a hypothesis (26) about a potentially detected object (6) for difference images (16a-c) which lie within a specific time window (T), using at least one of its classifiers (20a-c),- a verification module (28) which is set up to test the hypothesis (26) using all classifiers (20a-c) and the time points (ta-c) in the time window (T) on which the difference images (16a-c) are based and to provide an output variable (30) on the basis of this test, which indicates the detection of an object (6) if the hypothesis (26) is tested as correct.

3. Sensor arrangement (2) according to one of the preceding claims, characterised in thatthe object (6) is a fire feature or other damage feature.

4. Sensor arrangement (2) according to one of the preceding claims, characterised in thatthe classifier (20a-c) contains location data (0) of the potential object (6) in the difference image (16a-c).

5. Sensor arrangement (2) according to one of the preceding claims, characterised in thatthe images (lOa-d) are continuous images of a video stream (12) recorded by the camera (8).

6. Sensor arrangement (2) according to one of the preceding claims, characterised in thatthe difference images (16a-c) are formed from images (lOa-d) each with a constant time difference (D) of their acquisition times (ta-d).

7. Sensor arrangement (2) according to one of the preceding claims, characterised in thatthe neural network (18) is a CNN and / or the neural network (18) has been trained with difference images (16a-c).

8. Network arrangement (31),- with the sensor arrangement (2) according to one of the preceding claims,- with a training environment (32) for the neural network (18), which is set up for this purpose, - to record or receive positive images (34) of the monitoring area (4) known to be with objects (6) and negative images (36) known to be without objects (6),- difference images (16a-c) and to assign known correct classifiers (20a-c) to them or to receive such classifiers,- train the neural network (18) on the basis of the difference images (16a-c) and the known correct classifiers (20a-c).

9. Method for recognising potential moving objects (6) in a surveillance area (4), in particular for recognising a fire feature, wherein difference images (16a-c) of two respective images (lOa-d) of different points in time (ta-d) of the surveillance area (4) are created, wherein a neural network (18) is fed with the difference images (16a-d), wherein the neural network (18) assigns and provides a classifier (20a-c) correlated with potentially recognised objects (6) to a respective difference image (16a-c).

10. Method according to claim 9, characterised in thatthe neural network (18) is a CNN and / or the neural network (18) has been trained with difference images (16a-c).

11. Method according to claim 10, characterised in thatthe differential images (16a-c) used for training are formed as positive images (34) of the monitoring area (4) known to be with objects (6) and / or as negative images (36) known to be without objects (6).

12. A computer program which is arranged to carry out all the steps of the method according to any one of claims 9 through 11.

13. Machine-readable storage medium, in particular a non-volatile machine-readable storage medium, on which the computer program according to claim 12 is stored.

14. An electronic control unit adapted to perform all the steps of the method according to any one of claims 9 through 11.

Images