Monitoring system with personal detector

Abstract

The invention relates to a monitoring system (1), comprising a person detector (2a), which is designed to have an imaging sensor for recording a digital image file (D1a) or a point cloud (D2a), and comprising an evaluation unit (5a), which is designed to evaluate the image file (D1a) or the point cloud (D2a) and to output a position indication (P) of a person (M) detected within a monitoring region (FOV), wherein the diagnostic unit (6) has a first input channel (7a) for receiving a position indication (P) and a first output (8) for a detected position indication (P) of the identified person (M) and a second output (10) for an error signal (11), the diagnostic unit (6) being designed with a detection unit (12), the invention relates to a device for checking the plausibility of the output of the position indication (P) of the identified person (M) using a plurality of plausibility criteria (C1,..., Ci), further designed for this situation in such a way that an error is identified when at least one plausibility criterion (C1,..., Ci) is violated, and an error signal (11) is output at a second output (10).

Description

Technical Field

[0001] The invention relates to a monitoring system with a person detector which is equipped with an imaging sensor, preferably with a camera for recording digital image files or with a laser sensor system, or with a TOF camera (Time of Flight) for recording a point cloud, and with an evaluation unit which is designed to evaluate the image files or the point cloud and output an indication of the position of persons detected in the monitoring area. Background Art

[0002] Numerous methods and devices for identifying and locating persons are known in the literature, which use, in particular, a camera whose video stream is analyzed frame by frame by an artificial neural network. The neural network can provide the position and extent (length, height) of the person identified in the two-dimensional image. From this position, the actual position of the person can then be determined.

[0003] For example, the use of industrial robots in the vicinity of humans without protective separation requires suitable sensor-based environmental detection in order to reliably detect people and other objects and thus be able to react in an appropriate manner in the event of a threatening collision.

[0004] In the special paper "Workspace Monitoring and Autonomous Path Planning for Safe and More Flexible Robot-Assisted Systems in Manufacturing" by Christina Ramer, various methods for person recognition using camera image evaluation with the aid of neural networks are described.

[0005] EP 3 709 106 A1 discloses a safety system and a method for protecting a machine having a first sensor and a second sensor.

[0006] DE 10 2020 133 784 B4 shows a method with a security system and a security system for locating at least one object, the security system having at least one control unit and an evaluation unit and a radio positioning system.

[0007] EP 2 825 812 B1 describes a device for protecting a hazardous working area of ​​an automatically operated machine, which device has at least one sensor unit which in each case supplies a current 3D image of the working area within defined time intervals and monitors a defined protective zone.

[0008] In EP 3 588 365 A1, a reliable multi-channel evaluation is improved since, in order to check the integrity of the processing channels, not all data of the processing results have to be compared, but rather a small number of informative characteristic values ​​are sufficient.

[0009] EP 1 964 063 B1 discloses a method for configuring a monitoring device for monitoring a spatial area, in particular for the fail-safe monitoring of a working area of ​​an automatically operated machine or system. Summary of the invention

[0010] In the context of the present invention, a fail-safe automation system is understood to be an industrial controller which is certified according to the IEC 62061 standard and is therefore designed as a controller designed for functional safety.

[0011] Fail-safe automation systems control processes that can immediately assume a safe state as a result of an unexpected operating behavior or a fault. This is the case with fail-safe control processes, in which an immediate shutdown into a safe state is shown without risk to people or the environment. Fail-safe systems go beyond conventional safety engineering and activate a wide range of intelligent systems, which extend across electric drives and measuring systems. Users use fail-safe systems in applications with increased safety requirements. Due to improved error recognition and error localization in fail-safe systems with the aid of detailed diagnostic information, production can be quickly restarted again after safety-related interruptions.

[0012] According to the prior art, known personnel detectors today already achieve very high quality. However, the requirements arising from the context of "functional safety" are not met. These requirements include reliability diagnostics that reveal all possible errors in the monitoring system and its subsystems (such as, for example, cameras, laser scanners, artificial neural networks, communication links, etc.).

[0013] It is therefore an object of the present invention to provide a monitoring system which makes it possible to reliably detect the occurrence of errors, for example in camera-based people detectors.

[0014] For the above monitoring system, this object is achieved by the features of claim 1. The diagnostic unit is equipped with a first input channel for receiving a position indication and a first output for checking a position indication (P) of the identified person, for example a confirmation signal of the position indication. The diagnostic unit has a second output for an error signal and is equipped with a detection unit for checking the plausibility of the output of the position indication of the identified person using a plurality of plausibility criteria, and the diagnostic unit is further designed to detect an error and output an error signal at the second output if at least one plausibility criterion is violated.

[0015] In order to reliably detect errors occurring in camera-based person detectors, in addition to the first component of the person detector, i.e. the camera, and the second component, i.e. the evaluation unit, which is preferably implemented as an artificial neural network, a third component, i.e. the diagnostic unit, is inserted. The diagnostic unit is designed to check the plausibility of the output of the artificial neural network (evaluation unit). Preferably, the diagnostic unit is implemented in safety technology, for example using a programmable logic F controller designed to be fail-safe, so that it can be assumed that all errors of the diagnostic unit itself are also controlled.

[0016] The following plausibility checks can be performed on the diagnostic unit, for example:

[0017] a) Identify people who appear or disappear in the middle of the image,

[0018] b) sudden position changes,

[0019] c) Personnel speed greater than 2m / s,

[0020] d) persons in areas with restricted access (e.g. behind fences),

[0021] e) Personnel in physically impossible positions (e.g. suspended in mid-air).

[0022] These checks take advantage of the fact that artificial neural networks operate in a stateless manner and analyze the images individually, frame by frame. Therefore, in many cases, errors in the artificial neural network or in the camera can be identified by providing results that become inconsistent in the sequence.

[0023] If at least one plausibility check fails, the diagnostic unit has identified an error. The diagnostic unit can then decide whether to tolerate the error (for example, if a person is missing in exactly one image but is immediately detected again at the same location) or to send a danger signal of a detected fault to downstream decision processes.

[0024] In further exemplary embodiments, the detection of errors is improved because the diagnostic unit is also designed to receive and evaluate additional information and to include the additional information in the plausibility check.

[0025] According to the invention, the monitoring system is designed to have an allocation unit, which is designed to receive control signals from the diagnosis unit in combination with a first person detector and a second person detector, wherein the first person detector is designed to have a first imaging sensor, preferably a first camera for recording digital image files or a first laser sensor system for recording point clouds, and a first evaluation unit, and the second person detector is designed to have a second imaging sensor, preferably a second camera for recording digital image files or a second laser sensor system for recording point clouds, and a second evaluation unit, and the diagnosis unit is designed in this case to generate control signals so that the control signals from the first imaging sensor are transmitted to the diagnosis unit. The first imaging sensor is preferably a first camera or a first laser sensor system, or is assigned from a second imaging sensor to the first evaluation unit or to the second evaluation unit, the second imaging sensor is preferably a second camera or a second laser sensor system, the assignment being specified by a control signal, and in addition, the diagnostic unit is designed to have, after having an expected change in the assignment, use this assignment to generate fourth additional information, because the diagnostic unit is designed so that after each change of the assignment, the expectation from the recognized position indication of the previous assignment is compared with the recognized position indication from the already existing assignment, and if the expectations do not match, an error signal is output.

[0026] For example, the diagnostic unit can use additional information about the number of people located in the monitoring area in order to possibly use a plausibility check to identify errors. For example, the people at the entrance can be counted by a turnstile. Therefore, the diagnostic unit has an expectation, which is used as an additional diagnosis. For example, if two people are located in the monitoring area, the personnel detector must also always report exactly two people, otherwise a dangerous signal of a malfunction of the function of the personnel detector can be issued. For larger monitoring areas (for example, the field of view of an entire factory hall or an area blocked by larger objects), multiple personnel detectors must be connected, which cover the entire area as a whole. Then, a common diagnostic unit can be used for all personnel detectors. In this case, the common diagnostic unit compares the total number of people identified with the known number of people. Therefore, the diagnostic unit is designed to evaluate the number of people as the first additional information, which is located in the monitoring area to be monitored.

[0027] The monitoring system provides a further person detector, wherein the diagnosis unit is designed to use a cross comparison between a first channel from the first person detector and a second channel from the second person detector as second additional information for the position data, wherein the channels are designed to be diverse with respect to one another.

[0028] Using this technical solution, random hardware errors in individual components (e.g., cameras, or artificial intelligence hardware) can be identified by using multiple identification channels and cross-comparison in the diagnostic unit. The cross-comparison checks whether the relevant channels provide the same number of people at the same location. Otherwise, a malfunction of the person detector is signaled.

[0029] Advantageously, the individual channels have the highest possible diversity. For example:

[0030] a) The cameras of each channel observe the scene from different perspectives, for example, from the side (wall mounted) and from above (ceiling mounted).

[0031] b) Artificial neural network hardware and cameras from different manufacturers.

[0032] c) The model running on the artificial neural network is trained using different training data and tested.

[0033] d) Models running on artificial neural networks for object recognition are different (e.g., YOLO, compared to FairMOT).

[0034] e) Use different measurement principles, e.g., RGB cameras compared to infrared cameras, event cameras compared to frame cameras, cameras compared to radar, etc.

[0035] Due to the diversity, not only random hardware errors but also systematic errors can be identified.

[0036] A further degree of security is possible in that the diagnosis unit is designed to generate a phantom image, in which case the person detector has an image blending unit which is designed to superimpose the phantom image into the image file, whereby the current camera image is overlapped by the phantom image, thereby enabling evaluation of a third additional information.

[0037] For each individual channel, the correct functioning of the evaluation unit (artificial intelligence, artificial neural network, artificial neural network hardware) and the model configured therein is checked by feeding a so-called phantom in running operation. The phantom can be generated preferably on the diagnostic unit and initially consists of the position and, if necessary, other criteria such as the type of person (e.g. male, female, diverse), clothing (lab coat, work clothes), equipment carried (e.g. hard hat, cart) and the activity performed (e.g. walking, standing, kneeling). This information is sent from the diagnostic unit to a phantom image generator, which generates a realistic image (2D graphic) therefrom. This image, the phantom image, is superimposed onto the actual camera image of the channel using an image blending unit, that is, the camera image is partially covered by the phantom image. The artificial neural network associated with the channel should be able to correctly recognize this phantom, which can be reliably detected by the monitoring unit or diagnostic unit as additional information. If the phantom or phantom image cannot be recognized, a fault in the person detector can again be signaled.

[0038] Advantageously, the position of the phantom in the phantom image and the additional criteria are changed regularly (e.g. a different phantom can be used for each image, or the phantom is changed after fixed time intervals, e.g. every 5 seconds). This allows different aspects of the recognition of the person detector to be checked regularly (e.g. at least once an hour to check whether the person detector has recognized a man in work clothes who is wearing a helmet and working on the ground).

[0039] In a further advantageous embodiment, it is possible to randomly combine different criteria during operation. This is particularly useful when the number of combinations is so large that a systematic generation of all combinations is not feasible.

[0040] Another advantageous embodiment is that during the run time different criteria are combined so that at least all possible pairs of two criteria (e.g. person type and clothing or clothing and carried equipment) occur at least once per time unit. The remaining criteria are then randomly selected again.

[0041] Another possibility is that the diagnosis unit or the phantom image generator generates phantoms only at locations where no real person is present, in order to influence the recognition quality of the real person as little as possible.

[0042] For example, the allocation unit can use a round-robin method, in which case possible errors between cameras and artificial neural networks are revealed. The principle of the round-robin method is that the allocation between cameras and artificial neural networks is not determined in a static manner, but is changed regularly according to the requirements of the diagnosis unit. For example, with each picture or after a predefined time interval. With each change, the diagnosis unit has a fixed expectation from the previous cycle, which was determined in the corresponding other artificial neural network. If the expectation does not correspond to the output of the person detector, a fault signal can be issued.

[0043] Since it is possible to detect other errors by checking the round robin method by the diagnostic unit, since the diagnostic unit is again implemented in safe F-technology according to IEC Standard 61508, it does not happen that the round robin method freezes due to an error and is no longer ineffective.

[0044] A further level of safety is achieved if an imaging sensor, preferably a camera or laser sensor system, an actuator is configured in the system and the diagnostic unit is configured to drive the actuator via control commands so that the laser scanning area of ​​a part, preferably a camera part or a monitored area, is changed by a predetermined range, wherein this change can be evaluated in the diagnostic unit as fifth additional information.

[0045] The actuator can then be used to regularly change the camera section. For example, the motor can also change the focal length, the line of sight or the position of the camera. The changes should be small so that the camera always has the area to be monitored within the image range in all cases. For example, if a person is standing in the same position, the person detector still has to provide a slightly different position depending on the current camera section. This can be evaluated as a fifth additional information. In this way, errors can be reliably detected, for example, the camera image is frozen in the event of an error.

[0046] The diagnostic unit is responsible for controlling the current monitoring area or image area. It selects the viewing angle, sends corresponding signals to the actuators and is able to correct the position provided by the person detector accordingly.

[0047] In order to check the timeliness of the monitoring system, the monitoring system is equipped with a display device in the monitoring area, wherein the display device is arranged so that an imaging sensor, preferably a camera or a laser sensor system, can detect notifications from the display device, and the diagnostic unit is also configured to output a symbol sequence as a notification on the display device, wherein the evaluation unit is configured to recognize the symbol sequence and notify the diagnostic unit as the sixth additional information.

[0048] The diagnostic unit can, for example, cause a symbol sequence to be displayed (e.g., a three-digit number), which is displayed on the display device and can be detected by the camera, and which must be recognized in the evaluation unit by the artificial neural network. In this way, the diagnostic unit can check whether the person detector is actually operating with real image data.

[0049] In a further advantageous embodiment, the diagnostic unit causes the displayed symbol sequence to be changed regularly. This makes it possible to check whether the person detector is operating with the current image information. In addition, in this way, the reaction time of the person detector can be continuously checked during the operating time. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings show a number of embodiments and designs of the present invention.

[0051] Figure 1 shows a monitoring system with three main components,

[0052] Figure 2 shows a monitoring system in a two-channel design with cross comparison,

[0053] Figure 3 A monitoring system showing the feasibility of having input phantom images,

[0054] Figure 4 shows a monitoring system using a round-robin method,

[0055] Figure 5 A monitoring system is shown which changes the camera settings via a motor.

[0056] Figure 6 shows a monitoring system with superposition of additional symbol sequences,

[0057] Figure 7 A diagnostic unit is shown with a detection unit and additional information to be evaluated. DETAILED DESCRIPTION

[0058] according to Figure 1 , the monitoring system 1 is divided into three main components. The first component E1 forms a first camera 3a, the second component E2 forms a first evaluation unit 5a and the third component E3 forms a diagnostic unit 6. The monitoring system 1 is therefore equipped with a first person detector 2a. The person detector 2a has a first camera 3a for recording a digital image file D1a. Optionally, the person detector 2a can also have a laser sensor system 4a for recording a point cloud D2a. The first evaluation unit 5a is designed to evaluate the image file D1a or the point cloud D2a and output a position indication P of a person M identified in the monitoring area FOV.

[0059] A monitoring system 1 comprising a first component E1 and a second component E2 is known from the prior art. According to the invention, a third component E3 as a diagnostic unit 6 is now added to the monitoring system 1. The diagnostic unit 6 checks the plausibility of the output of the evaluation unit 5a. The evaluation unit 5a is designed as an artificial neural network (German abbreviation: KNN), and the artificial neural network advantageously operates in a stateless manner, which is used for a subsequent plausibility check. In order to receive data, the diagnostic unit 6 has a first input channel 7a. A position indication P is received via the input channel 7a. At a first output 8, the diagnostic unit 6 can output a confirmation signal 9 of the position indication P of the identified person M. At a second output 10 for an error signal 11, the diagnostic unit 6 can output an error signal 11. The diagnostic unit 6 is equipped with a detection unit 12, which checks the plausibility of the output of the position indication P of the identified person M using a plurality of plausibility criteria C1, ..., Ci. In addition, the diagnostic unit 6 is equipped with a detection unit 12, so that if at least one plausibility criterion C1, ..., Ci is violated, an error is detected and an error signal 11 is output on the second output 10.

[0060] The diagnostic unit 6 or the detection unit 12 is also equipped to receive and evaluate the additional information Z1, ..., Zi and to include the additional information in the plausibility check. The first additional information Z1 is provided via the turnstile DK of the diagnostic unit 6. The turnstile DK determines the number of persons located in the area to be monitored. For example, if two persons are located in the area to be monitored, the person detector 2a must also always report exactly two persons. The diagnostic unit 6 can then use this first additional information Z1 to determine an error.

[0061] For better decision making, a decision unit 20 can also be arranged downstream of the monitoring system 1. The decision unit 20 receives the confirmation signal 9 and the error signal 11. If the diagnosis unit 6 recognizes an error, the decision unit 20 can decide to first tolerate the error, for example, if a person is missing in exactly one image, but is immediately detected again at the same position in the next image. Otherwise, the downstream decision unit 20 can also signal a dangerous malfunction of the person detector.

[0062] With the help of Figure 2 The feasibility of using the second additional information Z2 about the cross comparison between the first channel K1 of the first person detector 2a and the second channel 2 of the second person detector 2a is shown. The first channel K1 and the second channel K2 are designed differently from each other. The cross comparison device KV constantly compares the first position data Pa and the second position data Pb and generates the second additional information Z2 from this. Random hardware errors in the individual components E1, E2 can then be discovered.

[0063] Figure 3The feasibility of generating the third additional information Z3 by forming ghost images G1, G2 is shown. For this purpose, the person detector 2a has an image mixing unit 30, which is designed to superimpose the ghost images G1, G2 into the image file D1a, thereby superimposing the current camera image by the ghost images G1, G2, so that the third additional information Z3 can be evaluated in the diagnosis unit 6. For this purpose, the diagnosis unit 6 has a ghost image generator PBG. The ghost image generator PBG generates, for example, a first ghost image G1 and a second ghost image G2. The ghost images G1, G2 are processed by the playback unit 31 so that the ghost images can be processed accordingly in the image mixing unit 30.

[0064] Figure 4 The feasibility of further improving the detection error is shown. Figure 2 , the monitoring system 1 with the first channel K1 and the second channel K2 now has an allocation unit 40. The allocation unit 40 is designed to receive a control signal C from the diagnosis unit 6 in a combination 41 of a first person detector 2a and a second person detector 2b, wherein the first person detector 2a is equipped with a first camera 3a for recording a digital image file D1a and is equipped with a first evaluation unit 5a. The second person detector 2b is equipped with a second camera 3b and a second evaluation unit 5b. The diagnosis unit 6 is designed to generate a control signal C so that an allocation ASM is given from the first camera 3 to the first evaluation unit 5a or to the second evaluation unit 5b. The allocation ASM can also include the allocation of the second camera 3b to the first evaluation unit 5a. Therefore, the allocation ASM is specified by the control signal C generated in the diagnosis unit 6. The diagnosis unit 6 is designed to be equipped with this specification, which uses the allocation ASM to generate the fourth additional information Z4 with the expectation change. The diagnosis unit 6 is designed so that after each change of the allocation ASM, the expectation of the recognized position input P from the previous allocation ASM can be compared with the recognized position input P of the currently existing allocation ASM. If the expectations do not match, an error signal 11 is output.

[0065] Figure 5 Further measures for improving error detection are shown. The monitoring system 1 now has an actuator M1 which is designed to influence the camera 3a so that the camera section 52 or the laser scanning area 53 of the monitoring area FOV changes by a predetermined range. For example, the first monitoring area FOV1 and the second monitoring area FOV2 can be expanded. The diagnosis unit 6 is designed to drive the actuator M1 by means of the control command 50 so that the camera section changes by a predetermined range, wherein this change can be evaluated in the diagnosis unit 6 as fifth additional information Z5.

[0066] With the help of Figure 6A further measure is shown. In the monitoring area FOV, the display device 60 is arranged so that the camera 3a records the notification 61 of the display device 60. The diagnostic unit 60 is also designed to output the symbol sequence 62 as the notification 61 on the display device 60. The evaluation unit 5 is designed to recognize the symbol sequence 62 "735" and to notify the diagnostic unit 6 of the symbol sequence as the sixth additional information Z6.

[0067] In summary, Figure 7 , a diagnostic unit 6 is shown which evaluates all plausibility checks C1 , . . . , Cn and all additional information Z1 , . . . , Z6 via a detection unit 12 .

Claims

1. A monitoring system (1) comprising a first person detector (2a), wherein the first person detector is configured as follows: - having an imaging sensor for recording a digital image file (D1a) or a point cloud (D2a), and - having an evaluation unit (5a) which is designed to evaluate the image file (D1a) or the point cloud (D2a) and to output a position indication (P) of a person (M) identified in the monitoring area (FOV), - also has a diagnostic unit (6) having a first input channel (7a) for receiving the position indication (P) and a first output (8) for the checked position indication (P) of the identified person (M) and a second output (10) for an error signal (11), wherein The diagnostic unit (6) is designed to have a detection unit (12) for checking the output of the position indication (P) of the identified person (M) for plausibility using a plurality of plausibility criteria (C1, . . . , Ci), the diagnostic unit being further designed to detect an error if at least one plausibility criterion (C1, . . . , Ci) is violated and to output the error signal (11) at the second output (10), wherein the diagnostic unit (6) is also designed to receive and evaluate additional information (Z1, . . . , Zi) and include the additional information in a plausibility check, and the first person detector also has a further person detector (2b), wherein the diagnostic unit (6) is designed to perform a cross comparison between a first channel (K1) from the first person detector (2a) and a second channel (K2) from the second person detector (2b) as additional information (Z2) for the position data (P), wherein the channels (K1, K2) are designed to be different from one another, It is characterized in that A distribution unit (40) is provided, which is designed to receive a control signal (C) from the diagnosis unit (6) into a combination (41) of the first person detector (2a) and the second person detector (2b), wherein: the first person detector (2a) is designed with a first imaging sensor for recording a digital image file (D1a) or a point cloud (D2a) and with a first evaluation unit (5a), the second person detector (2a) is designed with a second imaging sensor for recording a digital image file (D1a) or a point cloud (D2b) and with a second evaluation unit (5b), The diagnostic unit (6) is designed to generate the control signal (C) so that the control signal (C) specifies an assignment (ASM) of the first imaging sensor to the first evaluation unit (5a) or to the second evaluation unit (5a) or an assignment (ASM) of the second imaging sensor to the first evaluation unit (5a) or to the second evaluation unit (5a), and the diagnostic unit is also designed to generate fourth additional information using the assignment after changing the assignment (ASM) with the expectation, because the diagnostic unit (6) is designed to compare the expectation of the recognized position indication (P) from the previous assignment (ASM) with the recognized position indication (P) from the currently existing assignment (ASM) after each change of the assignment (ASM) and to output the error signal (11) if the expectations do not match.

2. The monitoring system (1) according to claim 1, wherein: The diagnosis unit (6) is designed to evaluate the number of persons located in the monitored monitoring area (FOV) as first additional information (Z1).

3. The monitoring system (1) according to any one of claims 1 or 2, wherein: The diagnosis unit (6) is designed to generate a ghost image (G1, G2), and the person detector (2) has an image blending unit (30) when generating the ghost image, which is designed to superimpose the ghost image (G1, G2) into the image file (D1a) so that the ghost image (G1, G2) overlaps with the current camera image, thereby enabling the evaluation of further additional information (Z3).

4. The monitoring system (1) according to any one of claims 1 to 3, wherein: The imaging sensor is designed to have an actuator (M1), and the diagnostic unit (6) is designed to control the actuator (M1) via a control command (50) so that the part of the monitoring area (FOV) is changed by a predetermined range, wherein the change can be evaluated in the diagnostic unit (6) as fifth additional information (Z5).

5. The monitoring system (1) according to any one of claims 1 to 4, wherein: The display device (60) is arranged in the monitoring area (FOV) so that the imaging sensor detects the notification (61) of the display device (60), and the diagnostic unit (6) is also designed to output a symbol sequence (62) as a notification (61) on the display device (60), wherein the evaluation unit (5) is designed to recognize the symbol sequence (62) and to notify the diagnostic unit (6) of the symbol sequence as sixth additional information (Z6).

6. The monitoring system (1) according to any one of claims 1 to 5, characterized in that: The diagnostic unit (6) is designed as an industrial controller and thus as a controller designed for functional safety, which is certified according to the IEC 61508 standard.

7. The monitoring system (1) according to any one of claims 1 to 6, wherein: The diagnostic unit is designed to perform the following plausibility checks: a) Identify people who appear or disappear in the center of the image, b) identify sudden position changes, c) Identify personnel speeds greater than 2m / s, d) identify persons in areas with restricted access, and e) Identify persons in locations that are physically impossible.

Images