System for monitoring a surveillance area

Abstract

System for monitoring a surveillance area (6), with a plurality of imaging sensors for detecting a movement behavior of at least one movable object (2, 4) in the surveillance area (6), an evaluation device (40) for calculating a trajectory (5) of the at least one movable object (2, 4) as a function of the detected movement behavior and an interface (210) for making the calculated trajectory (5) available for controlling an operating sequence in the monitoring area (6), wherein at least one imaging sensor for detecting the movement behavior of the at least one movable object (2, 4) is arranged on a mobile sensor carrier, characterized in that the mobile sensor carrier is designed to carry the at least one imaging sensor and the movable object (2, 4).

Description

Technical area

[0001] The invention relates to a system for monitoring a surveillance area and a corresponding method. In particular, the invention relates to a system and a method in which multiple imaging sensors are used to detect the movement behavior of at least one movable object in the surveillance area. State of the art

[0002] Systems with imaging sensors for detecting objects in a surveillance area are known, whereby such systems can be used to determine the locations of objects.

[0003] US 2015 / 0 012 396 A1 describes a stationary image capture system for monitoring inventory. The image capture system determines whether a warehouse worker has removed a stored object from a monitored storage location or placed an object in a storage location. Operationally relevant inventory turnover can thus be temporarily detected and evaluated.

[0004] However, operational processes are not only influenced or coordinated by the operationally relevant locations of objects, but are also dynamically influenced by object movements within an area relevant to the operational processes. Summary of the invention

[0005] The invention therefore provides solutions to efficiently detect object movements in an area to be monitored so that operational processes can be dynamically optimized.

[0006] Such a solution for a system for monitoring a surveillance area comprises a plurality of imaging sensors for detecting a movement behavior of at least one movable object in the surveillance area, an evaluation device for calculating a trajectory of the at least one movable object depending on the detected movement behavior and an interface for making the calculated trajectory available for controlling an operating sequence in the surveillance area, wherein at least one imaging sensor for detecting the movement behavior of the at least one movable object is arranged on a mobile sensor carrier.

[0007] The system can be configured to combine the calculated object trajectory with static and georeferenced map data to provide a dynamic map or a dynamic geographic information system for controlling an operational process. For this purpose, the calculated object trajectory can be transformed into a coordinate system of the map or geographic information system.

[0008] A surveillance area can be a factory, a production facility or a warehouse, each of which can be found in a hall.

[0009] An object located in the surveillance area and to be monitored can be an object, a material, or a person. The movement behavior of the object can be a movement of the object, a movement of material, a movement of a person, or a corresponding flow or current. The movement behavior can include object positions, an object speed, an object acceleration, and / or a residence time of the object at an object position.

[0010] An imaging sensor can be any sensor configured to capture two-dimensional or three-dimensional image information or image data. The captured image information can be pixel- or voxel-based information, which can include coordinates on the one hand and color-coded information on the other. The imaging sensor can also be a photogrammetric sensor, in particular a camera, an optical sensor, and / or a scanning sensor. In addition to one or more mobile imaging sensors, at least one stationary imaging sensor can be provided.

[0011] The mobile sensor carrier can be a vehicle, a system, a robot, or a carrier that can be carried by a person. The vehicle can be a person-driven or autonomous vehicle, such as an industrial truck. The system can be a stationary or mobile operating system, such as a crane.

[0012] The object's trajectory can contain location-related information that allows it to describe an object's movement or a change in its position. Two positions or a single vector can describe the object's movement. The trajectory can also contain or be based on multiple object positions, whereby the individual positions can be connected linearly or curvedly, or determined by interpolation.

[0013] The interface can be provided for data transfer of the calculated trajectory, whereby the location-related and / or geometric information of the trajectory can be taken into account when controlling an operational process. The operational process to be controlled can, for example, be material-based, logistical, production-dependent, building technology, or personnel processes for the operation of a production facility, factory, or warehouse.

[0014] The trajectory provided at the interface has the effect that, when controlling an operational process, not only static object information but also dynamic object information can be accessed. This allows past, current, or future object movements to be taken into account and thus optimize the operational process. The dynamic object information can include temporal and kinematic information. Thus, the movement behavior of monitored objects can be taken into account in an existing material flow system or logistics system, in a production control system, in a building control system, or in a personnel management system.

[0015] A core concept of the invention can be seen in the fact that movements of objects are detected with a plurality of imaging sensors, i.e., with at least two such sensors, which are sometimes or predominantly arranged on mobile sensor carriers, in order to realize object monitoring with high monitoring coverage in near real time. The provision of a mobile imaging sensor can be advantageous for essentially gapless area monitoring, since the detection range of such a sensor can be more variable and larger than that of a stationary imaging sensor, thus reducing the required number of imaging sensors.

[0016] The invention is also based on the concept that the movement behavior of dynamic objects can be continuously monitored or tracked by using mobile image acquisition systems, without the dynamic objects having to be equipped with their own sensors or object components that actively communicate with a sensor system.

[0017] In one embodiment, the plurality of imaging sensors comprises a plurality of cameras for continuously capturing image information, wherein at least one of the plurality of cameras is arranged on a mobile camera carrier. A camera can be a digital camera for capturing static or moving images, which can be a video camera. Continuously captured image information can be individual images captured at specific time intervals. The object to be monitored can be depicted in the image information of the individual images, wherein an object movement can be detected and calculated from a comparison of several individual images using a difference image. To calculate relative or absolute object coordinates or object movements, it may be necessary to determine a position, an orientation, and a calibration of the camera.Continuous determination of a camera position and orientation may be necessary if the camera is located on a mobile camera mount. Furthermore, it may be necessary to transform an object motion in image coordinates into an absolute object motion in a higher-level coordinate system.

[0018] The object to be monitored can be detected in the captured image information using known automatic image analysis methods, whereby matching with known templates can be performed to monitor the movement behavior of predefined objects. An automatically detected object can thus be assigned to a predefined or known object.

[0019] In a further embodiment, the mobile sensor carrier is designed to carry the at least one imaging sensor and the movable object. The mobile sensor carrier can also be a mobile object carrier. If the movable object is located on the mobile sensor carrier, whereby the movable object can be carried and moved by the mobile sensor carrier, the imaging sensor can continuously capture image information of the object's surroundings. The object itself may not be represented in the image information. The movement of the mobile sensor carrier can be calculated from several individual images of the surroundings and from a comparison of the individual images, from which the trajectory of the object moved by the sensor carrier can also be calculated. The trajectory of the object can thus be determined indirectly from image information of its surroundings, which is captured from different object positions.

[0020] In a further embodiment, the mobile sensor carrier is an autonomous vehicle. An environment detection system can be arranged on the autonomous vehicle, which can have at least one mobile imaging sensor for detecting the movement behavior of the object. The autonomous vehicle can be a surveillance vehicle intended to monitor the surveillance area. Furthermore, the autonomous vehicle can be a vehicle intended to transport objects in the surveillance area. Alternatively or additionally, the vehicle can also perform other tasks, in particular tasks related to the operational sequence to be controlled in the surveillance area.Based on the calculated trajectory of objects in the vehicle's surroundings, it is also possible to intervene in the vehicle's control and, for example, to stop or modify a vehicle movement in order to avoid a collision with the monitored object.

[0021] In another embodiment, the mobile sensor carrier can be carried by a person. The person can carry a wearable camera on a piece of clothing, a piece of work equipment, or a helmet. The camera can be a body cam, which can be mounted on the body or on a piece of work equipment worn by the person. Furthermore, an object to be monitored can also be moved by a person within the monitoring area.

[0022] In a further embodiment, the evaluation device is configured to transform the calculated trajectory into a dynamic map, and an information system based on the dynamic map has the interface configured to make the transformed trajectory available for controlling an operational sequence in the monitored area. The dynamic map can be based on a static map, which can be overlaid with dynamic map information. Such a dynamic map can contain and display information about the movement behavior of the objects to be monitored. The monitored objects can thus be represented as dynamic objects in the map. With the trajectory transformed into the map, past, current, and future location information of the objects can be visualized and analyzed.

[0023] Images captured by the imaging sensors can also be used to generate static map data for the dynamic map. For this purpose, several captured individual images can be fused, i.e., combined with each other. Combining multiple individual images can be achieved using image stitching. The result can be a photorealistic representation or map of the surveillance area, in which moving objects can be dynamically displayed.

[0024] A dynamic map-based information system can be a standalone dynamic geographic information system (GIS) or one associated with a system for controlling a production process. This system can also maintain dynamic map data on object movements based on georeferenced static map data. This could, for example, be a factory information system that can provide movement data on materials being processed and workers independently or to other factory control systems, such as a shop floor management system.

[0025] The calculated trajectory can, for example, also be directly integrated into the control system of an automated guided vehicle (AGV). This has the advantageous effect that monitoring object movements and their trajectories during operation of the AGV can prevent collisions between the transport systems and the monitored objects and enable route replanning.

[0026] In a further embodiment, at least one imaging sensor of the plurality of imaging sensors is located in a sub-area of the surveillance zone, which represents the potential location of the object depending on a modeled movement behavior of the object. For the presence of the sensor in the sub-area, the mobile sensor carrier, on which the sensor can be arranged, can be moved into the sub-area for this purpose. The likely future movement behavior of the object can be predicted by extrapolating a previously calculated trajectory or by probability calculations. For this purpose, the movement behavior of objects can also be learned and mapped using physical or static models or using a neural network.

[0027] In a further embodiment, a data source for metadata of the movable object is provided, wherein the metadata can be assigned to the trajectory of the object. The metadata can include additional object data or data about the trajectory of the object.

[0028] In a further embodiment, the plurality of imaging sensors are located in the surroundings of the movable object. All or some of the imaging sensors can be located within the monitoring area. In other words, no imaging sensor or element actively communicating with a sensor can be located on the object to be monitored. The object can thus behave passively when recording its movement behavior. It does not actively participate in the recording of its movement behavior by directly providing information.

[0029] In a further embodiment, the mobile sensor carrier has additional non-imaging sensors for detecting the movement behavior of the movable object. If the object to be monitored is located on the sensor carrier, a trajectory of the object can optionally be determined for image-based evaluation by sensors on the mobile sensor carrier, which detect a position, orientation, distance, speed, or acceleration of the mobile sensor carrier. The trajectory of the mobile sensor carrier can thus also be used to infer the trajectory of the object to be monitored if the object is moved by the mobile sensor carrier.

[0030] Alternatively or additionally, several reference points can be accessed to determine the position of the mobile sensor carrier, to which the distance from the sensor carrier can be determined by measurement in order to determine the position, for example by means of a spatial backward cut.

[0031] A further solution is a method for monitoring a surveillance area, which comprises detecting a movement behavior of at least one movable object in the surveillance area with a plurality of imaging sensors, calculating a trajectory of the at least one movable object depending on the detected movement behavior and making the calculated trajectory available for controlling an operating sequence in the surveillance area, wherein the detection of the movement behavior of the at least one movable object is carried out by at least one imaging sensor which is arranged on a mobile sensor carrier.

[0032] The invention and its embodiments are further explained in the accompanying schematic figures. Brief description of the drawings Fig. 1 shows an embodiment of the system for monitoring a surveillance area. Fig. 2 is a diagram for explaining a detection of a movement behavior of an object with an environment detection system of a vehicle when monitoring the surveillance area. Fig. 3 is another illustration for explaining a detection of a movement behavior of an object with an environment detection system of a vehicle when monitoring the surveillance area. Fig. 4 is a diagram for explaining a detection of a movement behavior of an object moved by a vehicle with an environment detection system of the vehicle when monitoring the monitoring area. Fig. 5 is another illustration for explaining a detection of a movement behavior of an object moved by a vehicle with an environment detection system of the vehicle when monitoring the surveillance area. Fig. 6 is a diagram illustrating continuous environment detection along a vehicle trajectory. Fig. 7 shows a flowchart of method steps of an embodiment of a method for monitoring a surveillance area. Detailed description of embodiments

[0033] In Fig. 1 shows a surveillance system 200 for monitoring an object 4 in a surveillance area 6. System components for object surveillance include a plurality of cameras 10, 20 located in the surveillance area 6, an autonomous vehicle 22 as a mobile camera carrier for a mobile camera 20, an evaluation device 40 for calculating an object trajectory from image information captured by the cameras 10, 20, and interfaces 210 for providing the calculated object trajectory to further systems 212, 214, 216, 218.

[0034] In the surveillance area 6, in addition to the mobile camera 20 arranged on the autonomous vehicle 22, further mobile cameras 20 are provided on a person 24 and a mobile camera 20 on an overhead crane as a mobile system 28. The mobile cameras 20 on the person 24 are provided as a body cam on the body and as a helmet cam on the helmet. In addition to the mobile cameras 20, a stationary camera 10 is also permanently arranged in the surveillance area 6. The cameras 10, 20 capture image information 8 in their respective detection area, which is determined by the fixed viewing angle of the stationary camera 10 and the variable viewing angles of the mobile cameras 20.

[0035] The cameras 10, 20 are connected to the interfaces 210 via a stationary transmitting and receiving unit 34, either wirelessly via WLAN or wired via LAN, and communicate with them accordingly (some connections not shown). The mobile camera 20 on the autonomous vehicle 22 is connected to a mobile transmitting and receiving unit 32 on the autonomous vehicle 22, which is wirelessly connected to the stationary transmitting and receiving unit 34. Image information 8 captured by the cameras 10, 20 is thus transmitted to the evaluation unit 40 via the transmitting and receiving units 32, 34.

[0036] The monitored objects 4 are Fig. 1 shows objects 4 transported on the autonomous vehicle 22, the movement behavior of which can be calculated at least from the image information 8 captured by the camera 20 arranged on the autonomous vehicle 22, in which a trajectory 5 of the autonomous vehicle 22 is calculated using the captured image information 8 and / or using an environment detection system 30, which has at least one further environment detection sensor 36. The at least one further environment detection sensor 36 can be designed, for example, as a radar sensor, lidar sensor, infrared sensor, or ultrasonic sensor. The calculation of the trajectory 5 is carried out using the evaluation device 40.

[0037] The calculated trajectory 5 is provided via the interfaces 210 to the systems 212, 214, 216, 218, which system users 202 access to obtain information about the movement behavior of the objects 4 or to further evaluate the movement behavior of the objects 4. Further systems provided include a logistics system 212, a production control system 214, a control system for driverless transport systems 216, and / or a facility management system 218.

[0038] The other systems incorporate the trajectory 5 of the objects 4 and the autonomous vehicle 22 calculated by the evaluation unit 40 into operational processes controlled by them. Controllable operational processes of the logistics system 212 include managing logistics orders, controlling the autonomous vehicle 22, recording the status of the objects 4 along a logistics chain, and / or recording the input and output of objects 4 or people 24. Controllable operational processes of the production control system 214 include controlling machine tools or other systems and / or performing a quality inspection with regard to objects 4 to be processed. Controllable operational processes of the control system for driverless transport systems 216 include navigating autonomous vehicles 22 and loading and unloading the autonomous vehicles 22 with objects 4, whereby collision avoidance with people 24 is also controllable.Controllable operating processes of the facility management system 218 include controlling ventilation, air conditioning, lighting, access control and time recording depending on the monitored movement behavior of objects 4 in the monitoring area 6.

[0039] In the Fig. 2 to 5, a dynamic map 100 of a warehouse is shown as a monitoring area 6. The dynamic map has a static mapping 110 of the warehouse. In the monitoring area 6, there is an object 2, for example, a worker or a pallet, and a warehouse object 4 whose movement behavior is monitored by the monitoring system 200.

[0040] As in Fig. 2, image information 8 is captured in the surroundings of the object 2, wherein the object 2 is depicted in the captured image information 8. The captured image information 8 is captured, for example, with the mobile camera 20 on the autonomous vehicle 22. As shown in Fig. 3, further image information 8 is captured in the surroundings of the object 2, wherein the object 2 is in turn represented in the captured image information 8. A trajectory 5 as the movement behavior of the object 2 is calculated from the image information 8 and represented as dynamic map information 120 in the dynamic map 100.

[0041] If the object 2 is no longer within the detection range of the mobile camera 20 of the autonomous vehicle 22 for a predetermined period of time, the last detected position of the object 2, as in Fig. 4, as static map information 122. According to Fig. 4, the storage object 4 has already been removed from its storage location and placed on the autonomous vehicle 22. Image information 8 is then captured in the surroundings of the storage object 4, wherein the captured image information 8 does not depict the storage object 4 itself, but rather its surroundings. The captured image information 8 is captured with the mobile camera 20 on the autonomous vehicle 22 after the storage object 4 is located on the autonomous vehicle 22 and is being moved by it. However, if the autonomous vehicle 22 is still moving towards the storage object 4, the storage object 4 is captured by the mobile camera 20 of the autonomous vehicle 22. The captured image information 8 also captures an identification of the storage object 4 attached to the storage object 4, such as a label, a number, or a barcode, via which further object data or object information of the storage object 4 becomes available.These object data or object information identifying storage object 4 can be transmitted to the evaluation unit 40 and checked for consistency by comparing them with existing object data or object information in the systems 212, 214, 216, 218. If there is a discrepancy between the recorded and the existing object data or object information, the object data available in the systems 212, 214, 216, 218 can be updated or supplemented accordingly via the interfaces 210.

[0042] In Fig. 5, further image information 8 of the surroundings of the storage object 4 is captured, wherein the object 4 is again not represented in the captured image information 8. A trajectory 5 as the movement behavior of the storage object 4 is calculated from the image information 8 of the surroundings along the trajectory 5 traveled by the autonomous vehicle 22 and represented as dynamic map information 120 in the dynamic map 100.

[0043] One of the systems 212, 214, 216, 218 can thus be informed about the current positions and trajectories 5 of the objects 2, 4, whereby, for example, an arrival time of one of the objects 2, 4 or a location of the objects 2, 4 is used to control an operating process.

[0044] In Fig. 6 shows continuous environmental detection along a trajectory 5 of an autonomous vehicle 22. An environmental detection system 30, which has a plurality of cameras 20, captures environmental images 60, which are merged into an overall image 62 using features 61 detected in the environmental images. The overall image 62 can be used as static mapping 110 and / or to calculate dynamic map information 120 in the dynamic map 100. The trajectory 5 of the autonomous vehicle 22 or of objects 4 located on the autonomous vehicle 22 can also be calculated from the fusion of the environmental images 60. The trajectory 5 can optionally also be calculated from further sensors (not shown), for example an odometer or acceleration sensors, on the autonomous vehicle 22, or the derivation of the trajectory 5 from the environmental images 60 can be supported with measurement data from such sensors, for example by means of SLAM methods.

[0045] With the Fig. 1 and the monitoring system 200 shown in the Fig. The movement behavior of objects 2, 4 is calculated using the automatic environment detection methods shown in Figures 2 to 6 and provided to further systems 212, 214, 216, 218 for controlling operational processes in a monitoring area 6.

[0046] In Fig. 7, in a flowchart for monitoring a surveillance area 6, the method steps are shown as a first step S1, which comprises detecting a movement behavior of an object 2, 4, a second step S2, which comprises calculating a trajectory 5 of the object 2, 4, and a third step S3, which comprises making the trajectory 5 available for further systems 212, 214, 216, 218. Reference symbol 2 objects 4 storage object 5 Trajectory 6 Monitoring area 8 Image information 10 stationary cameras 20 mobile cameras 22 autonomous vehicle 24 people 28 mobile system 30 Environment detection system 32 mobile transmitting and receiving units 34 stationary transmitting and receiving units 36 Environment detection sensor 40 Evaluation device 60 surrounding image 61 feature 62 Overall picture 100 dynamic map 110 static mapping 120 dynamic map information 122 static map information 200 surveillance system 202 system users 210 Interface 212 Logistics system 214 Production control system 216 Control system for driverless transport systems 218 Facility management system S1 Capture movement behavior S2 Calculate trajectory S3 Making trajectory available

Claims

[1] System for monitoring a surveillance area (6), with a plurality of imaging sensors for detecting a movement behavior of at least one movable object (2, 4) in the surveillance area (6), an evaluation device (40) for calculating a trajectory (5) of the at least one movable object (2, 4) as a function of the detected movement behavior and an interface (210) for making the calculated trajectory (5) available for controlling an operating sequence in the monitoring area (6), wherein at least one imaging sensor for detecting the movement behavior of the at least one movable object (2, 4) is arranged on a mobile sensor carrier, characterized by , that the mobile sensor carrier is designed to carry the at least one imaging sensor and the movable object (2, 4). [2] System according to claim 1, characterized by , that the plurality of imaging sensors comprises a plurality of cameras (10, 20) for continuously capturing image information and at least one camera (20) of the plurality of cameras (10, 20) is arranged on a mobile camera carrier. [3] System according to one of the preceding claims, characterized by that the mobile sensor carrier is an autonomous vehicle (22). [4] System according to one of the preceding claims, characterized by that the mobile sensor carrier can be carried by a person (24). [5] System according to one of the preceding claims, characterized by , that the evaluation device is arranged to transform the calculated trajectory into a dynamic map (100) and an information system based on the dynamic map (100) has the interface (210) which is configured to make the transformed trajectory available for controlling an operational sequence in the monitoring area. [6] System according to one of the preceding claims, characterized by that at least one imaging sensor of the plurality of imaging sensors is located in a sub-area of the monitoring area which represents a potential location of the object (2, 4) depending on a modeled movement behavior of the object. [7] System according to one of the preceding claims, characterized by that a data source for metadata of the movable object is provided, wherein the metadata can be assigned to the trajectory (5) of the object (2, 4). [8] System according to one of the preceding claims, characterized by that the plurality of imaging sensors are located in the vicinity of the movable object (2, 4). [9] System according to one of the preceding claims, characterized by that the mobile sensor carrier has additional non-imaging sensors for detecting a movement behavior of the movable object (2, 4). [10] Method for monitoring a surveillance area, with the steps Detecting (S1) a movement behavior of at least one movable object (2, 4) in the monitoring area (6) with a plurality of imaging sensors, calculating (S2) a trajectory (5) of the at least one movable object depending on the detected movement behavior and Making available (S3) the calculated trajectory (5) for controlling an operating sequence in the monitoring area, wherein the detection (S1) of the movement behavior of the at least one movable object is carried out by at least one imaging sensor, characterized by , that the imaging sensor in the detection step (S1) is arranged on a mobile sensor carrier which is designed to carry the at least one imaging sensor and the movable object (2, 4).

Images