Method and device for automated monitoring of the driving operation of a passenger transport system

Abstract

The invention relates to a method for the automated monitoring of a driving operation (F) of a passenger transport system (1) by means of a monitoring system (5). The monitoring system comprises a hazard analysis module (7) and a movement detection module (9). The movement detection module is directed towards the passenger transport system and configured for detecting electronically processible, real movement sequence recordings (15) of situations. The method comprises: receiving real movement sequence recordings in the hazard analysis module (7); receiving data (X, Y, Z) of a digital doppelganger (29) of the passenger transport system in the hazard analysis module, wherein the data includes at least information relating to physical properties of the passenger transport system, which allows for a conclusion to be drawn in terms of a visual appearance (30) of the passenger transport system in a predetermined state of movement; determining information relating to dynamic objects (31) at the passenger transport system using the hazard analysis module, wherein the information is determined on the basis of the received data of the digital doppelganger and the real movement sequence recordings; determining information relating to a current hazard situation (101A, 101B, 101C) at the passenger transport system using the hazard analysis module, based on an analysis of movements of the determined dynamic objects; and outputting a warning alert (26) using the hazard analysis module, based on the determined information relating to the current hazard situation.

Description

[0001] Method and device for automated monitoring of ferry operations of a passenger conveyor system

[0002] The invention relates to a method for monitoring the ferry operation of one or more passenger conveyor systems. The invention further relates to a device for implementing this method, a correspondingly equipped passenger conveyor system, a computer program product, and a computer-readable medium.

[0003] Passenger conveyor systems such as escalators and moving walkways are used to transport passengers in buildings such as department stores and large shopping centers, as well as in train stations, subway stations, and airports. In the latter three areas in particular, there can be an increased risk of accidents, for example, when users who are pressed for time crowd other users on the passenger conveyor systems. This can cause people to fall on the conveyor belt or step belt of the escalator or on the pallet belt of the moving walkway and sustain serious injuries if the step belt or pallet belt is not stopped, for example, by an emergency stop. Others, in turn, carelessly misuse passenger conveyor systems as sports or play equipment, thereby endangering themselves and other users.

[0004] Video cameras are traditionally used to monitor passenger conveyor systems. Their video sequences are transmitted in real time to a monitoring room and displayed on screens there. From the monitoring room, several passenger conveyor systems are usually monitored by monitoring personnel. However, inattention or fatigue among monitoring personnel can lead to critical situations arising on the passenger conveyor system not being recognized or not being recognized in a timely manner, and thus countermeasures or assistance not being initiated, or being initiated too late.

[0005] To solve the aforementioned problem, monitoring systems are configured to automatically detect critical situations involving users on the passenger conveyor system by processing motion sequence recordings using image recognition. As soon as a critical situation is detected, the corresponding motion sequence recording is displayed, for example, on the monitor in the monitoring room, so that monitoring personnel are alerted and can initiate countermeasures or assistance promptly.

[0006] EP 3 276 535 A1 describes such a monitoring system for a passenger conveyor system.

[0007] When analyzing motion sequence recordings, such as those provided by cameras used to monitor a passenger conveyor system, differential images are often used for image recognition. A reference image is typically subtracted from one or more currently recorded images of a motion sequence recording. The currently recorded images represent an image of the actual passenger conveyor system, including the passengers and / or objects on it. The reference image, on the other hand, represents the passenger conveyor system without passengers and / or objects and was recorded, for example, before the system was put into operation. The resulting differential images only show the passengers and / or objects.Such difference images can be evaluated automatically much more easily than the original motion sequence recordings, since the original motion sequence recordings depict many details of the passenger conveyor system, which complicate image analysis without contributing to the detection of critical situations on the passenger conveyor system.

[0008] However, the concrete implementation of generating difference images can be complex. In particular, the fact that parts of the passenger conveyor system, such as its conveyor belt, move relative to other parts during operation can make generating difference images difficult. For example, easily analyzable difference images can generally only be generated if the current operating state of the passenger conveyor system, i.e., the current positioning of its conveyor belt, for example, matches as precisely as possible the operating state the passenger conveyor system was in when the reference image was recorded. For example, the step edges of a monitored escalator in the currently recorded image of a motion sequence recording should correspond precisely to the step edges in a reference image.An alternative approach to the automated monitoring of passenger conveyor systems can utilize special cameras capable of providing three-dimensional images of a monitored area. For example, TOF (Time Of Flight) cameras can be used, which, in addition to lateral image resolution, also determine information on the distance of objects observed in the monitored area or captured pixels relative to the camera. A monitored distance range can be selected such that moving components of the passenger conveyor system, such as its steps or pallets, are no longer detected. If necessary, the lateral detection range of the TOF camera can also be limited so that balustrades arranged to the side of the conveyor belt are no longer visible. In this case, the generation of difference images can be dispensed with.However, important areas of the passenger conveyor system, such as the footwells on the treads of the steps, are not shown in the motion sequence recordings of TOF cameras set up in this way, so that they cannot be taken into account when assessing critical situations.

[0009] To solve the problems described above, there is a need to provide a monitoring method and a monitoring system that enable reliable and / or relatively easy-to-implement automated monitoring of ferry operations in a passenger conveyor system. Furthermore, there is a need for a correspondingly equipped passenger conveyor system, a computer program product for implementing the monitoring method, and a computer-readable medium storing the computer program product.

[0010] Such a need is met by the subject matter of the independent claims. Advantageous embodiments are specified in the dependent claims and the following description and illustrated in the figures.

[0011] According to a first aspect of the invention, a method for the automated monitoring of ferry operations of a passenger conveyor system using a monitoring system is described. The monitoring system comprises a hazard analysis module and at least one motion detection module. The motion detection module is directed toward the passenger conveyor system and configured to capture electronically processable motion sequence recordings of situations occurring on the associated passenger conveyor system. The method comprises at least the following steps, possibly, but not necessarily, in the specified order:

[0012] Receiving real motion sequence recordings in the hazard analysis module, wherein the real motion sequence recordings were captured by the at least one motion capture module;

[0013] Receiving data of a digital doppelganger of the passenger conveyor system in the hazard analysis module, wherein the data comprises at least information relating to physical properties of the passenger conveyor system that enable a conclusion to be drawn about a visual appearance of the passenger conveyor system in a predetermined state of movement;

[0014] Determining information concerning dynamic objects on the passenger conveyor system by the hazard analysis module, whereby the information is determined based on the data received from the digital doppelganger and the real movement sequence recordings;

[0015] Determining information concerning a current hazardous situation on the passenger conveyor system by the hazard analysis module based on an analysis of movements of the detected dynamic objects; and

[0016] Issuing a warning message by the hazard analysis module based on the information obtained regarding the current hazard situation.

[0017] According to a second aspect of the invention, a device in the form of a monitoring system for monitoring ferry operations of a passenger conveyor system is described. The monitoring system comprises a hazard analysis module configured to receive data from a motion detection module and a database module. The motion detection module is directed toward the passenger conveyor system and configured to capture electronically processable motion sequence recordings of situations occurring on the associated passenger conveyor system. Data from a digital doppelganger of the passenger conveyor system is stored in the database module, wherein the data comprises at least information relating to physical properties of the passenger conveyor system, which allow for inference regarding a visual appearance of the passenger conveyor system in a predetermined motion state.The hazard analysis module is configured to perform or control a method according to an embodiment of the first aspect of the invention.

[0018] According to a third aspect of the invention, a passenger conveyor system is described which comprises a conveyor belt, a drive for driving the conveyor belt, a controller for controlling the drive and a monitoring system according to an embodiment of the second aspect of the invention, wherein the movement detection module of the monitoring system is directed at least to partial areas of the conveyor belt.

[0019] According to a fourth aspect of the invention, a computer program product is described which comprises machine-readable program instructions which, when executed on a programmable device, cause the device to perform or control a method according to an embodiment of the first aspect of the invention.

[0020] According to a fifth aspect of the invention, a computer-readable medium having a computer program product stored thereon according to an embodiment of the fourth aspect of the invention is described.

[0021] In brief, and without limiting the scope of the invention, one idea underlying the invention can be seen in monitoring the ferry operation of a passenger conveyor system by not comparing, as is conventional, motion sequence recordings provided by a motion detection module with, for example, a camera with recordings that serve as reference images and that were also recorded with a camera in order to ultimately create difference images from which information about the current danger situation can then be derived.Instead of such reference images, data from a digital doppelganger of the passenger conveyor system should be used to determine information about dynamic objects, such as moving passengers, their luggage and objects carried with them, animals, and the like, on the passenger conveyor system. This can then ultimately be used to infer current dangerous situations by analyzing the movements of these dynamic objects. The data from the digital doppelganger reflects physical properties of the passenger conveyor system, such as the spatial dimensions of components and their dynamic movements, which allow conclusions to be drawn about their appearance. Accordingly, this data can be used, for example, to generate virtual motion sequence recordings that reproduce the area of ​​the passenger conveyor system monitored by the motion detection module.These virtual motion sequence recordings can then be used to derive information about the dynamic objects moving on the passenger conveyor system. If necessary, the virtual motion sequence recordings can be used similarly to conventional reference images to generate difference images by subtracting them from the real motion sequence recordings. However, compared to the use of conventional reference images, the virtual motion sequence recordings can be determined from the data of the digital doppelganger with greater precision, fewer image artifacts, and / or less computing power. Furthermore, the virtual motion sequence recordings can be relatively easily synchronized with the currently recorded real motion sequence recordings, which can significantly simplify the generation of difference images.

[0022] Possible configurations and advantages of embodiments are described in more detail below.

[0023] A passenger conveyor system is designed to transport passengers and / or objects within a building. The passenger conveyor system can be designed as an escalator or a moving walkway, for example. The passenger conveyor system has stationary components that are permanently connected to the building, such as a supporting structure, a drive, balustrades, etc. The passenger conveyor system also has movable components that can be moved relative to the stationary components. In escalators, for example, several steps that are coupled one behind the other in one direction of movement form a step belt that can be moved in a rotating manner relative to the stationary components by a drive. In moving walkways, several pallets are coupled one behind the other to form a pallet belt that can be moved in a rotating manner. Passengers can use the rotating step belt or the moving walkway.Use a pallet conveyor to be transported along a travel path.

[0024] In order to monitor the ferry operation of the passenger conveyor system, it is equipped with a monitoring system in which data collected by a motion detection module is evaluated by a hazard analysis module in order to derive information about current hazardous situations and, if necessary, issue warning messages.

[0025] The motion detection module is a device configured to capture motion sequence recordings of situations occurring on the passenger conveyor system. In particular, the motion detection module should be configured to optically or visually monitor situations on the passenger conveyor system and output them accordingly as image data representing the motion sequence recordings. The motion sequence recordings can be a plurality of consecutive image recordings that, for example, reproduce an area of ​​the passenger conveyor system in two-dimensional or three-dimensional form.For this purpose, the motion detection module can, for example, use one or more image capture devices such as photo cameras, video cameras, thermal imaging cameras, laser scanners, TOF cameras, a set of multiple sensors, and / or the like, whose motion sequence recordings are accordingly recorded in electronically processable form as an image sequence, video film sequence, thermal image sequence, etc. When using multiple motion detection modules per passenger conveyor system, each motion detection module is preferably assigned a specific section or area so that not the entire escalator or moving walkway is visible in one of the motion sequence recordings.

[0026] The actual motion sequence recordings currently captured by the motion detection module are transmitted to and received by the hazard analysis module as a data stream. To process this data, the hazard analysis module is designed as a data processing system and, among other things, has a processor that processes the data in a predetermined manner. The processor is preferably programmable using a computer program. In addition, the hazard analysis module generally has a data memory and data interfaces for exchanging data with external devices and / or databases.

[0027] The hazard analysis module described herein is particularly configured to receive and process data from a digital duplicate of the passenger conveyor system in addition to the real movement sequence recordings.

[0028] The digital doppelganger is sometimes also referred to as a digital twin. A digital doppelganger is generally understood to be a virtual representation of a real object, such as the passenger conveyor system in this case, which reproduces the object's physical properties as realistically as possible. The digital doppelganger is usually stored as a data set (sometimes also referred to as a digital doppelganger data set) in a database. Depending on the application and intended use, this data set can contain a wide variety of parameters or properties of the real object. For example, the data set can include information regarding the mechanical properties, geometric properties, optical properties, electrical properties, magnetic properties, material properties, etc. of the real object.The data contained in the digital doppelganger should reflect the real properties of the object as accurately as possible so that the properties of the real object can be reproduced very realistically at a later date, for example, without actually having access to the real object. Furthermore, the data contained in the digital doppelganger should also, if possible, allow the behavior of the real object to be reproduced or even predicted under certain conditions and / or influences. For this purpose, the data of the digital doppelganger can be used, for example, for physical calculations, modeling, and / or simulations.

[0029] For application in the monitoring method presented here, the data from the digital doppelganger should enable inference regarding the visual appearance of the passenger conveyor system. In particular, such inference should be possible for a predetermined movement state of the passenger conveyor system. The visual appearance should reflect an externally recognizable, external appearance of the passenger conveyor system, similar to a top view of the passenger conveyor system using, for example, a camera-based motion detection module whose field of view is directed towards the passenger conveyor system. In particular, the digital doppelganger should be able to represent the external appearance of the step belt or pallet belt, an entry and / or exit area, a handrail on a balustrade, etc. of the passenger conveyor system.

[0030] The digital doppelganger is intended to reproduce the appearance of the passenger conveyor system in at least one predetermined state of motion. A state of motion can be understood as a stationary or a dynamic state of the passenger conveyor system. The state of motion generally indicates at least the positions of movable components of the passenger conveyor system relative to their stationary components and / or to other movable components. In particular, the state of motion indicates the current location of steps or pallets of the step belt or pallet belt. Furthermore, the state of motion can indicate the speeds at which the movable components move relative to one another and / or relative to the stationary components.

[0031] The data in the digital doppelganger should enable the most accurate reproduction possible of the visual appearance of the passenger conveyor system in at least one predetermined motion state. For this purpose, the digital doppelganger can store, for example, design data (e.g., CAD data), material data, processing data, assembly data, maintenance data, etc., whereby this data represents information about physical parameters that influence the appearance (shell) of the passenger conveyor system. If necessary, the digital doppelganger can contain data that enables the reproduction of the visual appearance of the passenger conveyor system in several different motion states.In addition, the data stored in the digital doppelganger can be used to calculate, simulate, model, extrapolate, or otherwise determine the visual appearance of the passenger conveyor system in other states of motion. The hazard analysis module is designed to process the real movement sequence recordings received by the motion detection module, if possible in real time, with the help of the data received from the digital doppelganger in order to derive information about a potentially prevailing current hazardous situation on the passenger conveyor system. For this purpose, the hazard analysis module determines information about dynamic objects on the passenger conveyor system. Such dynamic objects can be, for example, passengers, objects, animals, etc. that are transported by the passenger conveyor system and which themselves remain passive. Dynamic objects can also be passengers or similar.which move relative to the passenger conveyor system, for example when a passenger falls or when a passenger actively moves along the step belt or pallet belt of the conveyor system.

[0032] The hazard analysis module then analyzes the movements of the detected dynamic objects in order to derive information regarding a current hazardous situation. For example, the hazard analysis module uses analysis algorithms to examine the motion sequence recordings for critical situations, taking into account the data from the digital doppelganger. For this purpose, automated detection processes or analysis methods known from the technical field of surveillance electronics, such as image analysis methods and corresponding algorithms, motion analysis methods and corresponding algorithms, statistical and heuristic evaluation methods, and the like, can be used to detect movement patterns of users of the passenger conveyor system that deviate from the usual or are atypical.Depending on the movement sequence of a deviating movement process, the hazard analysis module can assume a hazardous situation, e.g. an accident situation in which there is an acute danger that a passenger is or will be injured, or a critical situation in which there is at least a significant risk of such injuries.

[0033] If a dangerous situation is detected, a warning message can be issued. The warning message can, for example, be transmitted to another device. For example, the warning message can be transmitted to a remote monitoring center. The issuing of the warning message can trigger reactions in a receiving device, which can ultimately lead to measures being taken to counteract the dangerous situation. For example, the operation of the passenger conveyor system can be stopped or slowed down as a result of the warning message. Alternatively or in addition, the warning message can cause a warning signal to be issued to users of the passenger conveyor system to warn them that the operating mode of the passenger conveyor system will be changed at short notice, e.g. the passenger conveyor system will be braked.Such a warning signal can preferably be issued optically, acoustically or in another way in the immediate vicinity of the passenger conveyor system.

[0034] According to one embodiment, the hazard analysis module determines virtual motion sequence recordings based on the data received from the digital doppelganger. The information regarding dynamic objects on the passenger conveyor system is then determined by the hazard analysis module by comparing the virtual motion sequence recordings with the real motion sequence recordings.

[0035] In other words, the data from the digital doppelganger can be used to create virtual motion sequence recordings that reproduce the appearance of the passenger conveyor system in a similar or identical manner to the real motion sequence recordings captured by the motion capture module. By then comparing these virtual motion sequence recordings with the current real motion sequence recordings, information about the dynamic objects on the passenger conveyor system can be obtained and used to infer potential hazardous situations.

[0036] According to a specific embodiment, the comparison of the real motion sequence recordings with the virtual motion sequence recordings is carried out by forming a difference between the real motion sequence recordings and the virtual motion sequence recordings.

[0037] In such a difference calculation, a virtual motion sequence recording is subtracted from a corresponding real motion sequence recording, or vice versa. Those image components that are the same in both motion sequence recordings are thus no longer represented in the resulting difference image. This can significantly simplify the evaluation of the motion sequence recordings.

[0038] According to a further specific embodiment, the method further comprises synchronizing the real movement sequence recordings with the virtual movement sequence recordings such that the passenger conveyor system is in the same movement state in both movement sequence recordings.

[0039] Synchronizing the real and virtual motion sequence recordings can, in particular, result in moving components of the passenger conveyor system being in the same positions and / or moving in the same way in both recording types. Synchronization can be achieved within specified tolerances. Therefore, the real motion sequence recordings can be compared particularly well with the virtual motion sequence recordings after such synchronization.

[0040] If the two synchronized motion sequence recordings depict the passenger conveyor system in the same motion state, the components of the passenger conveyor system are generally no longer represented in a difference image. Passengers or objects that are currently on the passenger conveyor system and are therefore represented in the real motion sequence recording, but not in the virtual motion sequence recording, are thus represented as dynamic objects in the difference image and can be very well analyzed in this image.

[0041] According to a further specific embodiment, the synchronization takes place taking into account data received from a control system of the passenger conveyor system and containing information about an actual current movement state of the passenger conveyor system.

[0042] In other words, in the monitoring method presented here for synchronizing the real with the virtual movement sequence recordings, data can be used which can be provided by a controller of the passenger conveyor system and which provides information about the current movement state of the passenger conveyor system. For example, the controller can hold precise information about the current position of the step belt or pallet belt along the travel path, i.e. where the steps or pallets are currently arranged. Information about a current movement, in particular a current speed and / or direction of movement, of the step belt or pallet belt can also be available in the controller. If such information is retrieved orreceived, they can be used to suitably prepare the received data of the digital doppelganger in order to synchronize the virtual appearance represented by it or the virtual movement sequence recordings of the passenger conveyor system that can be generated from it with the real movement sequence recordings.

[0043] According to a further specific embodiment, the synchronization can additionally or alternatively be carried out taking into account speed information from a frequency converter of the passenger conveyor system or a signal from a sensor arranged on the passenger conveyor system, such as an encoder.

[0044] A frequency converter can be provided in the passenger conveyor system to control a drive of the step belt or pallet belt in a desired manner by appropriately adjusting frequencies within a power supply. The speed information used or determined by the frequency converter can be retrieved or received by the hazard analysis module. This information can then be used to process the data of the digital doppelganger in such a way that it reproduces the virtual appearance of the passenger conveyor system with moving components, in particular a moving step belt or pallet belt, in such a way that their speed matches the speed of the real components specified by the frequency converter.

[0045] Similarly, in a passenger conveyor system, an encoder can be provided to output signals that indicate the current speed of moving components of the passenger conveyor system. Such an encoder can, for example, have moving components that are moved by the moving components of the passenger conveyor system, and whose movement can then be detected by suitable sensors, thus indirectly inferring the movement of the components of the passenger conveyor system. In this case, too, the obtained speed information can be used to synchronize the data of the digital doppelganger regarding the speeds of the virtually moving components with the real moving components of the passenger conveyor system.

[0046] According to a further specific embodiment, the synchronization can additionally or alternatively be carried out taking into account moving position information, wherein the moving position information is determined by observing a marking which is fixed at a location of a moving component of the passenger conveyor system and is moved along by it.

[0047] In other words, for example, one or more markings can be applied to a step belt or a pallet belt at one or more predetermined positions. Such a marking can, for example, be visually recognizable and thus detected by a camera or the like. For example, the marking can be designed as a color marking. Alternatively, the marking can also be formed by intrinsic properties of the moving component itself, for example a visually easily recognizable edge of a step or a pallet. As a further alternative, the marking can be detectable in another way, for example electrically and / or magnetically detectable. Suitable sensors can be designed and / or arranged on the passenger conveyor system at suitable positions in order to be able to observe the marking(s) and provide corresponding moving position information.Based on this moving position information, the actual movements of the observed component, as well as the current position of this component's marker, can be inferred. The marker is also virtually represented in the digital doppelganger. Accordingly, the data obtained from the digital doppelganger can be appropriately processed to synchronize it with the actual movements and the correct alignment of the marker of the observed component.

[0048] According to one embodiment, in order to analyze movements of the detected dynamic objects, a comparison of the movements of the detected dynamic objects with stored movement scenarios that represent potential dangerous situations is carried out.

[0049] In particular, an automated detection and / or assessment process for critical situations can take place in the hazard analysis module, whereby the movements of the identified dynamic objects extracted using known analysis methods are assessed. To carry out the assessment process, a set of atypical movement scenarios of critical situations can be stored in the hazard analysis module. This set can comprise various movement scenarios that can be compared with the movements of the identified dynamic objects extracted from motion sequence recordings. This set of stored movement scenarios can be generated, for example, through a machine learning process by using a dummy or stuntman to recreate typical critical situations such as falls or careless actions on the passenger conveyor system, recording them, and incorporating the extracted movement sequences into the set as stored movement scenarios.It is, of course, also possible to create such movement scenarios purely virtually with suitable software and sufficient computing power. Of course, the set can also be supplemented with additional stored movement scenarios that led to an accident during operation of the passenger conveyor system and were not detected by the hazard analysis module. If there is a sufficient match between a dynamic object and a stored movement scenario, the hazard analysis module assumes the presence of a hazardous situation and sends a warning message and / or a warning signal.

[0050] According to one embodiment, the current hazardous situation on the passenger conveyor system is assessed based on a comparison of the detected dynamic objects with various stored movement scenarios. The warning message is then issued depending on the assessment of the hazardous situation.

[0051] Preferably, the various atypical movement scenarios of the set have different weightings in the sense of a hierarchy. Depending on these weightings, different actions can be defined to influence the movement of the passenger conveyor system. For example, a control module can only enable a connection between a manually operated emergency stop triggering device and the control system of the corresponding passenger conveyor system if a detected user fall has a very high weighting and an emergency stop is defined as the action for this high weighting. The emergency stop is initiated immediately when the monitoring personnel activates the emergency stop triggering device.

[0052] If, for example, a user enters the passenger conveyor system in the opposite direction to its conveying direction, this critical situation can have a medium weighting and for this medium weighting it can be provided that the control module only releases the connection between the manually operated speed controller and the control of the corresponding passenger conveyor system for operation.

[0053] It is also possible for a user to resolve the critical situation themselves, for example, by catching their fall by holding onto the handrail and then standing up again. Based on the detected situation, this movement sequence recording can be immediately displayed, for example, on the screen of an interaction module, so that monitoring personnel are aware of it and can, if necessary, rectify the causes of the near-fall (user distraction, disruptive influences outside the escalator or moving walkway). However, the low weighting ensures that the control module does not release any of the aforementioned connections. It should be noted at this point that establishing a connection does not automatically mean its release.

[0054] A passenger conveyor system according to an embodiment of the third aspect of the invention comprises a conveyor belt, a drive, a controller, and an embodiment of the monitoring system described herein. The conveyor belt can be designed as a step belt or a pallet belt. The drive is configured to move the conveyor belt, including passengers or objects standing on it, along a conveyor path. The drive is controlled by the controller. Power can be supplied via a frequency converter. An encoder can be provided on the conveyor belt or another component moving with the conveyor belt in order to detect movements of the conveyor belt. The monitoring system is designed to detect dangerous situations in the area of ​​the conveyor belt and / or adjacent regions. For this purpose, its motion detection module is directed at at least partial areas of the conveyor belt.Warning messages can be forwarded directly or indirectly to the control system in order to, for example, slow down or stop the movement of the conveyor belt in the event of a dangerous situation.

[0055] Embodiments of the computer program product according to the fourth aspect of the invention can be formulated in any processor-interpretable computer language. The computer program product can be stored on a computer-readable medium according to an embodiment of the fifth aspect of the invention. Such a computer-readable medium can be portable. In particular, the medium can store data in a non-volatile or volatile manner.

[0056] For example, the computer-readable medium may be a CD, a DVD, a flash memory, a ROM, an EPROM, or the like. Alternatively, the computer program may also be stored on a computer-readable medium in the form of another computer, a server, a data cloud, or the like, from where it can be downloaded via a data network, in particular the Internet.

[0057] It should be noted that possible advantages and configurations of embodiments of the invention are described herein partly with reference to a monitoring method according to the invention and partly with reference to a monitoring system according to the invention. A person skilled in the art will recognize that the described features can be appropriately transferred, adapted, exchanged, or modified to achieve further embodiments of the invention.

[0058] Embodiments of the invention are described below with reference to the accompanying drawings, wherein neither the drawings nor the description are to be interpreted as limiting the invention.

[0059] Figure 1 shows a three-dimensional view of a passenger conveyor system whose ferry operation is to be monitored, as well as a monitoring system for monitoring this passenger conveyor system according to an embodiment of the present invention. The figure is merely schematic and not necessarily to scale. The same reference numerals denote identical or equivalent features in the figure.

[0060] Figure 1 shows a three-dimensional view of a passenger conveyor system 1 with several escalators 3', 3", 3'", whose movement F is monitored by a monitoring system 5. The monitoring system 5 comprises a hazard analysis module 7 and a motion detection module 9.

[0061] In order to be able to monitor the multiple escalators 3', 3", 3'", the motion detection module 9 has a plurality of cameras 11', 11", 11'", 13', 13", 13"'. In the present exemplary embodiment, two escalators 3', 3" are shown in detail. The balustrade sketched with a broken line symbolizes a possible plurality of further escalators 3'". Each of these escalators 3', 3", 3'" is assigned two cameras 11', 13', 11", 13", 11'", 13'" of the motion detection module 9, which record real movement sequence recordings 15 from the ferry operation F of an escalator assigned to it.

[0062] The hazard analysis module 7 and the motion detection module 9 are preferably connected to each other via a data network 17. Furthermore, the hazard analysis module 7 is also connected to a database module 19, from which data X, Y, Z of a digital doppelganger 29 of the passenger conveyor system 1 can be made available. The data network 17 can be set up in a variety of ways, such as via a local wired and / or wireless data network, via network connections to a data cloud 47, via CAN bus systems, Bluetooth connections, and the like. The individual modules 7, 9, 19 can exchange data with each other, whereby this data exchange between the individual modules can be unidirectional or bidirectional, as well as continuous, sequential, or temporary, depending on requirements.

[0063] By means of the monitoring system 5, embodiments of the method according to the invention for monitoring the ferry operation F of the passenger conveyor system 1 can be carried out.

[0064] Each of the cameras 11', 13', 11", 13", 11'", 13"' of the motion detection module 9 is directed at an associated passenger conveyor system 3', 3", 3'" and captures electronically processable real motion sequence recordings 15 of situations 101A, 101B, 101C occurring on the passenger conveyor system 1. This can also include the immediate surroundings of the passenger conveyor system 1. This immediate surroundings include, for example, the areas upstream of the access areas 21, 23 of a building (not shown) in which the passenger conveyor system 1 is installed. The upstream areas of the building are also referred to as anterooms and are monitored, if necessary, by proximity sensors (not shown) of the passenger conveyor system 1. The proximity sensors transmit their detection data to an automatic start / stop system of the passenger conveyor system 1, which is usually implemented in the respective controls 25', 25" of the escalator 3', 3".The controls 25', 25" control the operation of a respective drive 45 of the corresponding escalator 3', 3".

[0065] The motion detection module 9 can use video cameras, thermal imaging cameras, easer scanners, TOF cameras, a combination of multiple sensors, and the like, with their actual motion sequence recordings 15 being recorded accordingly as a video sequence, image sequence, thermal image sequence, etc. in electronically processable form. When using multiple cameras 11, 13 or similar in the passenger conveyor system 1, each of the cameras 11, 13 is preferably assigned a specific section or area. These areas preferably overlap so that there are no monitoring gaps in which critical situations 101A, 101B, 101C of users 102A, 102B, 102C could occur without being observed. Since the aim is to achieve the most secure monitoring possible, the motion detection module 9 preferably continuously records what is happening on the passenger conveyor system 1.This also includes the motion detection module 9 transmitting its real motion sequence recordings 15 to the hazard analysis module 7 in real time.

[0066] The transmitted real motion sequence recordings 15 are received in the hazard analysis module 7. Furthermore, the hazard analysis module 7 receives data X, Y, Z from the digital doppelganger 29 stored in the database module 19. These data X, Y, Z allow a conclusion to be drawn about a visual appearance 30 of the passenger conveyor system 1 in a predetermined motion state. Accordingly, virtual motion sequence recordings 16 can be determined based on a digital doppelganger 29 and made available to the hazard analysis module 7.

[0067] The hazard analysis module 7 can then determine information about dynamic objects 31, such as moving users 102A, 102B, 102C on the passenger conveyor system 1, using both the received real motion sequence recordings 15 and the received data X, Y, Z of the digital doppelganger 29. In particular, a comparison of the virtual motion sequence recordings 16 with the real motion sequence recordings 15 can advantageously be carried out, for example by calculating the difference between the two types of motion sequence recordings 15, 16.

[0068] Preferably, the real motion sequence recordings 15 can be synchronized with the virtual motion sequence recordings 16 such that the passenger conveyor system 1 is in the same motion state in both motion sequence recordings 15, 16. For this purpose, for example, data can be queried from the controller 25 in the passenger conveyor system 1, which data represents information about a current motion state of the real passenger conveyor system 1. Alternatively or additionally, speed information from a frequency converter 33 of the passenger conveyor system 1 and / or signals from an encoder 35 on the passenger conveyor system 1 can be queried and used to synchronize the two motion sequence recordings 15, 16.As a further alternative or addition, special markings 37 can be provided on movable components 39 of the passenger conveyor system 1, such as steps 41 of a conveyor belt 43, which move with these movable components 39. These markings 37 can be monitored, for example, using a sensor system to obtain moving position information concerning the movable components 39, by means of which the virtual and real movement sequence recordings 15, 16 can be precisely synchronized.

[0069] The information about the dynamic objects 31 obtained in this way is subsequently examined for hazardous situations 101A, 101B, 101C using analysis algorithms. These analysis algorithms are based, for example, on known image processing techniques that are optimized and applied, for example, in self-learning processes using artificial intelligence in neural networks. A common image processing technique for generating information from an image is, for example, the calculation of a histogram, which provides information about a statistical brightness distribution in the image. Such a histogram can serve, for example, as a configuration for further image processing steps or as information for a human user of a software. Other calculable information about an image includes, for example, its entropy or average brightness.Based on this information, vector analyses can be carried out to determine how individual prominent points move relative to one another, and from this, conclusions can be drawn about movement scenarios 103A, 103B, 103C of users 102A, 102B, 102C.

[0070] Of course, instead of the method steps described above, other or further analysis techniques and analysis methods known from the technical field of video surveillance can also be used to extract movement sequences of the users 101A, 101B, 101C from the movement sequence recordings 15, 16.

[0071] Once a user's 102 movement sequence (102A, 102B, 102C) has been recognized by the hazard analysis module 7 and extracted, for example, as a skeletal movement sequence, the movement sequence can be compared with a stored set of possible hazardous or critical situations. More precisely, the situations stored in the set represent atypical movement scenarios (103A, 103B, 103C) in potentially hazardous situations.

[0072] As soon as the hazard analysis module 7 detects a hazardous situation 101A, 101B, 101C, it issues a warning signal 26. The warning signal 26 can be transmitted, for example, to a monitoring center 28. From there, in response to the warning signal 26, operation of the passenger conveyor system 1 can be slowed down or stopped, for example, by appropriately influencing the control system 25 of the affected escalator 3. Furthermore, a suitable visual or acoustic warning signal can be issued via signaling devices 27 on the passenger conveyor system 1 to warn users of the passenger conveyor system 1.

[0073] On the passenger conveyor system 1 shown in Figure 1 and in its immediate vicinity, three hazardous situations 101A to 101C are illustrated by way of example to better understand the present invention. A set of possible movement scenarios 103A, 103B, 103C, corresponding to these hazardous situations 101A, 101B, 101C, is present in the hazard analysis module 7, with which dynamic objects 31 extracted from the real movement sequence recordings 15 are compared. The possible location at which such a hazardous situation 101A, 101B, 101C could even occur can also play a role. An atypical movement scenario 103A, which, for example, represents a fall in the sloping middle section of an escalator 3'', cannot occur in the same way in the vestibule of the passenger conveyor system 1 or in one of the access areas 21, 23.The set of atypical movement scenarios 103A, 103B, 103C can be generated, for example, through a machine learning process by using a dummy or stuntman to recreate typical hazardous situations 101A, 101B, 101C, such as falls or careless actions on the passenger conveyor system 1. The dynamic objects 31 captured and extracted in this way, or their movement sequences, can be included in the set as atypical movement scenarios 103A, 103B, 103C. Of course, the set can also be supplemented with further atypical movement scenarios that led to an accident during the operation of the passenger conveyor system 1 and that were not detected by the hazard analysis module 7. If there is sufficient agreement between a dynamic object 31 and an atypical movement scenario 103A, 103B, 103C, the hazard analysis module 7 can assume or determine the existence of a hazardous situation and issue a warning message 26.

[0074] The various atypical movement scenarios 103A, 103B, 103C can have different weightings in the sense of a ranking. According to this ranking, instructions (not shown) can be stored in the monitoring center 28 regarding how the ferry operation F of the affected escalator 3', 3" should be influenced. Depending on the weighting, for example, an emergency stop must be initiated, the travel speed reduced, an acoustic and / or visual warning issued, etc. The dangerous situation 101A recorded by the motion detection module 9 is recognized in the hazard analysis module 7 as a "falling down," and the associated warning message 26 is assigned the highest weighting (= emergency stop), since continuing the ferry operation F could lead to severe injuries to the fallen user 102A.The critical situation 101B recorded by the motion detection module 9 is recognized in the hazard analysis module 7 as "entering in the wrong direction," and the associated warning message 26 is assigned the lowest priority. By entering incorrectly, user 102B does not directly endanger themselves, but rather disturbs oncoming users exiting the passenger conveyor system 1. In this case, for example, a visual and / or acoustic warning to the affected user 102B is sufficient.

[0075] The critical situation 10 IC recorded by the motion detection module 46 is recognized in the hazard analysis module 7 as "entering with a shopping cart," and the associated warning message 26 is assigned a medium weighting. In this case, the user 102C is only in danger when she reaches the rising central section of the passenger conveyor system 1 with her shopping cart. Depending on the medium weighting, the instruction can be to reduce the travel speed F and issue a visual and / or acoustic warning. Reducing the travel speed allows the user 102C to exit the escalator 3' in the opposite direction more easily than at normal travel speed. This prevents her from entering the rising section of the escalator with the shopping cart too quickly.

[0076] Finally, it should be noted that terms such as "having," "comprising," etc., do not exclude other elements or steps, and terms such as "a" or "an" do not exclude a plurality. Furthermore, it should be noted that features or steps described with reference to one of the above embodiments may also be used in combination with other features or steps of other embodiments described above. Reference signs in the claims are not to be considered as limitations.

Claims

Patent claims 1. Method for the automated monitoring of a ferry operation (F) of a passenger transport system (1) by means of a monitoring system (5), wherein the monitoring system (5) comprises a hazard analysis module (7) and at least one motion detection module (9), wherein the motion detection module (9) is directed towards the passenger transport system (1) and is configured to capture electronically processable real motion sequence recordings (15) of situations occurring on the associated passenger transport system (1), wherein the method comprises: Receiving real motion sequence recordings (15) in the hazard analysis module (7), wherein the real motion sequence recordings (15) were captured by the motion detection module (9); Receiving data (X, Y, Z) of a digital twin (29) of the passenger transport system (1) in the hazard analysis module (7), wherein the data (X, Y, Z) include at least information concerning physical properties of the passenger transport system (1) that allow inferences to be made about a visual appearance (30) of the passenger transport system (1) in a predetermined state of motion; Determining information concerning dynamic objects (31) on the passenger transport system (1) by the hazard analysis module (7), wherein the information is determined based on the received data (X, Y, Z) of the digital twin (29) and the real motion sequence recordings (15); Determining information concerning a current hazard situation (101A, 101B, 101C) at the passenger transport system (1) by the hazard analysis module (7) based on an analysis of movements of the identified dynamic objects (31); and issuing a warning (26) by the hazard analysis module (7) based on the determined information concerning the current hazard situation (101A, 101B, 101C).

2. Method according to claim 1, wherein virtual motion sequence recordings (16) are determined based on the received data (X, Y, Z) of the digital twin (29) by the hazard analysis module (7), and where the information concerning dynamic objects (31) on the passenger transport system (1) is determined by the hazard analysis module (7) by comparing the virtual motion sequence recordings (16) with the real motion sequence recordings (15).

3. Method according to claim 2, wherein the comparison of the real motion sequence recordings (15) with the virtual motion sequence recordings (16) is carried out by calculating a difference between the real motion sequence recordings (15) and the virtual motion sequence recordings (16).

4. Method according to one of claims 2 and 3, wherein the method further comprises synchronizing the real motion sequence recordings (15) with the virtual motion sequence recordings (16) such that movable components (39) of the passenger transport system (1) are in the same state of motion during both motion sequence recordings (15, 16).

5. Method according to claim 4, wherein the synchronization is carried out taking into account data received by a controller (25) of the passenger transport system (1) and which includes information about an actual current movement state of the passenger transport system (1).

6. Method according to one of claims 4 and 5, wherein the synchronization is carried out taking into account speed information from a frequency converter (33) of the passenger transport system (1) or a signal from an encoder (35) arranged on the passenger transport system (1).

7. Method according to one of claims 4 to 6, wherein the synchronization is carried out taking into account a moving position information, wherein the moving position information is determined by observing a marker (37) which is fixed at a point on a movable component (39) of the passenger conveying system (1) and is moved by it.

8. Method according to one of the preceding claims, wherein, for the analysis of movements of the identified dynamic objects (31), a comparison of the movements of the identified dynamic objects (31) with stored movement scenarios (103A, 103B, 103C) which represent potential hazard situations (101A, 101B, 101C) is carried out.

9. Method according to one of the preceding claims, wherein the current hazard situation (101A, 101B, 101C) at the passenger transport system (1) is evaluated based on a comparison of the identified dynamic objects (31) with various stored motion scenarios (103A, 103B, 103C), and wherein the warning is issued depending on the evaluation of the hazard situation (101A, 101B, 101C).

10. Monitoring system (5) for monitoring a ferry operation (F) of a passenger transport system (1), wherein the monitoring system (5) comprises a hazard analysis module (7) configured to receive data from a motion detection module (9) and from a database module (19), wherein the motion detection module (9) is directed towards the passenger transport system (1) and is configured to capture electronically processable real motion sequence recordings (15) of situations occurring on the associated passenger transport system (1), and wherein the database module (19) stores data of a digital twin (29) of the passenger transport system (1), wherein the data includes at least information concerning physical properties of the passenger transport system (1) that allows inferences to be made about a visual appearance of the passenger transport system (1) in a predetermined state of motion;wherein the hazard analysis module (7) is configured to perform or control a process according to any of the preceding claims.

11. Personnel conveying system (1) comprising: a conveyor belt (43); a drive (45) for driving the conveyor belt (43); a control unit (25) for controlling the drive (45); a monitoring system (5) according to claim 10, the motion detection module of which (9) is directed at least towards parts of the conveyor belt (43).

12. Computer program product comprising machine-readable program instructions which, when executed on a programmable device, cause the device to carry out or control a method according to any one of claims 1 to 9.

13. Computer-readable medium with a computer program product stored thereon according to claim 12.

Images