Visual inspection diagnostics

Abstract

A passenger conveyor diagnostic system comprises a processor storing at least one visual inspection diagnosis and at least one sensor configured to acquire one or both of an image and a sound. The processor applies the at least one visual inspection diagnosis to one or both of the image and the sound to generate a diagnosed situation of the passenger conveyor system and overlays the diagnosed situation of the passenger conveyor system onto the image.

Description

Technology Field

[0001] The present disclosure generally relates to passenger conveyor systems, and more specifically to passenger conveyor maintenance and operational diagnosis. Background Technology

[0002] For example, service providers of passenger conveyor systems, such as elevator and escalator systems, typically rely on user complaints to identify existing malfunctions or defects before repairing or replacing necessary components. Because it is impossible to know when a component or system will malfunction, not only must the system be taken offline at unscheduled times, but once a problem is identified, typically one or more follow-up visits are required to collect ride data for accurate diagnosis. Furthermore, diagnosing and monitoring the passenger conveyor system before and after component defects are identified requires access to circuits, control boxes, drive systems, or other components that are hidden behind various panels or otherwise invisible during normal operation. Accessing these circuits, control boxes, drive systems, or other components may require suspending system service to allow technicians access to the necessary parts.

[0003] According to a non-limiting embodiment, a passenger conveyor diagnostic system comprises a processor storing at least one visual inspection diagnostic, and at least one sensor configured to acquire one or both of an image and a sound. The processor applies the at least one visual inspection diagnostic to one or both of the image and the sound to generate a diagnosed situation of the passenger conveyor system and overlays the diagnosed situation of the passenger conveyor system onto the image.

[0004] According to another non-limiting embodiment, a mobile terminal device is configured to diagnose a passenger conveyor system. The mobile terminal device includes a memory unit storing a visual inspection application configured to perform at least one visual inspection diagnosis, and a network interface device establishing communication with at least one sensor installed on the passenger conveyor system. The mobile terminal device further includes an electronic hardware controller that applies the at least one visual inspection diagnosis to one or both of images and sounds obtained from the at least one sensor to determine at least one diagnosis result of the passenger conveyor system, and displays the at least one diagnosis result overlaid on the image.

[0005] According to another non-limiting embodiment, a method for diagnosing a passenger conveyor system is provided. The method comprises the steps of storing at least one visual inspection diagnosis through a processor, and acquiring one or both of an image and a sound through at least one sensor installed on the passenger conveyor system. The method further comprises the step of generating a diagnosed situation of the passenger conveyor system by applying the at least one visual inspection diagnosis to one or both of the image and the sound through the processor. The method further comprises the step of overlaying the diagnosed situation of the passenger conveyor system onto the image through the processor. Brief explanation of the drawing

[0006] The following description should not be construed as limiting in any way. Refer to the attached drawings, the same elements are denoted by the same symbols: FIG. 1 is a block diagram of a computing system according to a non-limiting embodiment; FIG. 2 illustrates a cloud computing network according to a non-limiting embodiment; FIG. 3 illustrates a mobile terminal device installed with a visual inspection software application operating near an elevator system according to a non-limiting embodiment; FIG. 4 illustrates a mobile terminal device installed with a visual inspection software application operating near an escalator system according to a non-limiting embodiment; FIG. 5a illustrates a mobile terminal device displaying a home interface provided by a visual inspection software application according to a non-limiting embodiment; FIG. 5b illustrates a mobile terminal device running visual inspection software that overlays a diagnostic mode interface on a first video frame illustrating the normal operating state of an elevator system according to a non-limiting embodiment; FIG. 5c illustrates a mobile terminal device running visual inspection software that overlays a diagnostic mode interface on a second video frame illustrating an abnormal operating state of an elevator system according to a non-limiting embodiment; FIG. 5d illustrates a mobile terminal device running visual inspection software that overlays a metering mode interface on a first video frame illustrating the normal operating state of an elevator system according to a non-limiting embodiment; FIG. 5e illustrates a mobile terminal device running visual inspection software that overlays a metering mode interface on a second video frame illustrating an abnormal operating state of an elevator system according to a non-limiting embodiment; and FIG. 6 is a flowchart illustrating a method for diagnosing a passenger conveyor system using a visual inspection software application installed on a mobile terminal device according to a non-limiting embodiment. Specific details for implementing the invention

[0007] Specific details regarding one or more embodiments of the disclosed apparatus and method are presented herein by reference to the drawings, not as a limitation but as an example.

[0008] The term “about” is intended to include a degree of error associated with a specific measurement based on equipment available at the time of application. For example, “about” may include a range of ± 8% or 5%, or 2% of a given value.

[0009] Now, referring to FIG. 1, a computing system (100) capable of carrying out one or more embodiments of the present invention is illustrated according to a non-limiting embodiment. The computing system (100) includes memory (102) and an electronic hardware processor (106). Memory (102) stores various instruction algorithms executable by the processor (106). Executable instructions may be stored or organized in any manner and at any level of abstraction, such as in relation to one or more processes, routines, procedures, methods, functions, etc. As an example, at least some of the instructions are illustrated in FIG. 1 as being associated with a first program (104a) and a second program (104b).

[0010] The processor (106) includes one or more input / output (I / O) devices (108). In some embodiments, the I / O device(s) (108) may include one or more of a keyboard or keypad, a touchscreen or touch panel, a display screen, a microphone, a speaker, a mouse, a button, a remote control, a joystick, a printer, a telephone, or a mobile device (e.g., a smartphone). The I / O device(s) (108) may be configured to provide an interface to enable a user to interact with the system (100).

[0011] The computing system (100) further includes a network interface (110) capable of communicating electrically with a cloud computing network (200) or any other remote network. The network interface (110) includes any communication device (e.g., a modem, a wireless network adapter, etc.) that operates according to a network protocol (e.g., Wi-Fi, Ethernet, satellite, cable communication, etc.) that establishes wired and / or wireless communication with the cloud computing network (200).

[0012] The computing system (100) is exemplary as an example. In some embodiments, one or more of the entities may be optional. In some embodiments, additional entities not shown may be included. For example, in some embodiments, the computing system (100) may be associated with one or more computers or one or more networks, such as telephone networks. In some embodiments, the entities may be arranged or organized in a manner different from that shown in FIG. 1.

[0013] Now, referring to FIG. 2, a cloud computing network (200) capable of carrying out one or more embodiments of the present invention is illustrated according to a non-limiting embodiment. The cloud computing network (200) includes a cloud server (202), one or more mobile terminal devices (204), and one or more backend computers (206).

[0014] The cloud server (202) can store setup parameters for selected functions. In some embodiments, the setup parameters may be stored in the controller. In some embodiments, there may be a simplified fault recovery function located in the controller in the event that the connection between the controller and the cloud server (202) is lost.

[0015] In some embodiments, operational data and meters may be collected and stored on a cloud server (202) across a portfolio of multiple units, sites, or groups. The data and meters may include, but are not limited to, acquired images, sounds, and / or measurements. The cloud server (202) may also store various models, algorithms, software programs, and pre-stored images and / or sounds that can be used with the collected data and meters to perform analysis operations by the backend computer (206). For example, the backend computer (206) includes a processor configured to perform analysis on the acquired images and / or sounds, which may indicate trends and be used to respond to requirements (e.g., product requirements or feature requirements). The analysis may also be used to enable diagnostic or troubleshooting functions. Meters may be used to trigger or improve the accuracy of sales proposals. Meters may be used to provide or schedule maintenance activities, such as preventive maintenance activities. In some embodiments, for example, interface protocols configured to operate with new devices may be stored in a cloud server (202) and used by a local controller. It should be understood that the computer (206) or processor performing the operations described above may be performed locally on a cloud network (200) or on a mobile terminal device (204).

[0016] In some embodiments, functional upgrades for diagnostic, prognostic, and remote repair / rescue functions may be deployed to customers when they are released and deployed to a cloud server (202). Functions may be developed on a backend computer (206) (installed at a server location (207) located remotely from a mobile terminal device (204)) and deployed to the cloud server (202). One or more tests may be performed to ensure that the functions satisfy operational or safety requirements.

[0017] A mobile terminal device (204) may communicate with a cloud server (202) through one or more connections, channels, or links enabled by a network interface. The mobile terminal device (204) may be located at or near a passenger conveyor location (205) and may include, for example, a smartphone (204) that can electrically communicate with the cloud server (202) through a connection (208). The connection (208) may follow one or more communication protocols, standards, etc. For example, the connection (208) may follow telephone, cellular, Wi-Fi, Ethernet, satellite, Bluetooth, Zigbee, zWave, cable communication, or any other known communication method. In some embodiments, the connection (208) may be constant or permanent. The mobile terminal device (204) will be referred to as a smartphone (204) below, but the mobile terminal device (204) may include any mobile device capable of enabling interaction between one or more stored software applications (i.e., apps) and a cloud server (202). For example, the mobile terminal device (204) may also include a computer tablet, a laptop computer, a smart watch, etc.

[0018] The smartphone (204) includes a display (220) and one or more sensors. The sensors include, but are not limited to, one or more cameras (222), speakers (224), microphones (226) and accelerometers (228). The camera (222) can acquire still images and / or videos that the smartphone (204) can play on the display (220). The speaker (224) outputs sound from the smartphone (204) to the external environment. The microphone (226) acquires sounds and noise present near the smartphone (204). The acquired noise can be recorded in memory and played back through the speaker (224). The accelerometer (228) can be configured as a single-axis or multi-axis accelerometer and can detect the magnitude and direction of the appropriate acceleration of the smartphone as a vector quantity. The accelerometer (228) can detect the orientation, coordinate acceleration, vibration, shock, and movement of the smartphone (204).

[0019] As described above, the function may be (re)located to a cloud server (e.g., cloud server (202)). For example, in relation to elevator systems, such function may include dispatch functions for one or more passenger conveyor devices, mobile terminals, operation mode determination, diagnostic functions, special provisions functions, etc. In relation to the dispatch functions, in some embodiments, for example, a user's service request received from a hall box located on a specific floor of a building may be transmitted to a cloud server, and the cloud server may transmit a command instructing a designated elevator car to be relocated to that specific floor to fulfill the service request.

[0020] In some embodiments, a local controller (not shown) may maintain some functions and, accordingly, may include hardware and computing resources to support such functions. The controller may include hardware and / or software for communicating with a cloud server (202). For example, the controller may exchange data and commands with the cloud server (202) to perform control functions.

[0021] Now, referring to FIG. 3, a mobile terminal device (204) with a visual inspection software application installed and operating near an elevator system (300) is illustrated according to a non-limiting embodiment. Although a specific elevator system (300) is illustrated and described as a disclosed embodiment, other configurations and / or systems, such as machine room-less, rope-less, or hydraulic systems, are within the scope of the invention. Additionally, it should be understood that the smartphone (204) may operate with other types of passenger conveyor systems, including but not limited to escalator systems and moving walkway systems, without departing from the scope of the invention.

[0022] The elevator system (300) includes a machine (301) that drives an elevator car (302) located in a vertical shaft (not shown). One or more elevator car rails (304) and guides (306) are attached to the shaft walls. The elevator system (300) further includes one or more hoists (308) that connect the elevator car (302) to a counterweight (not shown). In at least one embodiment, the hoists (308) may be formed as ropes, cables, or belts (308) configured to support the weight of the elevator car (302) and the counterweight in order to propel the elevator car (302) in a desired direction (e.g., up or down) within the shaft. Although one hoist (308) is illustrated, it should be understood that additional hoists (308) may be implemented without departing from the scope of the invention.

[0023] The elevator machine (301) is installed in a machine room (303) located remotely from the elevator car (302). The elevator machine (301) controls a traction sheave (305) that rotates lifting devices (308) to drive the elevator car (302) in a selected direction and causes their movement. The elevator system (300) further includes one or more deflectors or idler sheaves (310) that help guide the movement of the various traction lifting devices (308). In this way, the elevator car (302) can be driven to one or more landings or floors arranged perpendicularly to each other. Thus, the elevator car (302) can be driven to a requested floor, where passengers can then enter and exit the car (302) through the elevator doors (312).

[0024] The elevator car (300) may include a car controller (314) and one or more sensors (316 and 318). The car controller (314) may communicate signals with a cloud computing network (200) and / or a mobile terminal device (204) as will be described in more detail later. In this way, the car controller (314) may enable data exchange between the mobile terminal device (204) and the cloud computing network (200) and / or video cameras (316) and microphones (318).

[0025] The sensors include, but are not limited to, accelerometers and vibration sensors, along with one or more video cameras (316) and one or more microphones (318). The video cameras (316) and microphones (318) communicate signals with a cloud computing network (200). The video camera (316) is configured to acquire video frames and / or generate a video stream of an area and / or components located near the elevator car (302) and / or the elevator car (302). The area includes, for example, a surrounding elevator shaft, but the components include, for example, car rails (304), guides (306), lifting devices (308), and sheaves (310). The microphone (318) is configured to acquire sounds from outside the elevator car (302) (e.g., vibration, squeaking, creaking, etc.), sounds inside the elevator car (302), and / or sounds generated by various elevator components such as car rails (304), guides (306), lifting devices (308), sheaves (305, 310), and machinery (301). Although the video cameras (316) and microphones (318) are depicted as separate individual sensors, it should be understood that the video cameras (316) and microphones (318) can be integrated together to form a single sensor without departing from the scope of the invention.

[0026] Continuing with reference to FIG. 3, a maintenance technician or mechanic (250) can operate an elevator car (302) while diagnosing an elevator system (300) using a mobile terminal device (204). The mobile terminal device (204) will be described as a smartphone, but other types of mobile terminal or user equipment (UE) devices, including but not limited to computer tablets, laptop computers, smartwatches, etc., may be employed.

[0027] A maintenance technician (250) can perform a visual inspection of an elevator system (300) by operating a smartphone (204) without needing to locate one or more specific areas of the elevator system that may be experiencing technical problems. For example, a visual inspection software application is installed on the smartphone (204), which is configured to display an image or image stream overlaid with diagnostic results obtained using the visual inspection software. The technician can then diagnose the elevator system by analyzing the image or image stream and the overlaid diagnostic results without needing to be physically located within the internal space of the elevator system (e.g., the elevator shaft).

[0028] A visual inspection software application is executed by a processor installed on a smartphone (250) to activate video cameras (316) and microphones (318) and perform visual / auditory inspections along with quantitative measurements and analyses related to the operation of the elevator system (300). For example, the visual inspection software application and the processor work together to compare images and / or sounds acquired to diagnose the operation of the elevator system with images and / or sounds stored in advance. Analysis data generated by the visual inspection software application may be stored locally on the smartphone (204) and / or stored on a cloud server (202). The analysis data includes, but is not limited to, machine noise, machine vibration, elevator car vibration, rope noise, transient / movement noise, idler / deflector noise, rail and guide vibration, and elevator door noise. Analysis data stored on the cloud server (202) may also be archived and tracked over time to determine whether problems identified after recent repair activities have been resolved, or if problems remain or have worsened.

[0029] In one or more embodiments, a visual inspection software application transmits a request to perform a visual inspection test, which is transmitted to a Khan controller (314) and / or a cloud computing network (200). In response to receiving the visual inspection test request, the Khan controller (314) or the cloud computing network (200) initiates video cameras (316) and microphones (318). Initiation includes memory buffering associated with the video cameras (316) and microphones (318) before executing the visual inspection test.

[0030] A visual inspection test request may also include a diagnostic action requested by the user (250). While performing the selected diagnostic action, video cameras (316) and microphones (318) may acquire video, still images, and sound. When the selected diagnostic action is completed, the vehicle controller (314) and / or cloud computing network (200) execute the diagnostic action selected by the user (250), while the video cameras (316) and microphones (318) acquire video frames and / or sound during the ongoing diagnostic action. After buffering is complete, the cloud computing network (200) and / or visual inspection software may perform an analysis of the acquired data to identify one or more anomalies that occurred during the diagnostic test. The analysis may include audio processing to identify possible anomalies, such as defective components, for example. In one or more embodiments, the cloud computing network (200) and / or visual inspection software may determine whether the acquired component is normal or has an anomaly by comparing an image of the component acquired during the diagnostic action with a stored image of a known normally operating component. Similarly, a cloud computing network (200) and / or visual inspection software can compare sounds acquired by a microphone (318) with sounds stored in memory to distinguish between normal operating components and abnormal operating components.

[0031] When an anomaly is detected, a timestamp is determined corresponding to a time period included during the video stream in which the anomaly occurred or within the video frames. Then, a cloud computing network (200) and / or visual inspection software processes the corresponding acquired video frames by generating a video stream and overlaying the video stream with a result graphical user interface (GUI) that displays the results of the diagnostic operation. In one or more embodiments, the result GUI may operate in a diagnostic mode or a metering mode. When the diagnostic mode is invoked, the result GUI overlays graphical status indicators indicating the diagnosed status of one or more specified elevator system components. When the metering mode is invoked, the result GUI overlays graphical status indicators indicating normal or abnormal operating characteristics of the elevator system, such as vibration, maximum noise level, average noise level, etc. It should be understood that even though the diagnostic information and metering information are overlaid according to two separate result GUI modes, the result GUI may operate in a single mode to overlay both the diagnostic information and the metering information simultaneously without departing from the scope of the present invention.

[0032] Now, referring to FIG. 4, a mobile terminal device (204) with a visual inspection software application installed and operating near an escalator system (400) is illustrated according to a non-limiting embodiment. The mobile terminal device (204) will be described as a smartphone (204), which may include, but is not limited to, a computer tablet, a laptop computer, a smart watch, etc. The smartphone (204) includes a network interface device (not illustrated in FIG. 4) that enables data exchange between the smartphone (204) and a cloud computing network (200), or another remote computer or controller (not illustrated). As described herein, the network interface device includes any communication device (e.g., a modem, a wireless network adapter, etc.) that operates according to a network protocol (e.g., Wi-Fi, Ethernet, satellite, cable communication, etc.) that establishes wired and / or wireless communication with the cloud computing network (200), the remote controller, and / or the remote controller.

[0033] The escalator system (400) includes a truss (402) extending between a lower landing (404) and an upper landing (406). A plurality of sequentially connected treadplates or steps (408) are connected to a drive chain (not shown) and move along a closed loop path within the truss (402). A pair of handrails (410) are connected to the truss (402) and are located on opposite sides of the steps (408). The handrails (410) move together with the steps (408) to support passengers. A machine (412) drives the steps (408) and the handrails (410). The machine (412) is typically located in a machine space (414) below one of the landings, for example, contained within the lower landing (404).

[0034] In one or more non-limiting embodiments, the escalator system (400) may include a diagnostic system (450). The diagnostic system (450) may be integrated into one or more steps (408) of the escalator system (400) to provide a diagnostic step (409). In some embodiments, the diagnostic step (409) may be implemented as a fixed component of the escalator system (400). In other embodiments, the diagnostic step (409) may replace the standard escalator step (408) to enable diagnostic operations of the elevator system (400). When the diagnostic operations are completed, the diagnostic step (409) may optionally be removed and replaced with the standard step (408).

[0035] The diagnostic system (450) may include at least one camera (not shown), at least one microphone (not shown), various sensors (e.g., accelerometer, vibration sensor, etc.), and a stair controller (not shown), all of which are integrated within the diagnostic stair (409). The stair controller includes a network interface as described herein. Accordingly, the diagnostic system (450) included in the diagnostic stair (409) may communicate wirelessly with a cloud computing network (200) and / or a smartphone (204), for example, through the stair controller.

[0036] A camera is configured to acquire images of the interior space of the escalator (e.g., the interior space of the escalator truss), while a microphone is configured to acquire sounds such as vibrations, squeaking sounds, etc. In one or more embodiments, the cameras and microphones may collect images and / or sounds of various escalator components, including but not limited to handrail drive systems, tracking systems, and / or sprocket wheels. Additionally, the cameras and microphones may collect diagnostic data (e.g., images and / or sounds) at various locations by acquiring images and / or sounds as the diagnostic stairs (409) move along the track of the escalator system (400). A more detailed description of the diagnostic stairs (409) implementing the diagnostic system (450) is described in U.S. Patent Application Publication 2018 / 0208441, which is assigned to Otis Worldwide Corp. and whose entirety is expressly incorporated herein by reference.

[0037] Referring further to FIG. 4, a maintenance technician or mechanic (250) can diagnose the operation of an escalator system (400) using visual inspection software installed on a smartphone (204) together with a diagnostic system (450) implemented on a diagnostic staircase (409). In this way, the technician or mechanic (250) can visually inspect and diagnose the escalator system (400) (e.g., the interior space of the escalator truss (402)) by analyzing a video stream displayed on the smartphone (204) and an overlaid result GUI without being physically located inside the escalator system (400) (e.g., the interior space of the escalator truss (402)).

[0038] A visual inspection software application can provide quantitative measurement and analysis of the ride quality of the escalator system (400) obtained by a smartphone (204). For example, the visual inspection software application can perform a real-time comparison between the actual performance of the escalator system (400) and its expected performance. The analysis data generated by the visual inspection software application may be stored locally on the smartphone (204) and / or stored on a cloud server of the cloud network (200). The analysis data includes, but is not limited to, machine noise, machine vibration, truss vibration, stair vibration, stair noise, handrail vibration, handrail noise, and drive chain noise. The analysis data stored on the smartphone (204) and / or cloud server may also be archived and tracked over time to determine whether problems identified after recent repair activities have been resolved, or if problems remain or have worsened.

[0039] In one or more non-limiting embodiments, an inspection software application installed on a smartphone (204) communicates with a cloud network (200) and / or a stair controller and, while executing a selected diagnosis, activates the cameras and microphones of the escalator diagnostic system (450) to acquire images and sounds. After the diagnosis is completed, similar to the diagnostic operations described for the elevator system (300) described above, the camera and microphone data (e.g., acquired images and sounds) are buffered, a video stream is generated, and a result GUI is overlaid on the video stream. In one or more embodiments, the diagnostic and / or metering data displayed by the result GUI overlaid on the video stream may be modified based on the selected diagnostic operation. Additionally, timestamps may be generated in response to the detection of anomalies during the diagnostic operation, and notification indicators informing a technician of the detected anomalies may be included in the result GUI as described in detail above. As in the case of the elevator diagnostic tests described above, abnormal or suboptimal noise and vibration may be analyzed and associated with specific product parts or components. Symptoms, diagnoses, and / or measurements are displayed by a result GUI, which is overlaid on an image stream generated according to a diagnostic operation as described above with reference to the elevator system. Additionally, the diagnostic data and / or measurements displayed by the result GUI, which is overlaid on a video stream, change dynamically as the diagnostic stairs move along the track to different locations of the escalator system (400).

[0040] Referring to FIGS. 5A, 5B, 5C, 5D, and 5E, a smartphone (204) operating in various modes to perform visual inspections of an elevator system according to a non-limiting embodiment is illustrated. As described above, the visual inspection software overlays a result GUI according to a diagnostic mode or a metering mode selected by the user. Each mode includes, for example, overlaying a corresponding result GUI on a video stream displayed on a display (220) of a mobile terminal such as a smartphone (204). Although the elevator system is described with reference to FIGS. 5A through 5E, it should be understood that the various modes of the smartphone (204) can be used to diagnose other types of conveyor systems (e.g., escalators) without departing from the scope of the present invention.

[0041] Referring to FIG. 5a, a smartphone (204) displaying a home GUI (500) is shown. The user can operate the home GUI (500) by touching a touch screen actuator (501) displayed on a screen (220) and / or by using other input actuators (not shown) installed on the smartphone (204).

[0042] The home GUI (500) includes a diagnostic selection field (502) and a user instruction field (504). When the video stream contains one or more defects detected after the analysis of the diagnostic operation, the smartphone (204) also displays a notification indicator (554). For example, in the context of an elevator system environment, various diagnostic operations include, but are not limited to, "door cycle" diagnostic operations and "floor-to-floor" diagnostic operations. The "door cycle" diagnostic test executes a test routine to diagnose the operation of the elevator doors. The "floor-to-floor" diagnostic operation executes a test routine to diagnose any abnormal or suboptimal operation of the elevator as it moves within the building.

[0043] In the context of the escalator system environment, the diagnostic selection field (502) allows the user to select from various diagnostic operations, including but not limited to machine diagnostics, ascending stair operation diagnostics, descending stair operation diagnostics, full track operation diagnostics, handrail diagnostics, and braking diagnostics. The diagnostic operations will be described in more detail later.

[0044] The machine diagnostic operation enables the acquisition of noise and / or vibration generated within the machine space and compares the measured data with the expected noise range and / or expected vibration range for the equipment.

[0045] Ascending stair operation diagnostics, descending stair operation diagnostics, and full track operation diagnostics monitor the real-time operation of the escalator system as a user boards along the moving stairs. During these operations, the diagnostics include sound detection of stair and handrail noise, drive chain noise, stair vibration, and stair movement speed.

[0046] More specifically, handrail diagnosis may involve diagnosing problems with the handrails. The diagnosis performed during handrail diagnosis includes analyzing noise generated by the handrails and / or vibrations of the handrails.

[0047] Braking diagnostics can determine whether the escalator steps stop within the expected amount of time following an escalator braking event. The deceleration of the steps can be compared to a threshold to determine whether the escalator's braking function is performing as expected. Braking diagnostics can also diagnose the deceleration of the steps. For example, any pulsation in the deceleration of the steps may be associated with specific parts or components of the braking mechanism.

[0048] The user instruction field (504) displays diagnostic action instructions that instruct the user on how to properly execute the selected diagnostic action. For example, the instructions include instructing the user on the location of the smartphone in relation to the user's person, along with the location regarding the passenger conveyor system, while providing the user with optimal results. For example, when the "door cycle" diagnostic is selected, the user instruction field (504) may instruct the user to "hold the phone at chest height and press start toward the landing door." The instructions presented in the user instruction field (504) may vary based on the diagnostic action selected from the diagnostic selection field (502). For example, when performing the door cycle diagnostic, the instructions may instruct the user to position the smartphone (204) near the elevator doors or near specific components associated with the doors, whereas instructions corresponding to machine diagnostic instructions instruct the user to position the phone within the machine space of the escalator. For example, similar instructions may be provided to the user when performing diagnostic actions associated with other types of passenger conveyor systems, such as escalator systems.

[0049] Referring to FIG. 5b, a smartphone (204) is shown overlaying a diagnostic mode result GUI (510) on a first video frame (505) obtained after an inter-floor operation diagnostic operation performed by visual inspection software. In this example, the first video frame (505) occurs at time 0:58 of the video stream and includes elevator car lifting devices (308) in a normal state, that is, without any detected abnormalities, failures, defects, wear, etc.

[0050] The smartphone (204) also displays a video playback interface (550). The video playback interface (550) includes a progress bar (551) and a progress indicator (552). The progress bar (551) indicates the total duration of the video stream. The progress indicator indicates the current playback duration of the total duration of the video stream. When the video stream contains one or more defects detected after the analysis of the diagnostic operation, the smartphone (204) also displays a notification indicator (554). The notification indicator (554) is displayed at a time in the video stream corresponding to a timestamp corresponding to the anomaly detected during the diagnostic operation. In this way, the user can move (e.g., touch and slide) the progress indicator (552) toward the notification indicator (554) to quickly reach the point in the video stream containing the detected anomaly.

[0051] The diagnostic result GUI (510) includes a component field (512) and a status indicator field (514). The component field (512) lists one or more components that were diagnosed according to a selected diagnostic action (i.e., a diagnostic action selected in the home GUI (500)). One or more of the displayed components may vary depending on the selected diagnostic action. For example, "ropes" and "rails" may be displayed after performing an inter-floor diagnostic action, while "door frame" and "hanger" may be displayed after performing a door cycle diagnostic action. Similarly, after performing one or more escalator system diagnostic actions, escalator components such as "stair chain" and "handrail drive chain" may be displayed.

[0052] The status indicator field (514) displays a status indicator (516a) corresponding to each component name displayed in the component field (512). The status indicator (516a) may be displayed as a unique identifying (e.g., color, shape, etc.) indicating the diagnosed status of the component after the diagnostic operation. For example, a first status indicator (516a) (e.g., a green graphic indicator) may indicate that the corresponding component is operating normally or as expected after the diagnostic operation. A second status indicator (516b) (e.g., a yellow graphic indicator) may indicate that while it still provides appropriate results, the corresponding component has been detected to be operating with one or more irregularities. For example, the elevator car may have moved between one or more floors at a noise level generally acceptable to most passengers, but the noise may include features indicating that the roller bearings are worn and need to be replaced. Thus, a maintenance technician can carefully monitor these units during the next few service visits.

[0053] The result GUI can be configured to change dynamically (i.e., actively) as video playback progresses through the duration of the video stream. For example, FIG. 5b illustrates a result GUI overlaid on a second video frame (507) located at a subsequent time, for example, 1 minute 39 seconds (1:39) of the video stream. In the second video frame (507), an anomaly (518) present in one of the lifting devices (308) and acquired during the initial diagnosis is played during the playback of the video stream. Notification of the detected anomaly (518) is also indicated by a notification indicator (554) at the timestamp of the video stream (e.g., 1:39). The result GUI (510) can also be modified to reflect the detected anomaly (518). Accordingly, the result GUI (510) now includes a third situation indicator (516c) (e.g., a red graphic indicator) indicating which components are associated with the detected anomaly. These results may include, for example, vibration or noise exceeding a critical level for passenger comfort. Accordingly, a technician may determine that the elevator system and / or components require additional analysis or service without the need to physically be located in the detected abnormal area.

[0054] Referring to FIGS. 5d and 5e, an overlaid result GUI (510) operating in metering mode is illustrated. In this example, the result GUI (510) operates similarly to when operating in the diagnostic mode described above. However, the information overlaid on the video stream is different. For example, FIG. 5d illustrates the first video frame (505) described above with the result GUI (510) operating in metering mode overlaid. The first video frame (505) shows elevator devices (308) that are free of any abnormalities. Accordingly, the result GUI (510) displays various measured meters corresponding to the elevator system and related components shown in the first video frame (505). The measured meters may include, but are not limited to, maximum vibration, minimum vibration, average vibration, maximum noise level, minimum noise level, and average noise level. As described herein, the diagnostic result analysis may compare the measured meters with various threshold values. Accordingly, when the measured meter does not exceed the corresponding threshold value, a first status indicator (516a) (e.g., a green graphic indicator) is displayed to indicate that the displayed measured meter represents expected and normal operating states.

[0055] Referring to FIG. 5e, the result GUI is overlaid on the second video frame (507). As described above, the second video frame (507) shows an anomaly (518) present in one of the lifting devices (308) previously acquired by the video camera during the initial diagnostic operation. Accordingly, the displayed situation indicators may be dynamically changed to reflect the dynamic change of the measured meter acquired at a corresponding time in the video stream (e.g., 1:39). In this example, second situation indicators (516b) (yellow graphic indicators (516b)) are displayed indicating that the corresponding displayed measured meter is approaching an abnormal or unacceptable operating state, and third situation indicators (516c) (red graphic indicators (516c)) are displayed indicating that the corresponding displayed measured meter has exceeded an abnormal or unacceptable operating state. Additionally, the overlaid data may include location data indicating where the anomaly shown in the current video frame (e.g., the second frame (507)) can be physically located in the elevator system. Accordingly, a technician can determine that additional analysis or service is required without the need for the elevator system and / or components to be physically located in the detected anomaly area.

[0056] Now, referring to FIG. 6, a flowchart illustrates a method for diagnosing a passenger conveyor system using a visual inspection software application installed on an electronic mobile terminal device according to a non-limiting embodiment. The method begins with operation 600, and in operation 602, a diagnostic operation is selected from one or more available diagnostic operations. In step 604, diagnostic operation instructions corresponding to the selected diagnosis are displayed on the screen of the mobile terminal device. In operation 606, the selected diagnostic operation is initiated, and in operation 608, video cameras and / or microphones are buffered.

[0057] In operation 608, a selected diagnostic operation is performed, and in operation 610, real-time ride comfort data of the passenger conveyor system is acquired. The ride comfort data may include video frames, images, noises, sounds, and vibrations acquired during the real-time operation of the passenger conveyor system during the executed diagnostic operation. In operation 612, a determination is made as to whether the diagnosis is complete. When the diagnostic operation is not complete, the method returns to operation 610 and continues to acquire real-time data.

[0058] However, when the diagnostic operation is completed in operation 612, the method proceeds to operation 614 and analyzes the images and sounds acquired during the diagnostic operation. The images and sounds are analyzed using various algorithms and processing applications. For example, images and / or sounds acquired in real time may be compared with stored images and sounds associated with normal operating states to determine whether an abnormal image or abnormal sound is present. In some examples, vibrations and / or sounds may be compared with a threshold value for detecting an abnormal state.

[0059] In operation 616, a decision is made regarding whether an anomaly is detected based at least partially on the analysis of the results of the diagnostic operations, and a video stream including images and sounds acquired in operation 618 is generated. However, when an anomaly is detected, a time stamp corresponding to the detected anomaly is determined in operation 620. The time stamp includes the time stamp where the anomaly occurred in the video stream.

[0060] Referring to operation 622, a result GUI is generated based at least partially on the analysis of the diagnostic results. In operation 624, a video stream is displayed on a display screen (e.g., a smartphone screen), and a result GUI is overlaid on the video stream. When one or more abnormalities are detected in operation 616, a notification indicator based at least partially on the corresponding timestamp is added to the playback duration bar displayed on the display screen. In operation 626, the video stream is played (also referred to as video stream playback). Accordingly, the displayed result GUI indicates the status of the elevator system at the current playback duration of the video stream, and the operation ends in operation 628.

[0061] Accordingly, the various non-limiting embodiments described herein provide a visual inspection system that enables a technician to diagnose a passenger conveyor system without the need for the technician to be physically located in a specific area of ​​the conveyor system. For example, the technician can view a video stream with a result GUI overlaid to identify anomalies within the internal space (e.g., an elevator shaft or an internal escalator space) without being physically located within the internal space of the conveyor system. In this way, the safety of the technician is improved, and service response time can also be improved.

[0062] The terms used herein are merely for describing specific embodiments and are not intended to limit the disclosure. As used herein, singular forms (“one,” “one,” and “this”) are intended to include plural forms unless the context clearly indicates otherwise. It will also be understood that the terms “comprising” and / or “comprising,” as used herein, specify the presence of the mentioned features, integers, steps, actions, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, actions, elements, components, and / or groups thereof.

[0063] Although the present disclosure has been described with reference to representative embodiments or examples, it will be understood by those skilled in the art that various modifications may be made and equivalents may replace elements thereof without departing from the scope of the present disclosure. Furthermore, many variations may be made to adapt a particular situation or material to the teachings thereof without departing from the basic scope of the present disclosure. Accordingly, the present disclosure is not limited to the specific embodiments disclosed as the optimal mode considered for carrying out the present disclosure, but is intended to include all embodiments falling within the scope of the claims.

Claims

Claim 1 A passenger conveyor diagnostic system comprising: a processor storing at least one visual inspection diagnosis; and at least one sensor configured to acquire ride comfort data including one or both of an image and a sound, wherein the processor applies the at least one visual inspection diagnosis to one or both of the image and the sound to generate a diagnosed situation of the passenger conveyor system and overlays the diagnosed situation of the passenger conveyor system onto the image. Claim 2 A passenger conveyor diagnostic system according to claim 1, wherein the processor performs the at least one visual inspection diagnosis based at least partially on one or both of the image and the sound, generates a video stream based on one or both of the image and the sound, overlays a result graphical user interface (GUI) on the video stream, and the result GUI represents the diagnosed situation of the passenger conveyor system. Claim 3 A passenger conveyor diagnostic system according to claim 2, further comprising a mobile terminal device that communicates signals with one or both of the processor and the at least one sensor, wherein the mobile terminal device is configured to display the video stream and the result GUI overlaid on the video stream. Claim 4 A passenger conveyor diagnostic system according to claim 3, wherein the result GUI changes dynamically in response to a detected anomaly of the passenger conveyor system played during the playback of the video stream. Claim 5 A passenger conveyor diagnostic system according to claim 1, wherein the passenger conveyor system is one of an elevator system or an escalator system. Claim 6 A passenger conveyor diagnostic system according to claim 5, wherein the ride comfort data includes at least one of machine noise, elevator door noise, elevator car noise and elevator car vibration, stair vibration, stair noise, drive chain noise, handrail noise, handrail vibration, stair speed and braking speed. Claim 7 A passenger conveyor diagnostic system according to claim 2, wherein the result GUI operates in a first mode for displaying data configured to diagnose a predetermined component included in the passenger conveyor system, and in a second mode for displaying a measured meter obtained in response to performing the at least one visual inspection diagnosis. Claim 8 A passenger conveyor diagnostic system according to claim 3, wherein the mobile terminal device is one of a smartphone, a computer tablet, or a smart watch. Claim 9 A passenger conveyor diagnostic system according to claim 1, wherein the processor is installed in a cloud computing network. Claim 10 In claim 3, the processor is a passenger conveyor diagnostic system installed in the mobile terminal device. Claim 11 A mobile terminal device configured to diagnose a passenger conveyor system, comprising: a memory unit storing a visual inspection application configured to perform at least one visual inspection diagnosis; a network interface device establishing communication with at least one sensor installed on the passenger conveyor system; and an electronic hardware controller that applies the at least one visual inspection diagnosis to ride data including one or both of images and sounds obtained from the at least one sensor to determine at least one diagnosis result of the passenger conveyor system, and displays the at least one diagnosis result overlaid on the image. Claim 12 A mobile terminal device according to claim 11, wherein the controller generates a video stream based on one or both of the images and the sounds received from the at least one sensor, overlays a result graphical user interface (GUI) on the video stream, and the result GUI displays the at least one diagnostic result of the passenger conveyor system displayed on the video stream. Claim 13 A mobile terminal device according to claim 12, wherein the result GUI changes dynamically in response to a detected anomaly of the passenger conveyor system played during the playback of the video stream. Claim 14 A mobile terminal device according to claim 12, wherein the ride comfort data includes at least one of elevator machine noise, elevator door noise, elevator car noise, elevator car vibration, escalator machine noise, escalator stair vibration, escalator stair noise, escalator stair speed, escalator handrail noise, escalator handrail vibration, and escalator braking speed. Claim 15 A mobile terminal device according to claim 13, wherein the controller initiates a first mode for generating a first-type result graphical user interface (GUI) configured to display data configured to diagnose a predetermined component included in the passenger conveyor system, and a second mode for generating a second-type result GUI configured to display a measured meter. Claim 16 A mobile terminal device according to claim 15, wherein the first-type result GUI displays the name of at least one component of the component included in the video stream. Claim 17 A mobile terminal device according to claim 16, wherein the measured metric displayed by the second-type result GUI is obtained in response to performing the at least one visual inspection diagnosis. Claim 18 In claim 11, the mobile terminal device is a mobile terminal device that is one of a smartphone, a computer tablet, or a smart watch. Claim 19 A method for diagnosing a passenger conveyor system, comprising: storing at least one visual inspection diagnosis through a processor; acquiring one or both of an image and a sound through at least one sensor installed on the passenger conveyor system; applying the at least one visual inspection diagnosis to one or both of the image and the sound through the processor to generate a diagnosed situation of the passenger conveyor system; and overlaying the diagnosed situation of the passenger conveyor system onto the image through the processor. Claim 20 The method of claim 19, wherein the step of overlaying the image comprises: performing the at least one visual inspection diagnosis based on one or both of the image and the sound through the processor; generating a video stream based on one or both of the image and the sound through the processor; and overlaying a result graphical user interface (GUI) on the video stream through the processor, wherein the result GUI represents the diagnosed situation of the passenger conveyor system.

Images