Maintenance device, maintenance system, and maintenance program
Patent Information
- Application Number
- JP2023081525
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-17
- Publication Date
- 2024-11-28
AI Technical Summary
【0010】 上記構成の本発明によれば、保守点検時に、例えば、昇降路内の小さな異物、機器の微細の位置変化等の事象発生を、高精度で且つ低コストで検出することができる。
Smart Images

Figure 2024165376000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a maintenance device, a maintenance system, and a maintenance program. [Background technology]
[0002] Conventionally, various systems for inspecting elevator hoistways have been proposed (see, for example, Patent Document 1). Patent Document 1 proposes a system in which multiple cameras are installed inside the elevator hoistway, above and below the car, and the system compares current images of the hoistway taken by the cameras with images of the hoistway taken in the past to determine whether maintenance is required. When such a system is adopted, it becomes possible to automatically inspect for abnormalities even in buildings such as high-rise buildings where visual inspection is difficult. In addition, since the condition inside the hoistway can be confirmed without an operator visiting the site, the burden of maintenance work can be significantly reduced. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] European Patent Application Publication No. 3409629 Summary of the Invention [Problem to be solved by the invention]
[0004] As described above, a system has been proposed that detects changes in a hoistway by comparing a current image of the hoistway taken by a camera with a previously taken image of the hoistway. In such a system, in order to accurately detect, for example, small foreign objects in the hoistway or minute changes in the position of equipment, it is preferable that the photographing point (photographing position) of the current image of the hoistway and the photographing point of the previous image to be compared therewith are the same. In the system disclosed in the above Patent Document 1, a method is adopted in which a physical alignment mark formed by painting or the like is used to accurately align the photographing point of the current image of the hoistway with the photographing point of the previous image of the hoistway.
[0005] However, in the system disclosed in Patent Document 1, for example, when the alignment mark itself deteriorates due to aging or accumulation of dirt, alignment itself becomes difficult. In addition, in the system disclosed in Patent Document 1, it is necessary to install multiple alignment marks corresponding to the installation positions of multiple cameras, and the installation costs are high, especially when the system is applied to a high-rise building. Therefore, in a conventional system for performing maintenance and inspection by photographing the state inside an elevator hoistway with a camera, there is a demand for the development of a technology that can detect the occurrence of events such as small foreign objects in the hoistway and minute position changes of equipment with high accuracy and low cost.
[0006] The present invention has been made to solve the above problems. An object of the present invention is to provide a technology that can detect the occurrence of events, such as small foreign objects in an elevator shaft or minute changes in the position of equipment, with high accuracy and low cost in a maintenance device and maintenance system that performs maintenance and inspection by photographing the state inside an elevator shaft with a camera. [Means for solving the problem]
[0007] In order to solve the above problem, the maintenance device of the present invention is a maintenance device that performs inspection and maintenance of a hoistway based on first image data of the hoistway photographed by a photographing device during maintenance and inspection of an elevator and second image data of the hoistway to be compared with the first image data. The maintenance device includes an image analysis unit that performs image analysis of the hoistway based on the first image data and the second image data, and the image analysis unit has a difference area calculation unit and a deviation amount calculation unit. The difference area calculation unit generates a first difference image between an image included in the first image data and an image included in the second image data, and calculates the area of a difference region included in the first difference image. In addition, the deviation amount calculation unit calculates the amount of deviation between the photographing point of each image included in the first image data and the photographing point of each image included in the second image data based on the area of the difference region calculated by the difference area calculation unit.
[0008] In order to solve the above problems, the maintenance system of the present invention is a maintenance system including an image capturing device that captures first image data of a hoistway during elevator maintenance and inspection. The maintenance system performs inspection and maintenance of the hoistway based on the first image data and second image data of the hoistway that is compared with the first image data. The maintenance system further includes an image analysis device having a similar configuration to the maintenance device of the present invention, and a display device that displays the results of image analysis by the image analysis device.
[0009] Furthermore, in order to solve the above problems, the maintenance program of the present invention is a maintenance program that implements and executes the processing performed by an image analysis unit provided in the maintenance device of the present invention in the maintenance device. Effect of the Invention
[0010] According to the present invention having the above configuration, during maintenance and inspection, the occurrence of events such as small foreign objects in a hoistway and minute changes in the position of equipment can be detected with high accuracy and at low cost. [Brief description of the drawings]
[0011] [Figure 1] 1 is a schematic configuration diagram of an elevator maintenance system according to one embodiment of the present invention. [Diagram 2] 2 is a functional block diagram of a photographing device constituting an elevator maintenance system according to one embodiment of the present invention. FIG. [Diagram 3] FIG. 2 is a functional block diagram of a calculation device constituting the elevator maintenance system according to one embodiment of the present invention. [Figure 4] FIG. 2 is a hardware configuration diagram of a computing device that constitutes the elevator maintenance system according to one embodiment of the present invention. [Diagram 5] 13 is a diagram for explaining a method for calculating the amount of deviation of the shooting points between a current image and a reference image, which is performed by a calculation device of an elevator maintenance system according to one embodiment of the present invention. FIG. [Figure 6]13 is a diagram for explaining a method for calculating the amount of deviation of the shooting points between a current image and a reference image, which is performed by a calculation device of an elevator maintenance system according to one embodiment of the present invention. FIG. [Figure 7] 10 is a flowchart showing the steps of a process for calculating the amount of deviation between the shooting points of a current image and a reference image, which is performed by a calculation device of an elevator maintenance system according to one embodiment of the present invention. [Figure 8] 10 is a flowchart showing the steps of a differential image generation process performed by a calculation device of an elevator maintenance system according to one embodiment of the present invention. [Figure 9] FIG. 13 is a diagram showing an example of an image analysis result displayed on a display device of an elevator maintenance system according to one embodiment of the present invention. [Figure 10] FIG. 11 is a diagram showing an image analysis result (example) of foreign object detection displayed on the display device of the elevator maintenance system according to one embodiment of the present invention. [Figure 11] FIG. 11 is a diagram showing an image analysis result (comparative example) of foreign object detection displayed on the display device of the elevator maintenance system according to one embodiment of the present invention. [Figure 12] FIG. 11 is a diagram showing another display mode (variation example) of the image analysis result displayed on the display device of the elevator maintenance system according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] An elevator maintenance device and a maintenance system according to an embodiment of the present invention will be specifically described below with reference to the drawings. The degree of reduction in the load on maintenance workers obtained by the maintenance system of the present invention increases as the number of floors in a building increases. Therefore, the embodiment described below assumes that the elevator is installed in a building with six or more floors.
[0013] [Overall configuration of elevator maintenance system] Fig. 1 is a block diagram of an elevator maintenance system according to an embodiment of the present invention. For the sake of simplicity, only components related to control of image capture, image analysis, display of analysis results, etc., performed during maintenance and inspection of an elevator hoistway are shown. The example shown in Fig. 1 shows a state in which an elevator car 21 capable of ascending and descending within a hoistway 20 is stopped at the height of a landing door 22 on the first floor.
[0014] 1, the maintenance system 10 includes a camera 1 that captures images of the interior of the elevator shaft 20 of the car 21 during maintenance and inspection, a calculation device 2 that analyzes images captured by the camera 1 to detect changes in the situation, and a display device 3 that displays the analysis results of the calculation device 2. In this embodiment, the camera 1 is installed on the underside of the elevator car 21, for example, but it may be placed in any position as long as it is capable of capturing an image of the entire interior of the elevator shaft 20. For example, the camera 1 may be installed on the top or side of the car 21.
[0015] The arithmetic device 2 and the display device 3 are provided outside the building in which the elevator is installed. The arithmetic device 2 (maintenance device, image analysis device) may be, for example, a calculation processing device such as a personal computer having a calculation function and a communication function managed by an operator such as a maintenance worker, or the arithmetic device 2 may be provided in an external physical server or cloud. The arithmetic device 2 is connected to the imaging device 1 via a wireless and / or wired communication line (the communication path may include a wide area communication network). The arithmetic device 2 is also connected to the display device 3 via a wireless or wired communication line. In the example shown in FIG. 1, an example in which the arithmetic device 2 and the display device 3 are provided separately will be described, but the present invention is not limited to this, and both may be configured integrally, in which case the maintenance device is configured by the arithmetic device 2 and the display device 3.
[0016] [Configuration of imaging equipment] Fig. 2 is a functional block diagram of the photographing device 1. As shown in Fig. 2, the photographing device 1 has a camera 31, a light 32, an photographing processing unit 33, and an input unit 34. In the photographing device 1, the photographing processing unit 33 is connected to the camera 31 and the input unit 34. In addition, the photographing processing unit 33 (announcement unit 33b described later) is connected to the calculation device 2 (receiving unit 41 described later) via a wireless and / or wired communication line (the communication path may also include a wide area communication network).
[0017] During maintenance and inspection, instruction signals to start and end shooting are sent from the computing device 2 to the photographing device 1, and based on these signals, the photographing operation of the camera 31 and the lighting operation of the light 32 are controlled. In order to simplify the explanation, the connection (arrows) between the functional blocks related to this process are omitted in Fig. 2.
[0018] The camera 31 captures an image of the elevator hoistway 20. The image captured by the camera 31 may be a video or a still image captured continuously. The latter shooting operation may be, for example, a configuration in which still images are continuously captured at a predetermined interval while the car 21 is traveling. Specifically, when the maintenance inspection target is detection of foreign objects, fallen objects, etc. in the hoistway 20, a still image may be captured each time the car 21 travels to the 1st floor, 1.5th floor, 2nd floor, 2.5th floor, etc., that is, once for every half floor. In this case, the timing of shooting can be determined by the relationship between the distance between each floor of the elevator and the traveling speed of the car 21. For example, if the floors are 5 m apart and the moving speed of the elevator (car 21) is 60 m per minute, when shooting once for every half floor, it is sufficient to shoot once every 2.5 seconds. In this case, the amount of image data can be reduced compared to video shooting.
[0019] In this embodiment, the camera 31 is a camera equipped with a fisheye lens, but the present invention is not limited to this. For example, a lens with an arbitrary angle of view (wide-angle lens) can be used as long as it is capable of capturing an image of the entire interior of the elevator shaft 20.
[0020] Furthermore, for example, a visible light camera, an infrared camera, etc. can be used as the camera 31. When a visible light camera is used as the camera 31, image data including multidimensional information of red (R), green (G), and blue (B) can be obtained, enabling more accurate judgment during maintenance and inspection.
[0021] Light 32 is provided to illuminate the inside of elevator shaft 20, which is a very dark environment. Light 32 is installed near camera 31. As light 32, a light that irradiates light suitable for the wavelength band of camera 31 to be used is used. If camera 31 is a visible light camera, a visible light light that can irradiate visible light is used, and if camera 31 is an infrared camera, an infrared light that can irradiate infrared light is used.
[0022] When a visible light camera is used as the camera 31, a visible light is used as the light 32, and therefore if a vision glass is provided on the landing door 22, the light of the light 32 illuminating the inside of the hoistway 20 during the maintenance and inspection work of the elevator will reach the building side as well. In this case, the appearance of the car 21 running with the light 32 illuminating it may be observed by passengers, and from the passenger's point of view, the appearance of the car 21 running with the light 32 illuminating it may seem strange. On the other hand, when an infrared camera is used as the camera 31, an infrared light is used as the light 32, and therefore the light of the light 32 is not observed by passengers during the maintenance and inspection work of the elevator, and the above-mentioned situation can be avoided.
[0023] The photographing processing unit 33 is composed of an arithmetic processing device having a calculation function and a communication function for performing various controls related to photographing control. Specifically, the photographing processing unit 33 is composed of a computer device having a built-in CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), and a network interface. The photographing processing by the photographing processing unit 33, the transmission processing of image data, and other controls are executed by the CPU included in the arithmetic processing device.
[0024] 2, the photographing processing unit 33 functionally includes a photographing condition setting unit 33a and a notification unit 33b. The photographing condition setting unit 33a sets photographing conditions such as the shutter speed, aperture value, ISO sensitivity, and frame rate of the camera 31. The notification unit 33b transmits images (moving images / continuously photographed still images) photographed by the camera 31 to a receiving unit 41 in the computing device 2, which will be described later.
[0025] The input unit 34 is a user interface (UI) that accepts operations and inputs from an operator. The imaging conditions are set in the imaging condition setting unit 33a by the operator's input operation to the input unit 34.
[0026] [Configuration of the computing device] Fig. 3 is a functional block diagram of the calculation device 2. As shown in Fig. 3, the calculation device 2 has a receiving unit 41, an image analysis unit 42, and an input unit 43. The image analysis unit 42 is connected to the display device 3, the receiving unit 41, and the input unit 43. The image analysis unit 42 is also connected to the reference image database 40 via a wireless and / or wired communication line (the communication path may also include a wide area communication network). Although not shown in Fig. 3, the calculation device 2 has a reference image receiving unit for receiving reference image data, described below, transmitted from the reference image database 40.
[0027] The reference image database 40 stores a plurality of image data (hereinafter referred to as "reference image data") of the elevator shaft 20 taken in the past under normal conditions. Note that the reference image database 40 stores, for example, reference image data acquired by driving the car 21 in all possible driving patterns during maintenance and inspection for each driving pattern. In grouping the reference image data in the reference image database 40, the data groups for each driving pattern may be further subdivided by shooting conditions (exposure conditions).
[0028] The reference image database 40 may be provided inside the calculation device 2, or may be provided outside the calculation device 2. In the latter case, the reference image database 40 may be provided, for example, on a server of an elevator manufacturer, in a cloud, etc. The reference image data stored in the reference image database 40 is input to a difference area calculation unit 42a (described later) of the image analysis unit 42.
[0029] The receiving unit 41 is an interface that receives image data (hereinafter referred to as “current image data”) of the elevator shaft 20 during maintenance and inspection, transmitted from the notification unit 33b in the imaging device 1. The receiving unit 41 then outputs the received current image data to a difference area calculation unit 42a (to be described later) of the image analysis unit 42.
[0030] As shown in FIG. 3, the image analysis unit 42 has, as functional blocks, a difference area calculation unit 42a (difference area calculation unit), a shooting point adjustment amount calculation unit 42b (deviation amount calculation unit), a difference processing unit 42c, a noise removal unit 42d, and a contour extraction unit 42e.
[0031] The difference area calculation unit 42a calculates information on the area of a difference region between an image included in the reference image data (hereinafter referred to as a "reference image") and an image included in the current image data (hereinafter referred to as a "current image") based on the input reference image data and current image data. Then, the difference area calculation unit 42a outputs the information on the calculated area of the difference region to the shooting point adjustment amount calculation unit 42b.
[0032] Here, an overview of the calculation process of information regarding the difference area performed by the difference area calculation unit 42a will be described. First, the difference area calculation unit 42a extracts a predetermined number of frames ("F" described below) of temporally consecutive reference images from the reference image data as a reference image group. Next, the difference area calculation unit 42a extracts, from the current image data, current images that are temporally consecutive and have the same number of frames as the predetermined number of frames of the reference images included in the reference image group as the current image group.
[0033] Next, the differential area calculation unit 42a calculates, for each reference image included in the reference image group, the area of the differential region between the reference image and the current image in the current image group that is the target of differential processing (hereinafter referred to as the "difference area"), and then calculates the sum of the differential areas calculated for each reference image (hereinafter referred to as the "total differential area").
[0034] Thereafter, the differential area calculation unit 42a shifts the extraction position (frame position) of the current image group in the current image data by one frame, and extracts a predetermined number of frames of the current image group from the current image data again.Then, the differential area calculation unit 42a performs the above-mentioned series of arithmetic processing between the reference image group and the newly extracted current image group to calculate the total differential area.
[0035] In this embodiment, the difference area calculation unit 42a repeats the above-mentioned series of calculation processes of the sum difference area a predetermined number of times while changing the extraction position of the current image group in the current image data.Then, the difference area calculation unit 42a outputs the calculated sum difference area for the predetermined number of times to the shooting point adjustment amount calculation unit 42b as information on the area of the difference region.The details of the above-mentioned series of calculation processes of the sum difference area performed by the difference area calculation unit 42a will be described in detail later with reference to the drawings.
[0036] The shooting point adjustment amount calculation unit 42b calculates the amount of deviation of the shooting point in units of the number of frames between the reference image data and the current image data from the information on the area of the difference region input from the difference area calculation unit 42a (total difference area for a predetermined number of times). Note that the positive / negative of this deviation amount varies depending on, for example, the type of image data (reference image data or current image data) that is the basis of the deviation amount, the calculation conditions of the calculation process of the total difference area (reference current image number c and reference reference image number r described later), etc. Then, the shooting point adjustment amount calculation unit 42b outputs the calculated deviation amount of the shooting point between the reference image data and the current image data to the difference processing unit 42c as the shooting point adjustment amount (optimum adjustment amount). Note that the details of the calculation process of the shooting point adjustment amount performed by the shooting point adjustment amount calculation unit 42b will be described later in detail with reference to the drawings.
[0037] The difference processing unit 42c adjusts the shooting point (frame number) of one of the current image and the reference image based on the shooting point adjustment amount input from the shooting point adjustment amount calculation unit 42b to create a difference image between the two (second difference image). Specifically, the difference processing unit 42c generates a difference image between, for example, a reference image of a predetermined frame number and a current image of a frame number shifted from the predetermined frame number by the shooting point adjustment amount. That is, the difference processing unit 42c extracts a current image having a shooting point that is the same as or closest to the shooting point of the reference image of the predetermined frame number from the current image data based on the shooting point adjustment amount, and generates a difference image between the two. Then, the difference processing unit 42c outputs a group of calculated difference images (hereinafter referred to as "difference image data") to the noise removal unit 42d.
[0038] The noise removal unit 42d performs a noise removal process on each difference image included in the difference image data input from the difference processing unit 42c. In this noise removal process, noise components that appear as small, discontinuous points included in the difference image are removed. For example, an averaging filter process or a morphology process can be used as a noise removal method. Then, the noise removal unit 42d outputs the difference image data that has been subjected to the noise removal process to the contour extraction unit 42e.
[0039] The contour extraction unit 42e extracts the contours of consecutive difference regions in each difference image included in the difference image data input from the noise removal unit 42d, and generates an image of a contour line (hereinafter referred to as a "contour image") to be superimposed on the difference image. The contour extraction unit 42e then outputs the generated group of contour images (hereinafter referred to as "contour image data") to the display device 3. At this time, the contour extraction unit 42e also outputs the difference image data input from the noise removal unit 42d to the display device 3. Note that in this embodiment, the contour extraction unit 42e is provided to highlight and display the difference regions, but the present invention is not limited to this, and the contour extraction unit 42e may not be provided.
[0040] In this embodiment, not only the difference image data and contour image data but also the current image data received from the photographing device 1 and the reference image data acquired from the reference image database 40 are output from the image analysis unit 42 to the display device 3. Furthermore, the image analysis unit 42 also outputs to the display device 3 information related to the calculation conditions for the calculation process of the sum difference area performed by the difference area calculation unit 42a and information related to the calculated sum difference area.
[0041] The input unit 43 is a user interface (UI) that accepts operations and inputs from an operator. Various calculation conditions required for the calculation process of the sum difference area performed by the difference area calculation unit 42a are set by the operator's input operation to this input unit 43. Therefore, by the operator's operation to the input unit 43, various calculation conditions required for the calculation process of the sum difference area can be set to appropriate conditions depending on, for example, the type of equipment and parts in the elevator shaft 20 to be analyzed, the processing capacity of the calculation device 2, etc.
[0042] [Display device configuration] The display device 3 is composed of, for example, a liquid crystal panel. The display device 3 displays a reference image, a current image, and a difference image based on the reference image data, the current image data, the difference image data, and the contour image data input from the calculation device 2 (image analysis unit 42) (see FIG. 9 described later). At this time, the display device 3 displays a corresponding contour image superimposed on the difference image. In this case, the difference area is emphasized by the contour image on the display screen of the display device 3, making it easier for the operator to find the difference area. The display device 3 also displays the calculation conditions for the calculation process of the sum difference area input from the calculation device 2 (image analysis unit 42) and various information related to the calculated sum difference area (see FIG. 9 described later).
[0043] In this embodiment, the display device 3 may be configured as a touch panel, in which case the input unit 43 in the arithmetic device 2 is configured integrally with the display device 3. In this case, when the arithmetic device 2 and the display device 3 are configured separately as shown in Fig. 1, the input unit 43 is not included in the arithmetic device 2 (maintenance device). On the other hand, when the input unit 43 and the display device 3 are configured integrally, and when the arithmetic device 2 and the display device 3 are configured integrally, both the input unit 43 and the display device 3 become components of the arithmetic device 2 (maintenance device).
[0044] [Hardware configuration of the computing device] Here, a hardware configuration of a computer device applicable as the arithmetic device 2 will be described with reference to Fig. 4. Fig. 4 is a block diagram showing an example of a hardware configuration of a computer device 100 applicable as the arithmetic device 2.
[0045] The computer device 100 includes a CPU 101, a ROM 102, a RAM 103, a non-volatile storage 104, a network I / F (interface) 105, and an input unit 106, all of which are connected to a bus line 108. Although not shown, the computer device 100 also includes various interfaces used when executing input / output processing of various data (various information) between the computer device 100 and external devices.
[0046] The CPU 101 reads out program codes of software for implementing various processing functions of the arithmetic device 2 from the ROM 102 to the RAM 103 and executes the programs. At this time, variables and parameters generated during the arithmetic processing are also temporarily written to the RAM 103. The CPU 101 also reads out and executes the corresponding program codes of various processes of the difference area calculation unit 42a, the shooting point adjustment amount calculation unit 42b, the difference processing unit 42c, the noise removal unit 42d, and the contour extraction unit 42e included in the image analysis unit 42 described above. That is, the image analysis unit 42 (the difference area calculation unit 42a, the shooting point adjustment amount calculation unit 42b, the difference processing unit 42c, the noise removal unit 42d, and the contour extraction unit 42e) are functionally included in the CPU 101.
[0047] It is also possible to realize some or all of the various components included in the image analysis unit 42 by hardware. For example, the arithmetic device 2 may be configured using a custom IC such as an ASIC (Application Specific Integrated Circuit) or a programmable IC such as an FPGA (Field-Programmable Gate Array), and a circuit design may be performed to realize the functions of the various components included in the image analysis unit 42.
[0048] The non-volatile storage 104 can be configured with, for example, a hard disk drive (HDD), a solid state drive (SSD), a flexible disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a non-volatile memory, etc. In addition to an operating system (OS) and various parameters, the non-volatile storage 104 stores a program for making the computer device 100 function as the arithmetic device 2. Therefore, the reference image database 40 may be provided in the non-volatile storage 104. Note that information (data) such as programs, tables, and files that realize the various functions of the arithmetic device 2 may be stored in a recording medium such as an IC card, an SD card, or a DVD, in addition to the ROM 102 or the non-volatile storage 104.
[0049] The network I / F 105 is configured by, for example, a network interface card (NIC) and transmits and receives various data to and from each device connected via wireless communication. The receiving unit 41 of the arithmetic device 2 is included in the network I / F 105.
[0050] The input unit 106 is configured with, for example, keys and buttons, generates an operation signal according to the operation content input by an operator, and supplies the operation signal to the CPU 101. The input unit 43 of the arithmetic device 2 is included in the input unit 106.
[0051] Note that the arithmetic device 2 may not include the input unit 106 depending on the configuration of the display device 3. Furthermore, the arithmetic device 2 may include a GPU (Graphics Processing Unit) in addition to the components shown in FIG.
[0052] [Outline of the calculation method for the shooting point adjustment amount] Next, the amount of deviation between the reference image data and the current image data calculated by the difference area calculation unit 42a and the shooting point adjustment amount calculation unit 42b in units of the number of frames, i.e., the calculation method of the shooting point adjustment amount will be described in detail with reference to the drawings. Note that, in the following, an example will be described in which image data (image data with no deviation) that is the basis of the shooting point adjustment amount is used as the reference image data. The calculation method of the shooting point adjustment amount when the image data that is the basis of the shooting point adjustment amount is the current image data is simply replaced with the reference image data in the calculation method described below, so the description thereof will be omitted here.
[0053] Fig. 5 is a diagram showing the structure of reference image data acquired by the differential area calculation unit 42a from the reference image database 40, and the structure of current image data received by the differential area calculation unit 42a from the photographing device 1. Fig. 6 is a diagram showing the relationship between a reference image group of a predetermined number of frames to be subjected to difference calculation and a current image group, and the relationship between the difference image calculated between both image groups and the total difference area of each difference image.
[0054] In the calculation method of the shooting point adjustment amount described below, as shown in FIG. 5, the number of frames (number of images) of the reference image data (second image data) is M, and the reference image data is a reference image P 1 ~P M In addition, the number of frames of the current image data (first image data) is N, and the current image data is a current image R. 1 ~R N Here, the indexes added to the reference image P and the current image R are frame numbers within the respective image data, and hereinafter, these indexes will be referred to as "image numbers."
[0055] In addition, in the method of calculating the shooting point adjustment amount described below, the number of frames of images included in each of the reference image group and the current image group that are the subject of the difference calculation (hereinafter referred to as the "predetermined number of frames") is "F".
[0056] Furthermore, in the method of calculating the shooting point adjustment amount described below, the calculation process of the difference image and the total difference area between the reference image group and the current image group is repeatedly executed while sliding the current image group (reference current image number "c" described below) one frame (predetermined interval) relative to the reference image group. At this time, the maximum sliding amount (hereinafter also referred to as "frame adjustment amount") of the current image group (reference current image number "c" described below) relative to the reference image group is set to "±A" frames. In this case, the difference calculation between the reference image group and the current image group is repeatedly executed 2A+1 times (predetermined number of times), and 2A+1 total difference areas are calculated.
[0057] First, the difference area calculation unit 42a calculates the reference image data (reference image P 1 ~P M A reference image (hereinafter, a "reference reference image P") is used as a reference when extracting a reference image group of a predetermined number of frames F from the r The image number "r" (hereinafter referred to as the reference reference image number "r") of the first image is determined.
[0058] Standard reference image P r is selected so that the reference image group includes a reference image that is considered to include a portion (maintenance inspection target) required for image analysis in the maintenance and inspection of the elevator shaft 20. Therefore, the standard reference image P r is appropriately selected depending on the device or part in the elevator shaft 20 that is the subject of image analysis. For example, a reference image taken when the elevator starts moving from a stopped state or when the elevator passes a specified floor is adopted as the reference reference image. r may be automatically set based on the reference image data, or may be set by an operator operating the input unit 43.
[0059] Standard reference image P rAfter the difference area calculation unit 42a determines the difference area of the reference image data (reference image P 1 ~P M ) to the reference image P of the standard reference image number "r" r The difference area calculation unit 42a extracts F (a predetermined number) of temporally consecutive reference images P from the reference image group P as a reference image group (second image group) to be subjected to the difference calculation. r ~Reference image P of image number "r+F-1" r+F-1 A group of reference images up to (see FIG. 5) is extracted.
[0060] In addition, the difference area calculation unit 42a calculates the current image data (current image R 1 ~R N A current image to be used as a reference when extracting a current image group of a predetermined number of frames F from the current image group (hereinafter, “reference current image R c The image number "c" (hereinafter referred to as the reference current image number "c") of the reference current image R c is the standard reference image P r However, the reference current image number "c" is set to a value smaller than the absolute value "A" of the frame adjustment amount "±A" of the current image group relative to the reference image group. c may be automatically set based on the current image data, or may be set by an operator operating the input unit 43.
[0061] Reference current image R c After the difference area calculation unit 42a determines the current image data (current image R 1 ~R N ), F current images R that are consecutive in time from the current image R obtained by sliding the reference current image number "c" to the minus side by "A" are extracted as a current image group (first image group). That is, the difference area calculation unit 42a extracts F current images R with image number "cA" from the current image R with image number "cA". c-A ~Current image R of image number "c-A+F-1" c-A+F-1 The current image group up to is extracted as the current image group to be subjected to the difference calculation (see the left diagram in FIG. 6).
[0062] Next, the difference area calculation unit 42a calculates the reference image P r ~ Reference Image P r+F-1 A set of reference images consisting of the current image R c-A ~Current Image R c-A+F-1 A set of difference images is generated from the current set of images (see the left image in Figure 6).
[0063] Specifically, the difference area calculation unit 42a calculates the difference area of the first reference image P r ~Fth (last) reference image P r+F-1 and the current image R at the top of the current image group. c-A ~Fth (last) current image R c-A+F-1 In other words, in the process of generating the difference image D (first difference image), the kth (k=1, 2, ..., F) reference image P r+k-1 and the kth current image R c-A+k-1 Difference image D -A,k Generate.
[0064] In addition, the difference image D s,t The index "s" of the reference current image R c The slide amount (s=-A,-A+1,...,+A) of the (reference current image number "c"), that is, the slide amount of the current image group relative to the reference image group. s,t The index “t” in is the frame number of the difference image (t=1,2,…,F). Therefore, the reference image P r ~ Reference Image P r+F-1 A set of reference images consisting of the current image R c-A ~Current Image R c-A+F-1 The difference image D -A,1 ~Difference image D -A,F A difference image group consisting of (F difference images D) is generated.
[0065] And the difference image D -A,1 ~Difference image D -A,FAfter the difference image group consisting of the above images is generated, the difference area calculation unit 42a calculates the area of the difference region of each difference image D, and calculates a total difference area DA -A The total difference area DA s The index "s" of the difference image D s,t Similarly, the index "s" of the reference current image R c The slide amount (s=-A,-A+1,...,+A) of (reference current image number "c").
[0066] As described above, the sum difference area DA -A After calculating, the reference current image R c While adding "1" to the positive side from "-A" to the slide amount (the slide amount of the current image group relative to the reference image group), s,t Generation process and sum difference area DA s The calculation process is repeated. Then, the above-mentioned difference image D s,t Generation process and sum difference area DA s The calculation process is performed using the reference current image R c This is repeated until the slide amount of becomes "+A" (see the right diagram in Figure 6). Therefore, the above-mentioned difference image D s,t Generation process and sum difference area DA s The calculation process is repeated 2A+1 times.
[0067] Therefore, the reference current image R c When the slide amount of the reference image P is "+A", as shown in FIG. r ~ Reference Image P r+F-1 A set of reference images consisting of the current image R c+A ~Current Image R c+A+F-1 This process creates a differential image D +A,1 ~Difference image D +A,F Based on the generated difference image group, the difference area calculation unit 42a calculates a total difference area DA +A Calculate.
[0068] The above-mentioned calculation process of the sum difference area DA is performed on the reference current image R c By repeatedly adding "1" to the positive side from "-A" to "+A", the sum of 2A+1 areas of difference DA (sum of difference area DA -A ~DA +A ) is calculated. The differential area calculation unit 42a then calculates the calculated total sum differential area DA (total sum differential area DA -A ~DA +A ) to the shooting point adjustment amount calculation unit 42b.
[0069] 2A+1 sum of differential area DA -A ~DA +A The photographing point adjustment amount calculation unit 42b receives the input of 2A+1 total difference areas DA -A ~DA +A Then, the image number of the first current image R in the current image group used in the calculation of the minimum total sum difference area DA is determined as "i min ". Note that the image number "i min " is the reference current image R at the time of calculating the minimum total difference area DA in the reference current image number "c". c The slide amount of the
[0070] When the total difference area DA is smallest, it corresponds to the case where the difference area in the difference image D is smallest, that is, the case where the difference between the reference image P and the current image R is smallest, which means that the shooting points of both images are aligned or nearly aligned. min " Current image R imin The shooting point of the reference image P with the standard reference image number "r" r It will be the same as or closest to the shooting point of.
[0071] Then, the photographing point adjustment amount calculation unit 42b selects the image number "i min Based on this, the shooting point adjustment amount X, which is the frame shift amount between the reference image data and the current image data, is calculated. opt (=i minIn addition, the shooting point adjustment amount calculation unit 42b calculates the calculated shooting point adjustment amount X opt to the differential processing unit 42c.
[0072] As described above, the shooting point adjustment amount X opt When the above is calculated, for example, the image number of the current image R, which is the same or closest to the shooting point of the reference image P with image number "q", is "q+X opt In this case, the reference image P of image number "q" and the image number "q+X opt By generating a difference image between the current image R and the image obtained by the difference processing unit 42c, the occurrence of events such as small foreign objects in the elevator 20 and minute changes in the position of equipment can be detected with high accuracy.
[0073] It is preferable that the shooting conditions (shutter speed, aperture value, ISO sensitivity, frame rate, etc.) of the reference image P used in the calculation of the difference area by the above-mentioned difference area calculation unit 42a are the same as those of the current image R. Therefore, in this embodiment, the difference area calculation unit 42a compares the shooting conditions of the reference image P with those of the current image R, and if the shooting conditions are different, displays information to the effect that the shooting conditions are different on the display device 3. At this time, various calculations by the above-mentioned difference area calculation unit 42a and shooting point adjustment amount calculation unit 42b may or may not be performed.
[0074] [Total difference area DA and shooting point adjustment amount X opt Calculation process] Next, the calculation process of the total sum difference area DA by the difference area calculation unit 42a and the shooting point adjustment amount X by the shooting point adjustment amount calculation unit 42b, which are performed in the maintenance system 10 of this embodiment, opt The flow of the calculation process of the total difference area DA and the shooting point adjustment amount X opt 4 is a flowchart showing a procedure of a calculation process of the difference area calculation unit 42a and the shooting point adjustment amount calculation unit 42b, which will be described below, are performed by the CPU 101 (see FIG. 4) on software.
[0075] First, the differential area calculation unit 42a of the image analysis unit 42 acquires reference image data from the reference image database 40 (S1). In this process, the differential area calculation unit 42a acquires reference image data previously captured under the same or closest imaging conditions as the current image data and the driving pattern of the car 21 during maintenance inspection. Also, in this process, the differential area calculation unit 42a stores the acquired reference image data in a memory in the arithmetic device 2. Furthermore, in this process, the differential area calculation unit 42a outputs a predetermined reference image P from the reference image data to the display device 3. As a result, the predetermined reference image P is displayed on the display device 3 (see FIG. 9 described later).
[0076] The predetermined reference image P displayed on the display device 3 is selected by an operator's selection operation on the input unit 43 of the arithmetic device 2. As a mode of inputting the reference image data to the difference area calculation unit 42a, a mode in which the operator operates the input unit 43 to select the reference image data may be adopted. In this case, the reference image data desired by the operator can be input to the difference area calculation unit 42a.
[0077] Next, the differential area calculation unit 42a acquires current image data captured during maintenance inspection from the photographing device 1 via the receiving unit 41 (S2). In this process, the differential area calculation unit 42a stores the acquired current image data in a memory in the arithmetic unit 2. In addition, in this process, the differential area calculation unit 42a outputs a specific current image R from the current image data to the display device 3. As a result, the specific current image R is displayed on the display device 3 (see FIG. 9 described later).
[0078] At this point, the specific current image R displayed on the display device 3 is the current image R having the same frame number as the frame number of the predetermined reference image P displayed on the display device 3. The present invention is not limited to this, and the specific current image R displayed on the display device 3 may be selected by a selection operation by the operator on the input unit 43 of the arithmetic device 2. Furthermore, a plurality of current image data to be analyzed are stored in the memory in the arithmetic device 2, and the latest current image data among them is the subject of analysis in the difference area calculation unit 42a.
[0079] Next, the differential area calculation unit 42a sets a reference reference image number "r" of the reference image group that is to be subjected to differential processing from the reference image data (S3). Next, the differential area calculation unit 42a sets a reference current image number "c" of the current image group that is to be subjected to differential processing from the current image data (S4).
[0080] Next, the difference area calculation unit 42a obtains a group of reference images to be subjected to difference processing from the reference image data (S5). Specifically, the difference area calculation unit 42a calculates the reference image P of the reference reference image number "r" based on the reference reference image number "r" of the group of reference images and the predetermined number of frames "F" (extraction frame range). r ~Reference image P of image number "r+F-1" r+F-1 The predetermined number of frames "F" of the reference image group is set in advance by the operator through an input operation on the input unit 43.
[0081] Next, the differential area calculation unit 42a sets the image number (frame number) "i" (=cA: hereinafter referred to as "post-slide reference current image number") of the current image R obtained by sliding the reference current image number "c" of the current image group by the slide amount "-A" (S6). The absolute value "A" of the frame adjustment amount "±A" (frame adjustment amount of the current image group relative to the reference image group) is set in advance by the operator's input operation into the input unit 43. However, since the post-slide reference current image number "i" (information on the extraction position of the current image group) needs to be a number greater than 0 (1 or greater), the absolute value "A" of the frame adjustment amount is set so as to always satisfy this condition (c>A).
[0082] Next, the differential area calculation unit 42a judges whether the post-slide reference current image number "i" is equal to or less than its upper limit "c+A" (S7). That is, in this process, the differential area calculation unit 42a judges whether the post-slide reference current image number "i" is equal to or less than the image number "c+A" of the current image R obtained by sliding the reference current image number "c" by the sliding amount "+A".
[0083] In S7, when the difference area calculation unit 42a judges that the post-slide reference current image number "i" is equal to or less than "c+A" (YES judgment in S7), the difference area calculation unit 42a acquires a current image group to be subjected to difference processing from the current image data (S8). Specifically, the difference area calculation unit 42a extracts the current image R of the post-slide reference current image number "i" based on the post-slide reference current image number "i" and the predetermined number of frames "F" (extraction frame range). i ~Current image R of image number "i+F-1" i+F-1 Then, obtain a current image set consisting of
[0084] Next, the difference area calculation unit 42a calculates the reference image P r ~ Reference Image P r+F-1 A set of reference images consisting of the current image R i ~Current Image R i+F-1 Specifically, the difference area calculation unit 42a generates a difference image group from the current image group consisting of the first reference image P r ~Fth (last) reference image P r+F-1 and the current image R at the top of the current image group. i ~Fth (last) current image R i+F-1 Therefore, in this process, a difference image D i-c,1 ~Difference image D i-c,F A set of difference images (F difference images) is generated.
[0085] When the reference image P and the current image R are both color images, the difference area calculation unit 42a converts the reference image P and the current image R into grayscale images and then creates a difference image D between them in the process of generating the difference image D. The difference area calculation unit 42a may also generate a difference image D by calculating the absolute value of the difference image and binarizing it using a predetermined threshold value.
[0086] Next, the differential area calculation unit 42a calculates (S10) the area of the remaining differential region in each of the generated F differential images D. The area of the differential region is calculated, for example, by summing up the number of images (number of pixels) of the remaining differential region in the differential image.
[0087] Next, the difference area calculation unit 42a sums up the areas of the difference regions calculated in each of the F difference images D to obtain a total difference area DA i-c In this process, the difference area calculation unit 42a calculates the calculated total difference area DA i-c Further, in this process, the difference area calculation unit 42a outputs the post-slide reference current image number "i" and the calculated total difference area DA i-c is output to the display device 3. As a result, the post-slide reference current image number "i" and the total difference area DA i-c A graph showing the relationship between the post-slide reference current image number "i" and the total difference area DA i-c In the graph showing the relationship between the horizontal axis and the total difference area DA i-c (See FIG. 9 below.)
[0088] Next, the differential area calculation unit 42a adds 1 to the post-slide reference current image number "i" (S12), and returns the process to the process of S7.
[0089] Here, we will return to the description of the judgment process of S7. In S7, if the difference area calculation unit 42a judges that the post-slide reference current image number "i" is not equal to or less than "c+A" (if the judgment in S7 is NO), the shooting point adjustment amount calculation unit 42b of the image analysis unit 42 calculates the post-slide reference current image number "i" that has the smallest total difference area DA. min Specifically, the photographing point adjustment amount calculation unit 42b first extracts the sum of the 2A+1 total difference areas DA -A ~DA +A Then, the photographing point adjustment amount calculation unit 42b determines the post-slide reference current image number "i" when the minimum summation difference area DA is calculated as "i min "
[0090] Next, the photographing point adjustment amount calculation unit 42b calculates the post-slide reference current image number "i min " and the standard reference image number "r", the shooting point adjustment amount X opt (=i min -r) is calculated (S14). Note that the shooting point adjustment amount X opt The unit of is expressed in the number of frames, but in the process of S14, the shooting point adjustment amount X opt may be converted to time based on the frame rate of the camera 31.
[0091] In addition, in the process of S14, the shooting point adjustment amount calculation unit 42b calculates the shooting point adjustment amount X opt is output to the display device 3. As a result, the shooting point adjustment amount X opt is displayed on the display device 3 (see FIG. 9 described later). Furthermore, in this process, the shooting point adjustment amount calculation unit 42b calculates the shooting point adjustment amount X opt After the process of S14, the imaging point adjustment amount calculation unit 42b outputs the total difference area DA and the imaging point adjustment amount X opt The calculation process is then completed.
[0092] In the present embodiment, when the difference area calculation unit 42a calculates the total difference area DA, the difference process between the current image group and the reference image group is repeated while shifting the extraction position of the current image group in the current image data by one frame. However, the present invention is not limited to this. For example, in order to reduce the processing load on the image analysis unit 42, the difference process between the current image group and the reference image group may be repeated while shifting the extraction position of the current image group in the current image data by, for example, a number of frames other than one (for example, two frames). In addition, the shift amount (number of frames) of the extraction position of the current image group in the difference process between the current image group and the reference image group may be appropriately changed according to, for example, the frame rate of the camera 31. However, regardless of the performance of the camera 31, from the viewpoint of improving the positioning accuracy of the shooting point, it is preferable to repeat the difference process between the current image group and the reference image group while shifting the extraction position of the current image group in the current image data by one frame as in the present embodiment.
[0093] [Various processes performed in the differential processing section, noise removal section, and contour extraction section] 8 is a flowchart showing the procedure of various processes performed by the difference processing unit 42c, noise removal unit 42d, and contour extraction unit 42e of the image analysis unit 42. Note that the processes in the difference processing unit 42c, noise removal unit 42d, and contour extraction unit 42e described below are performed by the CPU 101 (see FIG. 4) on software. Also, the processes in the difference processing unit 42c, noise removal unit 42d, and contour extraction unit 42e shown in FIG. 8 are calculated based on the total difference area DA and the shooting point adjustment amount X calculated by the difference area calculation unit 42a and the shooting point adjustment amount calculation unit 42b described in FIG. opt This is carried out following the calculation process of
[0094] First, the difference processing unit 42c calculates the shooting point adjustment amount X opt Based on this, the reference image P and the image number of the reference image P are adjusted by the shooting point adjustment amount X opt The difference image D between the current image R and the image number shifted by the number of frames opt More specifically, the subtraction processing unit 42c generates a difference between the reference image P of image number "q" and the image number "q+Xopt " current image R and the difference image D opt It should be noted that the process of S21 is performed for each reference image included in the reference image data.
[0095] In addition, in the process of S21, if the reference image P and the current image R are both color images, the difference processing unit 42c converts the reference image P and the current image R into grayscale images, and then generates a difference image D between the two images. opt In the process of S21, the difference processing unit 42c obtains the absolute value of the difference image and binarizes it using a predetermined threshold value to create a difference image D opt Then, the subtraction processing unit 42c may generate the plurality of subtraction images D calculated in the process of S21. opt The differential image data consisting of the above is output to the noise removing section 42d.
[0096] Next, the noise removal unit 42d removes each of the differential images D included in the differential image data input from the differential processing unit 42c. opt In this process, the noise removal unit 42d performs a noise removal process on each difference image D opt The noise removing unit 42d removes noise components that appear as small, discontinuous points included in the image data. The noise removing unit 42d then outputs the noise-removed differential image data to the contour extracting unit 42e.
[0097] Next, the contour extraction unit 42e extracts each difference image D opt In this process, the contour extraction unit 42e generates a contour image as a result of the contour extraction, and outputs the contour image to the display device 3. At this time, the contour extraction unit 42e also outputs the difference image data input from the noise removal unit 42d to the display device 3. As a result of this process, a difference image D opt The corresponding contour image is displayed superimposed on the image, and the difference region is highlighted. After the process of S23, the above-mentioned various processes performed by the image analysis unit 42 are completed.
[0098] [Example of maintenance and inspection analysis results] Next, a display example of the analysis result of the maintenance inspection displayed on the display device 3 of the maintenance system 10 of this embodiment will be described. Fig. 9 is a diagram showing an example of the image analysis result displayed on the display screen 3a of the display device 3 when foreign objects in the elevator hoistway 20 are detected based on a difference image between a reference image and a current image of the elevator hoistway 20. Note that in this example, an example will be described in which multiple foreign objects 50 (triangle marks in the figure) are present at the bottom of the hoistway 20.
[0099] 9, a reference image, a current image, and a difference image are displayed in a horizontally arranged manner at the top of the display screen 3a of the display device 3. In order to simplify the explanation, the reference image, the current image, the difference image, and the foreign substance 50 are shown in schematic diagrams rather than as actual images.
[0100] The reference image displayed on the display screen 3a is a reference image of a predetermined image number included in the reference image data that is the target of differential processing by the image analysis unit 42 of the calculation device 2. The image number (frame number) of the reference image to be displayed is set by an input operation to the input unit 43 by the operator.
[0101] The current image displayed on the display screen 3a changes depending on whether or not an "Adjust" button or a "Cancel Adjustment" button, which will be described later, is pressed. Before the "Adjust" button, which will be described later, is pressed, and when the "Cancel Adjustment" button, which will be described later, is pressed, a current image with the same image number as the image number (frame number) of the reference image is displayed on the display screen 3a. In other words, a current image with a shooting point different from that of the reference image is displayed on the display screen 3a. On the other hand, when the "Adjust" button, which will be described later, is pressed, the image number of the reference image is adjusted by the shooting point adjustment amount X opt A current image having an image number shifted by 1 frame is displayed on the display screen 3a. That is, a current image having a shooting point that is the same as or closest to the shooting point of the reference image is displayed on the display screen 3a.
[0102] The difference image displayed on the display screen 3a is a difference image D generated by the difference processing unit 42c of the image analysis unit 42. optA difference image D obtained by performing noise removal processing on the difference image opt That is, the reference image and the image number of the reference image are calculated by the shooting point adjustment amount X opt The difference image D between the current image and the image number shifted by the number of frames opt is displayed on the display screen 3a. In this example, the difference image D opt In order to clarify the presence or absence of a foreign object 50 (a triangular mark in the figure) in the image, the difference image is displayed in black and white, and a corresponding contour image (a white contour line) generated by the contour extraction unit 42e is displayed in the difference image D. opt is displayed superimposed on the
[0103] In this embodiment, the difference image D opt is displayed on the display screen 3a only when an "Adjust" button described later is pressed, but the present invention is not limited to this. The difference image may be displayed on the display screen 3a before the "Adjust" button described later is pressed and when the "Cancel Adjustment" button described later is pressed. In this case, a difference image between the reference image and the current image having the same image number displayed on the display screen 3a is generated separately by the image analysis unit 42 and displayed on the display screen 3a.
[0104] As described above, the reference image, the current image, and the difference image (D opt ), the difference image shows the difference area between the reference image and the current image highlighted. Therefore, the operator can quickly and easily confirm any changes from the normal state captured in the past by simply looking at the difference image. In addition, the operator can easily confirm, for example, the type of foreign object or structural changes in the hoistway 20 by referring to the reference image and the current image of the area where a difference has occurred.
[0105] In addition, in the display screen 3a, a display block of the playback time of the reference image is arranged below the reference image, and a display block of the playback time of the current image is arranged below the current image. The value displayed in the display block of the playback time of the reference image (0.333 seconds in the example of FIG. 9) is the playback time in the reference image data of the reference image being displayed, and corresponds to the elapsed time from the start of shooting (first frame) of the reference image data. On the other hand, the value displayed in the display block of the playback time of the current image (0.433 seconds in the example of FIG. 9) is the playback time in the current image data of the current image being displayed, and corresponds to the elapsed time from the start of shooting (first frame) of the current image data. In this embodiment, the current image displayed changes before and after pressing the "Perform Adjustment" button described later, and therefore the playback time of the current image displayed before and after pressing the "Perform Adjustment" button described later also changes.
[0106] The display block for each playback time is a user interface that can be accessed (by inputting a numerical value) by an operator's input operation on the input unit 43. Therefore, in this embodiment, the numerical values displayed in the display blocks for the playback times of the reference image and the current image can be changed on the display screen 3a by the operator's input operation. When the numerical values displayed in the display blocks for the playback times of the reference image and / or the current image are changed by the operator's input operation, the reference image and / or the current image corresponding to the changed playback time are displayed. The change and display process of the above-mentioned analysis result by the operator's input operation is controlled by the image analysis unit 42 based on an instruction signal corresponding to the operator's input operation.
[0107] In addition, on the display screen 3a, below the display blocks of the playback times of the reference image and the current image, a seek bar (playback bar) indicating the playback position of the current image being displayed in the current image data (moving image) is displayed.
[0108] The seek bar is a user interface that can be accessed (changed) by an operator's input operation on the input unit 43 on the display screen 3a. Therefore, in this embodiment, the playback position of the current image can be changed on the seek bar by the operator's input operation. When the playback position of the current image is changed by the operator's input operation on the seek bar, the current image corresponding to the changed playback position is displayed on the display screen 3a. Note that the change and display process of the above-mentioned analysis result by the operator's input operation is controlled by the image analysis unit 42 based on an instruction signal corresponding to the operator's input operation.
[0109] In addition, in the display screen 3a, a graph showing the relationship between the total difference area DA calculated by the difference area calculation unit 42a and the post-slide reference current image number "i" of the current image (frame number "i" in the figure) is displayed in the left area below the seek bar. In addition, to the right of this graph, the reference reference image number "r", the reference current image number "c", the predetermined number of frames "F" (extraction frame range), and the frame adjustment amount "A", which are the calculation conditions of the total difference area DA, are also displayed.
[0110] FIG. 9 shows a graph illustrating the relationship between the total difference area DA and the post-slide reference current image number "i" of the current image when the reference reference image number "r" and reference current image number "c" are set to 10, the predetermined number of frames "F" is set to 10, and the frame adjustment amount "A" is set to 5. As described above, the post-slide reference current image number "i" of the current image is a number between "cA" and "c+A" based on the reference current image number "c" and the absolute value of the frame adjustment amount "A", so in the example shown in FIG. 9, the post-slide reference current image number "i" varies within the range of 5 to 15. In addition, in the example shown in FIG. 9, the post-slide reference current image number "i" of the current image that provides the smallest total difference area DA is the post-slide reference current image number "i" of the current image. min " (frame number) is "12", so the shooting point adjustment amount X optbecomes "+2" (=12-10). In this way, by displaying the relationship between the total sum difference area DA and the post-slide reference current image number "i" and the calculation conditions of the total sum difference area DA on the display screen 3a, the operator can easily grasp the amount of deviation between the shooting point of the reference image data and that of the current image data.
[0111] Moreover, each display block of the calculation conditions of the total difference area DA is a user interface that can be accessed (changed) by the operator's input operation to the input unit 43 on the display screen 3a. Therefore, when the operator changes the numerical value of the calculation conditions of the total difference area DA by the input operation and presses the "Execute adjustment" button described later, the total difference area DA and the shooting point adjustment amount X are calculated again in the image analysis unit 42. opt and the difference image D opt Then, the result of the analysis performed again by the image analysis unit 42 is displayed on the display screen 3a.
[0112] In addition, in the display screen 3a, the acquisition date of the reference image data ("reference data date" in the figure) and the acquisition date of the current image data ("current data date" in the figure) are displayed in the right area below the seek bar.
[0113] The display (selection) blocks for "reference data date" and "current data date" are user interfaces that can be accessed (selected) on the display screen 3a by the operator's input operation on the input unit 43. Therefore, the acquisition date of the reference image data and the acquisition date of the current image data can be changed on the display screen 3a by the operator's input operation. When the acquisition date of the reference image data and / or the current image data is changed by the operator's input operation on the display blocks for "reference data date" and / or "current data date", the reference image and / or the current image corresponding to the changed acquisition date are displayed. As a result, the difference image D displayed on the display screen 3a opt The above-mentioned process of changing and displaying the analysis result in response to an input operation by the operator is controlled by the image analysis unit 42 based on an instruction signal corresponding to the input operation by the operator.
[0114] In addition, on the display screen 3a, an "Adjust" button and a "Cancel Adjustment" button are displayed below the display blocks of "Reference Data Date" and "Current Data Date", respectively. Note that the "Adjust" button and the "Cancel Adjustment" button are user interfaces that can be accessed (pressed) by the operator's input operation on the input unit 43 on the display screen 3a.
[0115] When the operator presses the "Execute adjustment" button on the display screen 3a, the image number of the reference image is set to the shooting point adjustment amount X opt The current image having the image number shifted by the number of frames is displayed on the display screen 3a, and the corresponding difference image D opt Also, at this time, the shooting point adjustment amount X is displayed on the indicator below the "Execute adjustment" button and the "Cancel adjustment" button. opt (In the figure, "Optimal time adjustment amount: +0.1 seconds") and the extraction time of the image data are displayed (In the figure, "Extraction time range: 2.0 seconds") are displayed.
[0116] Furthermore, when the operator presses the "Cancel Adjustment" button on the display screen 3a, the numerical values of the optimal time adjustment amount and the extraction time range displayed within the indicator disappear. At this time, the current image with the same image number as the reference image is displayed on the display screen 3a, and the difference image also disappears. The above-mentioned process of changing and displaying the analysis results by the operator pressing each button is controlled by the image analysis unit 42 based on an instruction signal corresponding to the operator's pressing operation.
[0117] [Various effects] As described above, in the maintenance system 10 of this embodiment, the photographing point adjustment amount X that minimizes the total difference area DA between the reference image and the current image is opt As a result, the shooting point (image number) of the current image can be adjusted (corrected) based on the shooting point adjustment amount X optBased on this, the shooting points of the reference image and the current image, which are to be compared (subtraction processing), can be aligned with high accuracy. Therefore, in this embodiment, the error in the difference image generated from the reference image and the current image, whose shooting points are aligned, is also small, and small foreign objects in the hoistway 20 and minute positional changes of the equipment can be detected with high accuracy from the difference image.
[0118] In addition, the method of aligning the photographing points of the current image and the reference image in the maintenance system 10 of this embodiment is to adjust the photographing point by a photographing point adjustment amount X opt Since the inspection is performed based on the deviation amount of the image number, it is not affected by deterioration over time, and the cost of maintenance is not required. Therefore, in this embodiment, maintenance and inspection can be performed at low cost.
[0119] Incidentally, in the above-mentioned detection of a foreign object in the elevator shaft 20, it is necessary to match the shooting point of the reference image with that of the current image with high accuracy, but according to verification by the present inventors, it has been found that the shooting point of the camera 31 needs to be controlled with high accuracy of less than a few centimeters (for example, less than 3 cm). Here, an example of the verification will be described. However, in the drawings of the verification example shown below, the analysis results actually obtained in the verification experiment are not shown as they are, but are shown as schematic diagrams to further emphasize the knowledge obtained.
[0120] 10A, 10B, and 10C are schematic diagrams of a reference image, a current image, and a difference image obtained when foreign object detection is performed (example) in the same manner as the foreign object detection in the hoistway 20 described in Fig. 9. Since these images are acquired by the maintenance system 10 of this embodiment, the positioning of the shooting points of the displayed reference image and current image is performed with high precision, and only foreign object 50 (triangle mark in the figure) is displayed in the difference image D.
[0121] 11A, 11B, and 11C are the reference image, current image, and difference image shown in Fig. 10A, 10B, and 10C, respectively, in which the current image is shifted by one frame (comparative example). In this case, a different difference region 51 (an arc-shaped region in the figure) is detected in addition to the foreign object 50 in the difference image. That is, in this case, not only the foreign object 50 but also the difference region 51 is erroneously detected as a foreign object.
[0122] If the frame rate of the camera 31 is 30 fps for a general elevator speed (45 to 105 m / min), the elevator will move 1.7 cm to 5.8 cm during one frame. For this reason, in foreign object detection, even if the shooting point of the current image and the shooting point of the reference image differ by only a few centimeters, a change in a non-abnormal part will be detected as a foreign object (difference). The cause of the above-mentioned false detection is due to a slight deviation in the coordinates of each device captured within the field of view, and if the false detection area increases in this way, there is a possibility that the operator will overlook a foreign object that should be found. It is difficult to completely eliminate the causes of the above-mentioned false detection, but in order to reduce the influence as much as possible, it is possible to apply an algorithm that removes the causes by morphological processing, etc.
[0123] Therefore, from the above verification experiment, it was found that in order to capture a small foreign object or change of about several pixels with good sensitivity by differential detection, it is necessary to align the shooting point of the reference image and that of the current image as much as possible, and it is also necessary to suppress the error of one frame. On the other hand, in the maintenance system 10 of the above-mentioned embodiment, it is possible to align the shooting points of the reference image and the current image with high accuracy, and it is clear from the verification results of the difference images shown in Figures 10C and 11C that the above problem can be solved.
[0124] Another possible solution to the above problem is to increase the sensitivity of the difference detection (e.g., the frame rate of the camera 31, the image resolution, the number of pixels, etc.) in order to accurately detect small foreign objects in the elevator 20 or minute positional changes of equipment to the order of a few pixels. However, if the sensitivity of the difference detection is too high, there is a disadvantage that even non-abnormal locations may be erroneously detected as differences due to slight differences in the shooting points. In contrast, the maintenance system 10 of the above-mentioned embodiment of the invention can, in principle, align the shooting points of the reference image and the current image with high accuracy without increasing the sensitivity of the difference detection, thereby eliminating such disadvantages.
[0125] Furthermore, in the maintenance system 10 of this embodiment, as described above, a function is provided for manually changing the current image to be compared with the reference image by an operator performing a predetermined operation on the input unit 43 on the display screen 3a of the display device 3. Therefore, even if a situation occurs in which the positioning accuracy of the shooting point is slightly reduced due to the influence of noise or the like, the shooting point of the current image can be finely adjusted in units of one frame, and highly accurate maintenance and inspection can be easily performed.
[0126] [Various variations] Although the maintenance system 10 and the maintenance device according to an embodiment of the present invention have been described above, the present invention is not limited thereto. For example, the following various modified examples can be adopted.
[0127] The display mode of the analysis results of the detection of a foreign object in the elevator shaft 20 is not limited to the mode shown in Fig. 9. Some of the various display items (user interfaces) shown on the display screen 3a in Fig. 9 may be appropriately deleted, or other user interfaces may be appropriately added, as necessary. For example, in the display mode shown in Fig. 9, in order to display only the minimum necessary functions, display items such as the playback time of each image, a graph showing the relationship between the sum difference area DA and the post-slide reference current image number "i", the optimal time adjustment amount, and the extraction time range may be deleted.
[0128] Also, for example, the "Adjust" button and the "Cancel Adjustment" button may not be provided on the display screen 3a. In this case, the reference image and the image number of the reference image are displayed on the display screen 3a in accordance with the shooting point adjustment amount X opt The current image having the image number shifted by 1 frame and the difference image between them may be automatically displayed.
[0129] Furthermore, as another display mode of the analysis result of the detection of a foreign object in the elevator 20, a mode in which the difference image is not displayed may be adopted. Fig. 12 is a diagram showing one example. Note that in Fig. 12, in order to simplify the explanation, only the display mode above the seek bar is shown. In the example shown in Fig. 12, a reference image and the image number of the reference image are displayed in accordance with the shooting point adjustment amount X opt The current image having the image number shifted by one frame and the current image having the image number are displayed on the display screen 3a, but the difference image between them is not displayed.
[0130] Instead, in the example shown in Fig. 12, a contour image (a thick black contour line in the figure) indicating the difference region generated by the contour extraction unit 42e is superimposed on the current image. In this case, too, the difference region (the region of the multiple foreign objects 50) is highlighted by the contour image on the current image, so that the operator can quickly and easily confirm changes from the normal state captured in the past by simply looking at the current image. In addition, the operator can easily confirm, for example, the type of foreign object or structural changes in the hoistway 20 by referring to the reference image and the current image of the region where a difference has occurred.
[0131] In addition, the above-mentioned embodiment examples are detailed and specific descriptions of the configuration of the device in order to easily explain the present invention, and are not necessarily limited to those having all of the described configurations. In addition, the position, size, shape, range, etc. of each component shown in the drawings, etc. may not represent the actual position, size, shape, range, etc. in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings, etc. Furthermore, the present invention may take on various other application examples and modified examples as long as they do not deviate from the gist of the present invention described in the claims. [Explanation of symbols]
[0132] 1...imaging device, 2...computing device, 3...display device, 10...maintenance system, 20...hoistway, 21...car, 31...camera, 33...imaging processing unit, 33a...imaging condition setting unit, 33b...notification unit, 34, 43...input unit, 40...reference image database, 41...receiving unit, 42...image analysis unit, 42a...difference area calculation unit, 42b...imaging point adjustment amount calculation unit, 42c...difference processing unit, 42d...noise removal unit, 42e...contour extraction unit
Claims
1. 1. A maintenance device for inspecting and maintaining an elevator shaft based on first image data of the elevator shaft photographed by a photographing device during maintenance and inspection of the elevator, and second image data of the elevator shaft to be compared with the first image data, An image analysis unit that performs image analysis of the elevator based on the first image data and the second image data, The image analysis unit a difference area calculation unit that generates a first difference image between an image included in the first image data and an image included in the second image data, and calculates an area of a difference region included in the first difference image; a deviation amount calculation unit that calculates a deviation amount between a shooting point of each image included in the first image data and a shooting point of each image included in the second image data based on the area of the difference region calculated by the difference area calculation unit. A maintenance device comprising:
2. The image analysis unit further includes a difference processing unit that extracts an image included in the first image data and an image included in the second image data, the image having the same or closest shooting points to each other, based on the shift amount calculated by the shift amount calculation unit, and generates a second difference image between the extracted images.
2. The maintenance device according to claim 1.
3. The difference area calculation unit extracting a first image group including a predetermined number of images that are temporally consecutive from the first image data; extracting a second image group including the predetermined number of images that are temporally consecutive from the second image data; generating the first difference images between each image included in the first image group and each image included in the second image group to generate the predetermined number of the first difference images; calculating an area of a difference region in each of the predetermined number of first difference images, and calculating a total difference area by summing up the areas of the calculated difference regions in the predetermined number of first difference images; a series of processes, which include the extraction process of the first image group, the generation process of the first difference image, and the calculation process of the total difference area, are repeated a predetermined number of times while shifting the extraction position of the first image group in the first image data at a predetermined interval, and a total difference area is calculated for the predetermined number of times; The deviation amount calculation unit The amount of deviation is calculated based on the extraction position of the first image group at which the minimum total sum difference area is obtained among the total sum difference areas calculated by the difference area calculation unit for the predetermined number of times.
2. The maintenance device according to claim 1.
4. The difference area calculation unit repeats the series of processes the predetermined number of times while shifting the extraction position of the first image group in the first image data at one frame intervals.
4. The maintenance device according to claim 3.
5. a display unit that displays information indicating a relationship between the total sum difference area for the predetermined number of times and information regarding the extraction position of the first image group.
4. The maintenance device according to claim 3.
6. The image analysis unit further includes a contour extraction unit that extracts a contour of a difference region included in the second difference image generated by the image analysis unit.
3. The maintenance device according to claim 2.
7. a display unit that displays information about the second difference image, The display unit displays the second difference image and the image showing the contour in a superimposed manner.
7. The maintenance device according to claim 6.
8. The apparatus further includes an input unit that receives an operation by an operator and enables the calculation conditions by the difference area calculation unit and the calculation conditions by the deviation amount calculation unit to be set by the operation by the operator.
2. The maintenance device according to claim 1.
9. A maintenance system comprising: a photographing device that photographs first image data of a hoistway during maintenance and inspection of an elevator; and performing inspection and maintenance of the hoistway based on the first image data and second image data of the hoistway that is compared with the first image data, an image analysis device that performs image analysis of the elevator shaft based on the first image data and the second image data; A display device that displays a result of the image analysis by the image analysis device, The image analysis device includes: a difference area calculation unit that generates a first difference image between an image included in the first image data and an image included in the second image data, and calculates an area of a difference region included in the first difference image; a deviation amount calculation unit that calculates a deviation amount between a shooting point of each image included in the first image data and a shooting point of each image included in the second image data based on the area of the difference region calculated by the difference area calculation unit. A maintenance system comprising:
10. A maintenance program to be implemented in and executed by a maintenance device that performs inspection and maintenance of an elevator shaft based on first image data of the elevator shaft captured by a photographing device during maintenance and inspection of the elevator and second image data of the elevator shaft to be compared with the first image data, an image analysis unit included in the maintenance device generates a first difference image between an image included in the first image data and an image included in the second image data, and calculates an area of a difference region included in the first difference image; The image analysis unit calculates a shift amount between a shooting point of each image included in the first image data and a shooting point of each image included in the second image data based on the area of the differential region calculated in the process of calculating the area of the differential region, and the image analysis unit is implemented in the maintenance device and executed. A maintenance program comprising:
Citation Information
Patent Citations
Image analytics for elevator maintenance
EP3409629A1
