Panel diagram information management system, panel diagram information management device, panel diagram information management method and program

Abstract

This invention provides a panel diagram information management system, a panel diagram information management device, a panel diagram information management method, and a program that can generate appropriate panel diagram information for managing equipment installed in control panels, distribution boards, lighting boards, power distribution boards, etc. [Solution] The control panel information management server 1 includes: an equipment mounting position estimation unit 112 that estimates the position of the equipment mounting unit image within the captured image for each captured image; an equipment mounting state estimation unit 113 that estimates whether equipment is mounted on the equipment mounting unit, the state of the equipment, and the type of equipment of the mounted equipment for each of the multiple captured images; an equipment information reading unit 114 that reads equipment information contained in at least one of the equipment mounting unit image and the equipment image for each of the multiple captured images; a captured image position relationship estimation unit 115 that estimates the positional relationship between the multiple captured images; and a control panel information generation unit 116 that generates control panel information.

Description

Technical Field

[0001] The present invention relates to a panel diagram information management system, a panel diagram information management device, a panel diagram information management method, and a program.

Background Art

[0002] An information processing apparatus has been proposed that includes an image recognition unit that recognizes an image of a two-dimensional ID code for identifying control devices included in a captured image obtained by a camera capturing a control panel, a device identification unit that specifies the model information and installation position of each control device based on the recognized device identification image, and a layout diagram generation unit that generates a layout diagram of each control device on the control panel based on the specified model information and installation position of each control device (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, in the control panel, there may be a case where the device is left unattached to the device mounting portion for mounting the device. In this case, if the presence or absence of the device mounted on the device mounting portion and the state of the device when the device is mounted, for example, the on / off state, cannot be grasped, there is a possibility that an appropriate panel diagram for managing the devices connected to the control panel cannot be obtained.

[0005] [[ID=�9]]The present invention has been made in view of the above circumstances, and an object thereof is to provide a panel diagram information management system, a panel diagram information management device, a panel diagram information management method, and a program that can generate appropriate panel diagram information for managing devices mounted on a control panel, a distribution board, a lighting board, a power panel, and the like.

Means for Solving the Problems

[0006] To achieve the above objective, the control panel information management system according to the present invention is: A photographic image storage unit stores multiple images obtained by photographing the disc in multiple pre-set states, in association with disc identification information that identifies the disc. For each of the multiple captured images, a device mounting position estimation unit estimates the in-image position of at least one device mounting image included in the captured image, and For each of the multiple captured images, the device mounting state estimation unit estimates whether the device is mounted on at least one of the device mounting parts included in the captured image, the state of the device, and the type of device mounted on the device mounting part. A device information reading unit reads device information contained in at least one of the device mounting part image and the device image, based on the position within the image of at least one of the device images and the type of at least one of the device, for each of the multiple captured images. A unit for estimating the positional relationship between multiple captured images, The system includes a panel diagram information generation unit that generates panel diagram information based on the position within the image of at least one of the equipment mounting parts, whether or not equipment is mounted on at least one of the equipment mounting parts and the type of equipment mounted on the equipment mounting part, equipment information and the positional relationship between a plurality of captured images. [Effects of the Invention]

[0007] According to the present invention, the equipment mounting status estimation unit estimates, for each of the multiple captured images, whether or not equipment is mounted on at least one equipment mounting section included in the captured image, and the type of equipment mounted on the equipment mounting section. As a result, it is possible to generate panel diagram information that includes information on whether or not equipment is mounted on the equipment mounting section, thereby generating appropriate panel diagram information for managing equipment mounted on control panels, distribution boards, lighting boards, power panels, etc. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of the configuration of a control panel information management system according to an embodiment of the present invention. [Figure 2] This is a block diagram showing the hardware configuration of a control panel information management system according to an embodiment. [Figure 3] This is a block diagram showing the functional configuration of a control panel information management system according to an embodiment. [Figure 4] This figure shows an example of the information stored by the captured image storage unit according to the embodiment. [Figure 5] (A) is a diagram showing an example of information stored by the local arrangement storage unit according to the embodiment, and (B) is a diagram showing an example of information stored by the global arrangement storage unit according to the embodiment. [Figure 6] This figure shows an example of the information stored by the control panel information storage unit according to the embodiment. [Figure 7] This is a sequence diagram illustrating the operation of a control panel information management system according to an embodiment. [Figure 8] (A) is a diagram showing an example of a photograph of the control panel with the door closed, and (B) is a diagram showing an example of a photograph of the control panel with the door open. [Figure 9] (A) is a diagram showing an example of a photograph of a portion of the control panel with the door open, and (B) is a diagram showing an example of a photograph of another portion of the control panel with the door open. [Figure 10] (A) is a diagram showing an example of a photograph of a part of the disc, and (B) is a diagram showing an example of a photograph of another part of the disc. [Figure 11] This figure shows an example of a photograph of a portion of the disc. [Figure 12] This is a sequence diagram illustrating the operation of a control panel information management system according to an embodiment. [Figure 13] This flowchart shows an example of the flow of the image information acquisition process performed by the terminal device according to the embodiment. [Figure 14]This flowchart shows an example of the flow of the panel diagram information generation process executed by the panel diagram information generation server according to the embodiment. [Figure 15] (A) is a diagram showing an example of a photograph of a part of the board, (B) is a diagram showing a photograph corresponding to area A71 in (A), (C) is a diagram showing a photograph corresponding to area A72 in (A), and (D) is a diagram showing a photograph corresponding to area A73 in (A). [Figure 16] This is a block diagram showing the functional configuration of a control panel information management system related to a modified example. [Figure 17] This figure shows an example of a composite image displayed on the display unit of a modified terminal device. [Modes for carrying out the invention]

[0009] Hereinafter, a panel diagram information management system according to an embodiment of the present invention will be described with reference to the drawings. The panel diagram information management system according to the present embodiment includes a photographed image storage unit, a device mounting portion estimation unit, a device mounting state estimation unit, a device information reading unit, a photographed image positional relationship estimation unit, and a panel diagram information generation unit. The photographed image storage unit stores a plurality of photographed images obtained by photographing a panel in a plurality of preset states in association with panel identification information for identifying the panel. The device mounting portion estimation unit estimates the in-image position of a device mounting portion image, which is an image of at least one device mounting portion included in a photographed image, within the photographed image for each of the plurality of photographed images. The device mounting state estimation unit estimates, for each of the plurality of photographed images, whether a device is mounted on at least one device mounting portion, the state of the device, and the device type of the device mounted on the device mounting portion. The device information reading unit reads device information included in at least one of the at least one device mounting portion image and the device image based on the in-image position of at least one device image and the type of at least one device for each of the plurality of photographed images. The photographed image positional relationship estimation unit estimates the positional relationship between the plurality of photographed images. The panel diagram information generation unit generates panel diagram information based on the in-image position of the device mounting portion image of at least one device mounting portion, whether a device is mounted on at least one device mounting portion and the device type of the device mounted on the device mounting portion, the device information, and the positional relationship between the plurality of photographed images.

[0010] As shown in FIG. 1, the panel diagram information management system according to the present embodiment includes a terminal device 3 held by an operator who photographs a panel at a location where a panel such as a control panel, a distribution board, a lighting board, or a power panel, which is an object for creating a panel diagram, is installed, a terminal device 5 held by an administrator who manages the panel diagram, and a panel diagram information management server 1. The terminal devices 3 and 5 and the panel diagram information management server 1 can communicate with each other via a wide area network NW1. The wide area network NW1 is, for example, the Internet.

[0011] The terminal device 3 is, for example, a smartphone, and as shown in FIG. 2, includes a CPU 301 (Central Processing Unit) 301, a main memory unit 302, an auxiliary storage unit 303, a display unit 304, an input unit 305, a communication unit 306, an imaging unit 307, and a bus 309 that connects these to each other. The main memory unit 302 has a volatile memory such as a RAM (Random Access Memory) and is used as the working area of the CPU 301. The main memory unit 302 also has an image dedicated memory (not shown) that temporarily stores the image information to be displayed on the display unit 304. The auxiliary storage unit 303 is a non-volatile memory such as a semiconductor flash memory and stores programs for the CPU 301 to execute various processes. The display unit 304 is a display device such as a liquid crystal display or an organic EL (Electro-Luminescence) display. The input unit 305 is, for example, a transparent touch pad arranged over the display unit 304. The communication unit 306 is connected to the local network NW3 and transmits the information transferred from the CPU 301 to the drawing information management server 1 via the local network NW3 and the wide area network NW1, or transfers various information obtained from the drawing information management server 1 via the wide area network NW1 and the local network NW3 to the CPU 301.

[0012] The imaging unit 307 has a camera and a DMA (Direct Memory Access) controller. After being activated, the DMA controller transfers the captured image information obtained by imaging with the camera to the image dedicated memory in real time. Also, when the imaging unit 307 is notified of the shooting command information from the CPU 301, it notifies the CPU 301 of the captured image information obtained by shooting at the timing when the shooting command information is notified.

[0013] The CPU 301 functions as a display control unit 311, an imaging control unit 312, an image management unit 313, and an image information notification unit 314, as shown in Figure 3, by reading the program stored in the auxiliary storage unit 303 into the main storage unit 302 and executing it. The auxiliary storage unit 303 shown in Figure 2 also has an image capture storage unit 331, as shown in Figure 3. The image capture storage unit 331 stores image capture information, which shows an image obtained by imaging unit 307 to capture at least a part of the panel; location information, which shows the location of the building where the panel to be photographed is installed; installation location information, which shows the installation location of the panel within the building; and image type information, which shows the type of the object to be photographed, in association with image identification information that identifies the image capture information. Here, location information is information that shows, for example, the building name and address of the building to be photographed, such as the building name "○○ Building" or the address "○○ City ○× Town △-□". Installation location information is information that shows the floor number of the room in the building where the panel is installed, the room number of that room, etc. Image type information identifies the state of the control panel at the time of shooting (e.g., the open / closed state of the door) and the part of the control panel that is to be photographed (e.g., "the entire control panel," "a pre-set part of the control panel," etc.). Image identification information identifies the captured image information and includes, for example, the date and time of shooting.

[0014] When a user performs a panel photography preparation operation to the input unit 305 to photograph a panel, the display control unit 311 receives the panel photography preparation operation and, based on the image type information described later that is notified by the image management unit 313, forms a photography method notification image to notify the user of how to photograph the panel and displays it on the display unit 304. Furthermore, when all the photographed images necessary for generating the panel diagram information of the panel to be photographed have been acquired and the image management unit 313 notifies the user of the photography completion notification information described later, the display control unit 311 forms an image transmission prompt image to prompt the user to send at least one photographed image obtained from photographing the panel to the panel diagram information management server 1 and displays it on the display unit 304.

[0015] When a user performs an imaging operation to the input unit 305 to photograph the panel to be photographed, the imaging control unit 312 receives the imaging operation and generates imaging command information that instructs the imaging unit 307 to photograph the panel, and then transmits it to the imaging unit 307.

[0016] When the user inputs the aforementioned location information and installation location information for the panel to be photographed to the input unit 305 during the panel photography preparation operation, the image management unit 313 stores the input location information and installation location information in the captured image storage unit 331. Furthermore, after receiving the panel photography preparation operation, the image management unit 313 sequentially notifies the display control unit 311 of at least one pre-set image type information. This image type information is determined based on the method of photographing the panel. For example, different image type information is assigned to a photograph taken of the entire panel with the door closed, a photograph taken of the entire panel with the door open, and a photograph taken of a part of the panel with the door open. Moreover, after receiving the aforementioned imaging operation from the user, when the captured image information obtained by photographing the panel is transferred from the imaging unit 307, the image management unit 313 stores the transferred captured image information in the captured image storage unit 331, along with the image identification information, the corresponding location information and installation location information, and the image type information most recently notified to the display control unit 311. Subsequently, the image management unit 313 determines whether all the image information necessary for generating the board diagram information of the board to be photographed has been acquired. Specifically, if the image management unit 313 stores image information corresponding to at least one pre-set image type information in the image storage unit 331, it determines that all the image information necessary for generating the board diagram information of the board to be photographed has been acquired and notifies the display control unit 311 of the completion of photography. On the other hand, if the image management unit 313 does not store image information corresponding to any of the at least one pre-set image type information in the image storage unit 331, it selects one of the image type information corresponding to the image information not stored in the image storage unit 331 and notifies the display control unit 311 of it.

[0017] When a user performs a captured image transmission operation to the input unit 305 to send captured image information to the panel diagram information management server 1, the image information notification unit 314 receives the captured image transmission operation and generates captured image notification information, which includes the captured image information, location information, installation location information, and image type information stored in the captured image storage unit 331. The image information notification unit 314 then transmits the generated captured image notification information to the panel diagram information management server 1.

[0018] Terminal device 5 is, for example, a personal computer and includes a display unit (not shown) and an input unit (not shown). Terminal device 5 also includes a panel diagram information storage unit (not shown) that stores panel diagram information acquired from the panel diagram information management server 1. Here, the display unit is a display device such as a liquid crystal display or an organic EL display. The input unit is an input device such as a keyboard. When an administrator performs a panel diagram information request operation via the input unit to request the transmission of panel diagram information from the panel diagram information management server 1, terminal device 5 accepts the panel diagram information request operation and generates panel diagram information request information. Then, terminal device 5 transmits the generated panel diagram information request information to the panel diagram information management server 1, thereby acquiring panel diagram information notification information transmitted from the panel diagram information management server 1, extracting various information contained in the acquired panel diagram information notification information, and storing it in the panel diagram information storage unit. Then, terminal device 5 forms a panel diagram image based on the various information stored in the panel diagram information storage unit and displays it on the display unit.

[0019] Returning to Figure 2, the control panel information management server 1, as shown in Figure 2, comprises a CPU 101, a main memory unit 102, an auxiliary memory unit 103, a communication unit 106, and a bus 109 connecting them to each other. The CPU 101 is, for example, a multi-core processor. The main memory unit 102 has volatile memory such as RAM and is used as a workspace for the CPU 101. The auxiliary memory unit 103 consists of large-capacity non-volatile memory and stores programs for realizing various functions of the control panel information management server 1. The communication unit 106 is connected to the wide-area network NW1.

[0020] The CPU 101 reads the program stored in the auxiliary storage unit 103 into the main storage unit 102 and executes it, so as shown in Figure 3, it functions as an image information acquisition unit 111, an image region extraction unit 118, an equipment mounting unit position estimation unit 112, an equipment mounting state estimation unit 113, an equipment information reading unit 114, a captured image position relationship estimation unit 115, a control panel information generation unit 116, and a control panel information notification unit 117. Furthermore, the auxiliary storage unit 103 shown in Figure 2 has a captured image storage unit 131, a local placement storage unit 132, a captured image placement storage unit 133, and a control panel information storage unit 134, as shown in Figure 3. The captured image storage unit 131 stores captured image information and corresponding location information, installation position information, and image type information, associating them with image identification information, as shown in Figure 4, for example.

[0021] The local placement storage unit 132 stores, in association with image identification information, the following: equipment mounting relative position information indicating the local coordinates of the equipment mounting position where equipment is mounted in each captured image shown by the captured image information; equipment status information indicating whether equipment is mounted on the equipment mounting position and the state of the mounted equipment; equipment type information indicating the type of equipment mounted; equipment image feature quantity information indicating the image feature quantities of the equipment portion; and equipment information indicating the name of the equipment, rated voltage, etc., as written on a sticker attached to the equipment. Here, examples of equipment include switches such as main circuit breakers and branch circuit breakers mounted on the equipment mounting portion of the panel, instruments, and programmable logic controllers. Examples of equipment information include the model number of the equipment and the name of the load written on the nameplate of the equipment mounting portion. The captured image placement storage unit 133 stores, in association with image identification information, the following: equipment mounting relative position information indicating the coordinates of the representative position of the target portion corresponding to the captured image information of the entire panel, as shown in, for example, Figure 5(B).

[0022] As shown in Figure 6, for example, the panel diagram information storage unit 134 stores absolute position information of the equipment mounting area, which indicates the global coordinates of the equipment mounting area within the entire panel; equipment status information, which indicates whether or not equipment is mounted on the equipment mounting area and the state of the mounted equipment; equipment type information, which indicates the type of equipment mounted; equipment image feature quantity information, which indicates the feature quantities of the image of the equipment portion; and equipment information, all associated with image identification information.

[0023] Returning to Figure 3, the image information acquisition unit 111 acquires the captured image notification information transmitted from the terminal device 3. It then extracts the captured image information, location information, installation location information, image type information, and image identification information contained in the acquired captured image notification information, and stores these extracted pieces of information in the captured image storage unit 131, associating them with each other. The image region extraction unit 118 uses an image region extraction technique that employs the graph cut method to extract multiple characteristic image regions from each captured image indicated by the captured image information stored in the captured image storage unit 131. The image region extraction unit 118 then notifies the device mounting position estimation unit 112, the device mounting state estimation unit 113, and the device information reading unit 114 of the image region information indicating each of the extracted multiple image regions corresponding to each of the multiple captured images, and the image region position information indicating the relative position coordinates of the image region within the entire captured image. Furthermore, the image region extraction unit 118 notifies the captured image position relationship estimation unit 115 of the image region information and image region position information corresponding to each of the multiple captured image information, as well as the image type information stored in the captured image storage unit 131.

[0024] The device attachment position estimation unit 112 estimates the in-image position of the device attachment image, which is the image of at least one device attachment included in the captured image, for each of the captured images indicated by the multiple captured image information stored in the captured image storage unit 131. Specifically, the device attachment position estimation unit 112 uses a device attachment estimation model for estimating the image region corresponding to the device attachment to estimate the image region corresponding to the device attachment from among the image regions indicated by each of the multiple image region information notified by the image region extraction unit 118. Here, the device attachment estimation model is represented, for example, by a convolutional neural network (CNN), and is a trained model for determining whether or not an image region is a device attachment image. Then, based on the image region position information notified by the image region extraction unit 118, the device attachment position estimation unit 112 identifies the in-image position, which is the relative position coordinate of the device attachment image within the entire captured image, and stores the identified in-image position information in the local placement storage unit 132 in association with the image identification information.

[0025] The device mounting state estimation unit 113 estimates, for each of the multiple captured images, whether or not a device is mounted on at least one device mounting area included in the captured image, the state of the device, and the type of device mounted on the device mounting area. Specifically, the device mounting state estimation unit 113 refers to the device mounting area location information stored in the local placement memory unit 132 and selects at least one image area adjacent to the device mounting area image from among the image areas indicated by each of the multiple image area information notified by the image area extraction unit 118. Next, the device mounting state estimation unit 113 uses a device estimation model for estimating the image area corresponding to the device mounted on the device mounting area, as well as the state and type of the device, to estimate the image area corresponding to the image of the device from among the at least one selected image area, and also estimates the type and state of the device. Here, the device estimation model is, for example, represented by a CNN and is a trained model for determining whether or not an image area is an image of a device, and if it is a device, the type and state of the device. This equipment estimation model is trained using training information that includes, for example, a combination of an image of a switch when the switch lever is in the ON position and a training label indicating that the switch is ON, and a combination of an image of a switch when the switch lever is in the OFF position and a training label indicating that the switch is OFF. As a result, the equipment mounting state estimation unit 113 can estimate the ON / OFF state of a switch even if, for example, the switch does not have characters indicating its ON / OFF state displayed on it. Furthermore, this equipment estimation model is trained using training information that includes, for example, a combination of an image of a switch without test buttons etc. and a training label indicating a circuit breaker (MCB), and a combination of an image of a switch with test buttons etc. and a training label indicating an earth leakage breaker (ELB). As a result, the equipment mounting state estimation unit 113 can estimate the type of switch based on, for example, the presence or absence of images of test buttons etc. in the image portion of the switch. The device mounting status estimation unit 113 then stores device status information, which indicates whether or not a device is mounted on the device mounting unit and the status of the mounted device, and device type information, which indicates the type of device, in the local placement storage unit 132, associating them with the device mounting unit location information of the corresponding device mounting unit.Furthermore, the device mounting state estimation unit 113 notifies the device information reading unit 114 of the image region position information corresponding to the image of the device mounted on the device mounting unit, which has been notified by the image region extraction unit 118. In addition, the device mounting state estimation unit 113 estimates the feature vectors of each image region corresponding to the estimated image of the device, for example, using a feature estimation model. Here, the device mounting state estimation unit 113 estimates the feature vectors of each image region corresponding to the image of the device mounting unit or the image of the device mounted on the device mounting unit, for example, using a feature estimation model. Here, the feature estimation model is, for example, represented by a CNN and is a trained model for obtaining the feature vectors of the image region or the captured image. Then, the device mounting state estimation unit 113 stores the feature information indicating the estimated feature vectors in the local placement storage unit 132, associating it with the corresponding device mounting unit position information.

[0026] The device information reading unit 114 reads device information contained in at least one device image for each of the multiple captured images, based on the aforementioned in-image position and at least one type of device. Specifically, the device information reading unit 114 selects a device mounting image from among multiple image region information notified by the image region extraction unit 118 by referring to the device mounting location information stored in the local placement memory unit 132. The device information reading unit 114 also selects image region information corresponding to the image region location information notified by the device mounting state estimation unit 113 from among the multiple image region information notified by the image region extraction unit 118. Next, the device information reading unit 114 reads the device information contained in the selected image region using a character recognition model for recognizing characters contained in the image region indicated by each of the selected image region information. Here, the character recognition model is, for example, represented by a CNN and is a trained model for recognizing strings that indicate device information contained in an image region. The device information reading unit 114 then stores the read device information in the local placement storage unit 132, associating it with the device mounting position information of the corresponding device mounting part. The device information reading unit 114 also selects image area information and image area position information of a captured image that corresponds to the image type information stored in the captured image storage unit 131, which is notified by the image area extraction unit 118 that the image is a captured image of the entire panel with the door closed. Based on the selected image area position information, it identifies the image area information corresponding to the part in which the panel information is written. The device information reading unit 114 then reads the panel information contained in the image area indicated by the identified image area information using the aforementioned character recognition model. The device information reading unit 114 then stores the read panel information in the panel diagram information storage unit 134, associating it with the corresponding location information and installation position information stored in the captured image storage unit 131.

[0027] The captured image position relationship estimation unit 115 estimates the positional relationships between multiple captured images. Specifically, the captured image position relationship estimation unit 115 selects image region information and image region position information of captured images corresponding to image type information indicating that the image is a captured image of the entire board with the door open, which is notified by the image region extraction unit 118, and estimates the feature vector of the image region indicated by each of the selected image region information. The captured image position relationship estimation unit 115 also estimates the feature vector of each captured image indicated by the captured image information corresponding to image type information indicating that the image is a captured image of a part of the board with the door open, which is stored in the captured image storage unit 131. Here, the captured image position relationship estimation unit 115 estimates the feature vector of the image region or captured image using, for example, a feature estimation model. Here, the feature estimation model is, for example, represented by a CNN, and is a trained model for finding the feature vector of the image region or captured image. The captured image position relationship estimation unit 115 then determines the positional relationship between captured images based on the similarity, for example, the Euclidean distance, between the feature vectors of each of the multiple image regions constituting the captured image of the entire board with the door open, and the feature vectors of the captured images of a part of the board with the door open, using the image type information corresponding to the captured image information indicating that it is a captured image of a part of the board with the door open. Furthermore, based on the determined positional relationship, the captured image position relationship estimation unit 115 generates image position information indicating the coordinates of the representative position of the target portion of each captured image of a part of the board with the door open, within the entire board, and stores it in the captured image arrangement storage unit 133 in association with the image identification information of the captured image.

[0028] The control panel information generation unit 116 generates control panel information based on the image position of at least one device mounting part in the device mounting part image, whether or not a device is mounted on at least one device mounting part, the status of the device, the type of device mounted on the device mounting part, device information, and the positional relationship between multiple captured images. Specifically, the control panel information generation unit 116 calculates absolute position information for the device mounting part, which indicates the global coordinates of the device mounting part within the entire control panel, based on the device mounting part position information and image identification information stored in the local placement storage unit 132, and the image position information and image identification information stored in the captured image placement storage unit 133. Then, the control panel information generation unit 116 associates the device status information, device type information, and device information stored in the local placement storage unit 132, the location information and installation position information stored in the captured image storage unit 131, and the calculated absolute position information for the device mounting part with each other and stores them in the control panel information storage unit 134. Here, the control panel information generation unit 116 refers to the image position information stored in the captured image arrangement storage unit 133 and selects two captured images in which the target areas are adjacent to each other. Then, the control panel information generation unit 116 refers to the feature quantity information stored in the local arrangement storage unit 132 and identifies any device mounting part images or device images included in the two selected captured images that have similar feature quantity vectors indicated by the feature quantity information. The control panel information generation unit 116 then stores in the control panel information storage unit 134 only the combination of various information corresponding to the location information of one of the device mounting parts, assuming that the identified device mounting part images or device images were taken of the same device mounting part or device.

[0029] When the control panel information notification unit 117 receives control panel information request information transmitted from the terminal device 5, it generates control panel information notification information containing various information stored in the control panel information storage unit 134 in response. The control panel information notification unit 117 then transmits the generated control panel information notification information to the terminal device 5 that sent the control panel information request information.

[0030] Next, the operation of the control panel information management system according to this embodiment will be explained with reference to Figure 7. First, when a user possessing the terminal device 3 performs the aforementioned control panel photography preparation operation to the input unit 305 of the terminal device 3, the terminal device 3 receives the control panel photography preparation operation, forms the aforementioned photography method notification image, and displays it on the display unit 304 (step S1). At this time, the terminal device 3 stores the location information and installation position information of the control panel to be photographed, which the user entered in the aforementioned control panel photography preparation operation, in the photographed image storage unit 331.

[0031] Next, when the user performs the aforementioned imaging operation on the input unit 305, the terminal device 3 receives the imaging operation, images the panel with the imaging unit 307, and stores the captured image information obtained from the imaging operation in the captured image storage unit 331, along with image identification information that identifies the captured image information, corresponding location information and installation location information, and image type information that indicates the type of the captured image (step S2). Subsequently, the terminal device 3 again forms an imaging method notification image and displays it on the display unit 304 (step S3). After that, when the user performs the aforementioned imaging operation on the input unit 305, the terminal device 3 receives the imaging operation, images the panel with the imaging unit 307, and stores the captured image information obtained from the imaging operation in the captured image storage unit 331, along with image identification information that identifies the captured image information, corresponding location information and installation location information, and image type information that indicates the type of the captured image (step S4). Thereafter, steps S3 and S4 are repeated until all the image information necessary to generate the diagram information of the board to be photographed has been acquired.

[0032] Here, the terminal device 3 displays a notification image on the display unit 304, for example, informing the user to first photograph the entire panel with the door closed, as shown in Figure 8(A). When the user then takes an image of the entire panel with the door closed, the terminal device 3 displays a notification image on the display unit 304 informing the user to photograph the entire panel with the door open, as shown in Figure 8(B). When the user then takes an image of the entire panel with the door open, the terminal device 3 displays a notification image on the display unit 304 informing the user to photograph a part of the area around the main switch of the panel with the door open, as shown in Figure 9(A). When the user then takes an image of the entire panel with the door open, the terminal device 3 displays a notification image on the display unit 304 informing the user to photograph a part of the area around the branch switch of the panel with the door open, as shown in Figure 9(B). In this way, the terminal device 3 sequentially displays notification images on the display unit 304 that inform the user of the method of taking an image to obtain the image necessary for creating the panel diagram information.

[0033] Returning to Figure 7, when the terminal device 3 determines that it has acquired the image information for all the images necessary to generate the panel diagram information of the target panel (step S5), it forms the aforementioned image transmission request image and displays it on the display unit 304 (step S6). At this point, when the user performs the aforementioned image transmission operation to the input unit 305 of the terminal device 3, the terminal device 3 accepts the image transmission operation and generates image notification information including the image information, location information, installation location information, and image type information stored in the image storage unit 331 (step S7). Then, the generated image notification information is transmitted from the terminal device 3 to the panel diagram information management server 1 (step S8).

[0034] Meanwhile, when the control panel information management server 1 acquires captured image notification information, it extracts the captured image information, location information, installation location information, image type information, and image identification information contained in the acquired captured image notification information, and stores these extracted pieces of information in the captured image storage unit 131, associating them with each other (step S9). Next, for each captured image indicated by the captured image information stored in the captured image storage unit 131, the control panel information management server 1 extracts multiple characteristic image regions contained in the captured image (step S10).

[0035] Next, the control panel information management server 1 identifies at least one image of an equipment mounting part from the extracted image region for each of the multiple captured image information stored in the captured image storage unit 131, and estimates the in-image position of the identified equipment mounting part image within the captured image. Here, for example, the control panel information management server 1 estimates the in-image positions of equipment mounting part images A41 and A43 in Figure 9(B). Then, as shown in Figure 7, the control panel information management server 1 stores the estimated in-image position information of the equipment mounting part in the local placement storage unit 132, associating it with the image identification information (step S11).

[0036] Subsequently, the control panel information management server 1 estimates, for each of the multiple captured images, whether or not equipment is mounted on at least one equipment mounting section included in the captured image, the state of said equipment, and the type of equipment mounted on the equipment mounting section. Here, for example, the control panel information management server 1 estimates the state of equipment and the type of equipment based on equipment images A42 and A44 in Figure 9(B). Also, if the equipment is a switch, the control panel information management server 1 estimates the state of the switch to be "on" based on image A51 in Figure 10(A), and the state of the switch to be "off" based on image A52 in Figure 10(A). Alternatively, if the equipment is a switch, the control panel information management server 1 estimates the state of the switch to be "on" based on image A61 in Figure 10(B), and the state of the switch to be "off" based on image A62 in Figure 10(B). Furthermore, if the device is a switch, the panel diagram information management server 1 estimates whether the switch is a circuit breaker (MCB) or an earth leakage breaker (ELB) based on the presence or absence of the button image portion BU63 in the captured image in Figure 11. Then, as shown in Figure 7, the panel diagram information management server 1 stores device status information indicating whether or not the device is mounted on the device mounting section and the state of the mounted device, and device type information indicating the type of device, in the local placement storage unit 132 in association with the device mounting section location information of the corresponding device mounting section (step S12). The panel diagram information management server 1 also estimates the feature vectors of each image region corresponding to the image of the device mounted on the device mounting section. Then, the panel diagram information management server 1 stores the feature information indicating the estimated feature vectors in the local placement storage unit 132 in association with the device mounting section location information (step S13).

[0037] Next, for each of the multiple captured images, the control panel information management server 1 reads the equipment information contained in at least one equipment image based on the aforementioned in-image position and at least one equipment type. Here, for example, the control panel information management server 1 reads the equipment information contained in each of the image regions A41, A42, A43, and A44 included in the captured image in Figure 9(B). Then, as shown in Figure 7, the control panel information management server 1 stores the read equipment information in the local placement storage unit 132 in association with the equipment mounting position information of the corresponding equipment mounting part (step S14). The control panel information management server 1 also selects the image region information and image region position information of the captured image of the entire control panel with the door closed, which is stored in the captured image storage unit 131, and identifies the image region information corresponding to the part in which the control panel information is written based on the selected image region position information. Then, the control panel information management server 1 reads the control panel information contained in the image region indicated by the identified image region information. Here, the control panel information management server 1 reads the control panel information contained in image region A11, for example, in the captured image in Figure 8(A). Then, as shown in Figure 7, the control panel information management server 1 stores the read control panel information in the control panel information storage unit 134, associating it with the corresponding location information and installation position information stored in the captured image storage unit 131 (step S15).

[0038] Next, the board diagram information management server 1 estimates the positional relationship between multiple captured images. Here, the board diagram information management server 1 estimates the feature vectors of each of the multiple image regions, including, for example, the image region A21 shown in Figure 8(B), and also estimates the feature vectors of the captured image shown in Figure 9(A). The board diagram information management server 1 then determines that the two feature vectors are similar to each other and estimates the positional relationship of the target portion of the captured image shown in Figure 9(A) within the entire board. Based on the identified positional relationship, the board diagram information management server 1 generates image position information indicating the coordinates of the representative position of the target portion of each captured image of a part of the board with the door open within the entire board, and stores it in the captured image arrangement storage unit 133 in association with the image identification information of the captured image (step S16).

[0039] Subsequently, the control panel information management server 1 generates control panel information based on the image position of at least one equipment mounting part in the equipment mounting part image, whether or not equipment is mounted on at least one equipment mounting part, the status of the equipment, the type of equipment mounted on the equipment mounting part, equipment information, and the positional relationship between multiple captured images. Then, the control panel information management server 1 associates the equipment status information, equipment type information, and equipment information stored in the local placement storage unit 132 with the location information and installation position information stored in the captured image storage unit 131, and the calculated absolute position information of the equipment mounting part, and stores them in the control panel information storage unit 134 (step S17).

[0040] Next, when the administrator performs the aforementioned panel diagram information request operation to the input section of the terminal device 5, the terminal device 5 receives the panel diagram information request operation and generates panel diagram information request information (step S18). Then, the generated panel diagram information request information is transmitted from the terminal device 5 to the panel diagram information management server 1 (step S19). Meanwhile, when the panel diagram information management server 1 receives the panel diagram information request information, it generates panel diagram information notification information containing various information stored in the panel diagram information storage unit 134, as shown in Figure 12 (step S20). Then, the panel diagram information notification unit 117 transmits the generated panel diagram information notification information to the terminal device 5, the source of the panel diagram information request information. Meanwhile, when the terminal device 5 receives the panel diagram information notification information, it extracts the panel diagram information contained in the acquired panel diagram information notification information and stores it in the panel diagram information storage unit (step S21). Subsequently, the terminal device 5 forms a panel diagram image based on the panel diagram information stored in the panel diagram information storage unit and displays it on the display unit (step S22).

[0041] Next, the image acquisition process performed by the terminal device 3 according to this embodiment will be described with reference to Figure 13. This image acquisition process is initiated when power is turned on to the terminal device 3, a program for executing the image acquisition process is started in the terminal device 3, and the user performs the aforementioned panel shooting preparation operation to the input unit 305. Here, when the image management unit 313 receives the aforementioned location information and installation location information of the panel to be photographed from the input unit 305 during the panel shooting preparation operation, it stores the input location information and installation location information in the image storage unit 331.

[0042] First, the display control unit 311 receives the aforementioned panel shooting preparation operation and forms a shooting method notification image based on the aforementioned image type information notified by the image management unit 313 and displays it on the display unit 304 (step S201). Next, the imaging control unit 312 determines whether or not it has received the aforementioned imaging operation performed by the user on the input unit 305 (step S202). If the imaging control unit 312 determines that it has not received the imaging operation (step S202: No), the process in step S201 is executed again.

[0043] Meanwhile, when the imaging control unit 312 determines that it has received an imaging operation (step S202: Yes), it generates imaging command information instructing the imaging unit 307 to photograph the panel and transfers it to the imaging unit 307. At this point, the imaging unit 307 photographs the panel when it receives the imaging command information and transfers the resulting image information to the image management unit 313. The image management unit 313 then stores the transferred image information in the image storage unit 331, associating it with the image identification information, the corresponding location information and installation position information, and the image type information most recently notified to the display control unit 311 (step S203).

[0044] Next, the image management unit 313 determines whether all the necessary image information for generating the board diagram information of the board to be photographed has been acquired (step S203). If the image management unit 313 determines that there is necessary image information that has not yet been acquired (step S203: NO), it selects one of the image type information corresponding to the image information that has not yet been acquired and notifies the display control unit 311. Then, the process in step S201 is executed again. On the other hand, if the image management unit 313 determines that all the necessary image information for generating the board diagram information of the board to be photographed has been acquired (step S203: Yes), it notifies the display control unit 311 of the completion of photography information. Then, the display control unit 311 forms the aforementioned image transmission reminder image and displays it on the display unit 304 (step S204).

[0045] Next, the image information notification unit 314 determines whether or not it has received the aforementioned captured image transmission operation performed by the user to the input unit 305 (step S205). Here, if the image information notification unit 314 determines that it did not receive the captured image transmission operation within a predetermined waiting time after the image transmission reminder image was displayed on the display unit 304 (step S205: No), the captured image information acquisition process ends there. On the other hand, if the image information notification unit 314 determines that it has received the captured image transmission operation (step S205: Yes), it generates captured image notification information including the captured image information, location information, installation location information, and image type information stored in the captured image storage unit 331. Then, after the image information notification unit 314 sends the generated captured image notification information to the panel diagram information management server 1 (step S206), the captured image information acquisition process ends.

[0046] Next, the panel diagram information management process executed by the panel diagram information management server 1 according to this embodiment will be explained with reference to Figure 14. This panel diagram information management process is started when a program for executing the panel diagram information management process is started on the panel diagram information management server 1. First, the image information acquisition unit 111 determines whether or not it has acquired the captured image notification information transmitted from the terminal device 3 (step S101). If the image information acquisition unit 111 determines that it has not acquired the captured image notification information (step S101: No), the process of step S111 described later is executed. On the other hand, if the image information acquisition unit 111 determines that it has acquired the captured image notification information (step S101: Yes), it extracts the captured image information, location information, installation location information, image type information, and image identification information contained in the acquired captured image notification information, and stores these extracted pieces of information in the captured image storage unit 131 in association with each other (step S102). Next, the image region extraction unit 118 extracts multiple characteristic image regions from each of the captured images indicated by the captured image information stored in the captured image storage unit 131 (step S103). Here, the image region extraction unit 118 notifies the device mounting unit position estimation unit 112, the device mounting state estimation unit 113, and the device information reading unit 114 of the image region information indicating each of the extracted multiple image regions corresponding to each of the multiple captured images, and the image region position information indicating the relative position coordinates of the image region within the entire captured image. The image region extraction unit 118 also notifies the captured image position relationship estimation unit 115 of the image region information and image region position information corresponding to each of the multiple captured image information, as well as the image type information stored in the captured image storage unit 131.

[0047] Next, the device mounting part position estimation unit 112 estimates the in-image position of the device mounting part image within the captured image for each of the multiple captured image information stored in the captured image storage unit 131. Then, based on the image region position information notified by the image region extraction unit 118, the device mounting part position estimation unit 112 identifies the in-image position, which is the relative position coordinate of the device mounting part image within the entire captured image, and stores the identified in-image position information in the local placement storage unit 132 in association with the image identification information (step S104).

[0048] Subsequently, the device mounting status estimation unit 113 estimates, for each of the multiple captured images, whether or not a device is mounted on at least one device mounting unit included in the captured image, the state of the device, and the type of device mounted on the device mounting unit. The device mounting status estimation unit 113 then stores the device status information indicating whether or not a device is mounted on the device mounting unit and the state of the mounted device, and the device type information indicating the type of device, in the local placement storage unit 132, associating them with the device mounting unit location information of the corresponding device mounting unit (step S105). At this time, the device mounting status estimation unit 113 notifies the device information reading unit 114 of the image region location information corresponding to the image region corresponding to the image of the device mounted on the device mounting unit, which has been notified by the image region extraction unit 118.

[0049] Next, the device mounting state estimation unit 113 estimates the feature vectors for each image region corresponding to the estimated device image. Then, the device mounting state estimation unit 113 stores the feature information representing the estimated feature vectors in the local placement storage unit 132, associating it with the corresponding device mounting position information (step S106).

[0050] Next, the device information reading unit 114 reads device information contained in at least one device image for each of the multiple captured images, based on the aforementioned in-image position and at least one type of device. Then, the device information reading unit 114 stores the read device information in the local placement storage unit 132, associating it with the device mounting position information of the corresponding device mounting part (step S107).

[0051] Subsequently, the device information reading unit 114 selects image area information and image area position information of the captured image that correspond to the image type information stored in the captured image storage unit 131, which is notified by the image area extraction unit 118 that the image is a captured image of the entire panel with the door closed. Based on the selected image area position information, it identifies the image area information corresponding to the portion in which the panel information is written. Then, the device information reading unit 114 reads the panel information contained in the image area indicated by the identified image area information using the aforementioned character recognition model. Finally, the device information reading unit 114 stores the read panel information in the panel diagram information storage unit 134, associating it with the corresponding location information and installation location information stored in the captured image storage unit 131 (step S108).

[0052] Next, the captured image position relationship estimation unit 115 estimates the positional relationship between multiple captured images. Then, based on the estimated positional relationship, the captured image position relationship estimation unit 115 generates image position information indicating the coordinates of the representative position of the target portion of each captured image of a part of the board within the entire board, and stores it in the captured image arrangement storage unit 133 in association with the image identification information of the captured image (step S109).

[0053] Next, the control panel information generation unit 116 generates control panel information based on the image position of at least one equipment mounting part in the equipment mounting part image, whether or not equipment is mounted on at least one equipment mounting part, the status of the equipment, the type of equipment mounted on the equipment mounting part, equipment information, and the positional relationship between multiple captured images. Then, the control panel information generation unit 116 associates the equipment status information, equipment type information, and equipment information stored in the local placement storage unit 132 with the location information and installation position information stored in the captured image storage unit 131 with the calculated absolute position information of the equipment mounting part and stores them in the control panel information storage unit 134 (step S110).

[0054] Subsequently, the control panel information notification unit 117 determines whether or not it has received the control panel information request information transmitted from the terminal device 5 (step S111). If the control panel information notification unit 117 determines that it has not received the control panel information request information (step S111: No), the process in step S101 is executed again. On the other hand, if the control panel information notification unit 117 determines that it has received the control panel information request information (step S111: Yes), it generates control panel information notification information that includes various information stored in the control panel information storage unit 134. Then, the control panel information notification unit 117 transmits the generated control panel information notification information to the terminal device 5, the source of the control panel information request information (step S112). Next, the process in step S101 is executed again.

[0055] As described above, according to the panel diagram information management system of this embodiment, the equipment mounting status estimation unit estimates for each of the multiple captured images whether or not equipment is mounted on at least one equipment mounting section included in the captured image, and the type of equipment mounted on the equipment mounting section. As a result, it is possible to generate panel diagram information that includes information on whether or not equipment is mounted on the equipment mounting section, thereby generating appropriate panel diagram information for managing equipment mounted on control panels, distribution boards, lighting boards, power panels, etc.

[0056] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above. For example, at least one of the equipment installation unit estimation model, equipment estimation model, character recognition model, and feature quantity estimation model used in the control panel information management server 1 may be updated periodically or in response to requests from users or administrators based on training data.

[0057] In this embodiment, when the image management unit 313 of the terminal device 3 receives the aforementioned panel shooting preparation operation, it may generate directory information for grouping and managing all the captured image information necessary for generating the panel diagram information of the panel to be photographed. Here, after the image management unit 313 receives the panel shooting preparation operation and the first captured image information is transferred from the imaging unit 307, it generates directory information indicating that the transferred captured image information should be associated with a newly generated group of group identification information, and stores this information in the captured image storage unit 331. Here, the group identification information may, for example, represent the date on which the shooting preparation operation was received. Then, from the time the image management unit 313 receives the panel shooting preparation operation until it determines that all the captured image information necessary for generating the panel diagram information of the panel to be photographed has been acquired, it updates the directory information each time captured image information is transferred from the imaging unit 307 so that the transferred captured image information is associated with the same group identification information. Then, when a user performs the aforementioned captured image transmission operation to the input unit 305 of the terminal device 3, the terminal device 3 accepts the captured image transmission operation and generates captured image notification information, which includes directory information along with the captured image information stored in the captured image storage unit 331. The generated captured image notification information is then transmitted from the terminal device 3 to the panel diagram information management server 1. Meanwhile, when the panel diagram information management server 1 acquires the captured image notification information, it extracts the directory information along with the captured image information contained in the acquired captured image notification information, associates these extracted pieces of information with each other, and stores them in the captured image storage unit 131. Subsequently, when the administrator performs an operation to display the captured image information stored in the panel diagram information management server 1's captured image storage unit 131 on the input unit of the terminal device 5, the terminal device 5 first refers to the directory information and displays a folder image on the display unit that has been assigned the name indicated by the aforementioned group identification information. Then, when the administrator performs an operation to select a folder image on the input unit, a file image indicating the captured image information is displayed.

[0058] Furthermore, the image management unit 313 may, after receiving the disc shooting preparation operation, recognize the disc name from the portion of the image showing the disc name included in the captured image indicated by the captured image information, based on the captured image information initially transferred from the imaging unit 307, and generate group identification information based on the recognized disc name.

[0059] In this embodiment, the control panel information management server 1 may include: an equipment number identification unit that identifies the number of devices included in each of the multiple captured images based on the equipment type estimated by the equipment mounting state estimation unit 113 for each of the multiple captured images; and an equipment use identification unit that determines the use of the devices that were the subject of each of the multiple captured images based on the identified number of devices. Specifically, the control panel information management server 1 may have a switch number identification unit that identifies the number of switch images included in each of the multiple captured images based on the equipment type of the devices mounted on the equipment mounting unit estimated by the equipment mounting state estimation unit 113 for each of the multiple captured images. Furthermore, the control panel information management server 1 may have a switch image type identification unit that determines, based on the identified number of switch images, whether each of the multiple captured images is an image of a main switch inserted in a main line or an image of a branch switch inserted in a branch line. Here, the switch image type identification unit may, for example, determine that an image is a main switch if there is one switch image included in the captured image, and determine that an image is a branch switch if there are six or more switch images. Furthermore, the switch image type discrimination unit may generate directory information to store in the image storage unit 131, associating it with the type of switch identified, i.e., switch type information indicating whether it is a main switch or a branch switch, for each of the multiple captured image information stored in the captured image storage unit 131.

[0060] In this embodiment, the control panel information management server 1 may further include a capture image division unit (not shown) that generates capture image information for each of the multiple capture images obtained by dividing a single capture image containing images of multiple devices. For example, as shown in Figure 15(A), this capture image division unit generates capture image information for a capture image of the entire control panel, corresponding to three image regions A71, A72, and A73 (see Figures 15(B) to (D)). Here, the capture image division unit may generate the capture image information such that the number of device mounting parts included in each of the multiple capture images generated by division is 24 or less (2 rows × 12 or less), based on the positions of device mounting parts estimated by the device mounting part position estimation unit 112 included in the capture image.

[0061] In this embodiment, the terminal device 3 may further include: an equipment mounting part image identification unit (not shown) that identifies at least one equipment mounting part image, which is a part corresponding to an equipment mounting part included in a captured image obtained by taking a photograph of a control panel; and a captured image determination unit that determines whether the captured image is suitable for generating control panel diagram information based on the number of pixels of each of the identified at least one equipment mounting part image. Here, the captured image determination unit determines, for example, that the captured image is suitable for creating control panel diagram information if the number of pixels of the identified equipment mounting part image is equal to or greater than a preset number of pixels. For example, if the number of pixels of the equipment mounting part image is equal to or greater than a preset pixel count threshold, the captured image determination unit determines whether the captured image is suitable for creating control panel diagram information. Here, the pixel count threshold is set to, for example, 270,000 pixels. In addition, if multiple equipment mounting part images are identified, the captured image determination unit may determine whether the average, median, maximum, or minimum value of the number of pixels of each of the multiple equipment mounting part images is equal to or greater than the pixel count threshold. Furthermore, if the display control unit 311 of the terminal device 3 determines that the captured image is unsuitable for creating diagram information, it may create a message image prompting the user to take another picture of the panel and display it on the display unit 304.

[0062] In this embodiment, the control panel information management server may identify the correspondence between the local coordinates of the equipment mounting position where the equipment is installed in each of the images shown by at least one image information, and the global coordinates of the equipment mounting position in the entire control panel. It may then generate image information notification information including the mounting position correspondence information showing the identified correspondence and the corresponding image information, and transmit this information to a terminal device owned by the administrator.

[0063] The control panel information management system according to this modified example comprises, for example, a control panel information management server 2001 as shown in Figure 16, and a terminal device 2005 owned by the administrator. In Figure 16, components similar to those in the embodiment are denoted by the same reference numerals as in Figure 3. The control panel information management server 2001 shown in Figure 16 has the same functional components as the control panel information management server 1 described in the embodiment, but some are omitted in Figure 16. The hardware configuration of the control panel information management server 2001 and the terminal device 5 is the same as that of the control panel information management server 1 and the terminal device 5 described in the embodiment. In addition to the functional components described in the embodiment, the control panel information management server 2001 includes a mounting part position correspondence relationship identification unit 2119, a captured image notification unit 2120, and a control panel information update unit 2121. When the mounting part position correspondence relationship identification unit 2119 receives notification of image identification information corresponding to at least one captured image information to be transmitted from the captured image notification unit 2120 to the terminal device 2005, it identifies the correspondence between the local coordinates of the equipment mounting part position where the equipment is mounted in each captured image indicated by the captured image information corresponding to the identified image identification information, and the global coordinates of the equipment mounting part in the entire panel, based on the relative position information of the equipment mounting part corresponding to each notified image identification information stored in the local position storage unit 132 and the image position information corresponding to each notified image identification information stored in the captured image position storage unit 133. The mounting part position correspondence relationship identification unit 2119 then notifies the captured image notification unit 2120 of the correspondence relationship identified for at least one image identification information, i.e., the mounting part position correspondence relationship information indicating the combination of local coordinates and global coordinates.

[0064] When the captured image notification unit 2120 receives the captured image request information described below from the terminal device 2005, it selects from the captured image information stored in the captured image storage unit 131 that is identified by at least one image identification piece included in the captured image request information, and notifies the attachment location correspondence relationship identification unit 2119 of this at least one image identification piece. Then, when the captured image notification unit 2120 receives the aforementioned attachment location correspondence relationship information from the attachment location correspondence relationship identification unit 2119, it generates captured image information notification information including the selected at least one captured image information, its image identification information, and the notified attachment location correspondence relationship information, and transmits it to the terminal device 2005.

[0065] When the panel diagram information update unit 2120 receives panel diagram information update request information, which includes the updated panel diagram information described later, transmitted from the terminal device 2005, it extracts the panel diagram information contained in the acquired panel diagram information update request information. Then, the panel diagram information update unit 2120 updates the panel diagram information stored in the panel diagram information storage unit 134 using the extracted panel diagram information.

[0066] The terminal device 2005 includes a display unit 504 and an input unit 505 as described in the embodiment. The terminal device 2005 also includes a CPU (not shown), a main memory unit (not shown), and an auxiliary memory unit (not shown). The CPU functions as a panel diagram information acquisition unit 511, a captured image acquisition unit 2512, a composite image generation unit 2513, a display control unit 514, a panel diagram information update unit 2515, and a panel diagram information update request unit 2516 by reading a program stored in the auxiliary memory unit into the main memory unit and executing it. The auxiliary memory unit includes a panel diagram information storage unit 531 and a captured image storage unit 2532. The panel diagram information storage unit 531 stores absolute position information of an equipment mounting unit indicating the global coordinates of the equipment mounting unit in the entire panel, equipment status information indicating whether or not equipment is mounted on the equipment mounting unit and the state of the mounted equipment, equipment type information indicating the type of equipment mounted, equipment image feature quantity information indicating the feature quantities of the image of the equipment part, and equipment information, all associated with image identification information. The captured image storage unit 2532 stores at least one captured image information in association with image identification information that identifies the captured image information and the aforementioned mounting part position correspondence relationship information that corresponds to the captured image indicated by the captured image information.

[0067] As described in the embodiment, when an administrator performs the aforementioned panel diagram information request operation via the input unit 505, the panel diagram information acquisition unit 511 receives the panel diagram information request operation, generates panel diagram information request information, and transmits it to the panel diagram information management server 1. As a result, the panel diagram information acquisition unit 511 acquires the panel diagram information notification information transmitted from the panel diagram information management server 1, extracts the panel diagram information contained in the acquired panel diagram information notification information, and stores it in the panel diagram information storage unit 531.

[0068] When an administrator requests an image capture operation via the input unit 505, which includes at least one image identification piece of information, and requests the control panel information management server 2001 to send image capture information, the image capture unit 2512 receives the request, generates the image capture request information, and sends it to the control panel information management server 1. As a result, the image capture unit 2512 receives the image capture information notification information sent from the control panel information management server 1, extracts the image capture information, image identification information, and mounting part position correspondence pattern contained in the acquired image capture information notification information, and stores these extracted pieces of information in the image capture storage unit 2532, associating them with each other.

[0069] When the composite image generation unit 2513 receives a captured image selection operation performed by the administrator via the input unit 505 to select captured image information, it selects one captured image information and its corresponding mounting part position correspondence relationship information from the captured image information stored in the captured image storage unit 2532. The composite image generation unit 2513 then refers to the absolute position information of the equipment mounting part stored in the panel diagram information storage unit 531 and the mounting part position correspondence relationship information stored in the captured image storage unit 2532 to identify the local coordinates of the equipment mounting part within the captured image indicated by the selected captured image information. The composite image generation unit 2513 also refers to the equipment status information stored in the panel diagram information storage unit 531 to identify the status of the equipment mounted on the equipment mounting part within the captured image indicated by the selected captured image information. The composite image generation unit 2513 then generates icon images representing the status of each device included in the captured image indicated by the selected captured image information, based on the identified local coordinates and the status of the devices. It then generates a composite image by combining the generated icon images with the captured image indicated by the selected captured image information. The display control unit 514 displays the composite image generated by the composite image generation unit 2513 on the display unit 504.

[0070] When an administrator performs a modification operation to correct the information stored in the board diagram information storage unit 531 via the input unit 505 while confirming the composite image displayed on the display unit 504, the board diagram information update unit 2515 accepts the modification operation and updates the information stored in the board diagram information storage unit 531 using the modified information specified by the administrator in the modification operation.

[0071] When an administrator performs a panel diagram information update request operation via the input unit 505 to request that the panel diagram information storage unit 134 of the panel diagram information management server 2001 update the information stored in the panel diagram information storage unit 531, the panel diagram information update request unit 2516 accepts the panel diagram information update request operation and generates panel diagram information update request information that includes the updated information stored in the panel diagram information storage unit 531. Then, the panel diagram information update request unit 2516 transmits the generated panel diagram information update request information to the panel diagram information management server 2001.

[0072] Terminal device 5 displays a composite image GA81 on the display unit 504, which is created by combining an image of a panel equipped with multiple switches, such as the one shown in Figure 17, with icon images BU81 and BU82 indicating the on / off state of each of the switches. When terminal device 5 is displaying the composite image A81 on its display unit, and a user performs the aforementioned modification operation by switching the on / off state represented by icon images BU81 and BU82 on the input unit of terminal device 5, the device state information of each of the multiple switches stored in the panel diagram information storage unit 531 is updated to match the on / off state of the icon images BU81 and BU82. That is, if a switch is in the on state, it is updated to the off state, and if it is in the off state, it is updated to the on state.

[0073] With this configuration, the administrator can modify the various types of information stored in the control panel information storage unit 134, thus ensuring that the various types of information stored in the control panel information storage unit 134 are maintained with appropriate information.

[0074] In this embodiment, some of the functions of the control panel information management server 1 may be implemented in terminal device 3 or terminal device 5.

[0075] Furthermore, the various functions of the control panel information management server 1 and terminal devices 3 and 5 according to the present invention can be implemented using a normal computer system, without requiring a dedicated system. For example, the control panel information management server 1 and terminal device 3 that perform the above processing may be configured by distributing a program for performing the above operations to computers connected to a network, stored on a non-temporary recording medium (such as a CD-ROM (Compact Disc Read Only Memory)) that can be read by the computer system, and then installing the program into the computer system.

[0076] Furthermore, the method of providing the program to the computer is arbitrary. For example, the program may be uploaded to a bulletin board system (BBS) on a communication line and distributed to the computer via the communication line. The computer then launches this program and executes it under the control of the operating system (OS), just like any other application. In this way, the computer functions as the panel information management server 1 and terminal devices 3 and 5 that perform the processes described above.

[0077] Although embodiments and variations of the present invention have been described above, the present invention is not limited thereto. The present invention includes embodiments and variations that are appropriately combined, and those that are appropriately modified thereto. [Industrial applicability]

[0078] This invention is suitable as a system for creating panel diagrams for distribution boards, control panels, and the like. [Explanation of Symbols]

[0079] 1: Panel diagram information management server, 3,5: Terminal device, 101,301: CPU, 102,302: Main memory unit, 103,303: Auxiliary memory unit, 106,306: Communication unit, 109,309: Bus, 111: Image information acquisition unit, 112: Equipment mounting unit position estimation unit, 113: Equipment mounting state estimation unit, 114: Equipment information reading unit, 115: Captured image position relationship estimation unit, 116: Panel information generation unit, 117: Panel information notification unit, 131,331: Captured image storage unit, 132: Local placement storage unit, 133: Overall Layout storage unit, 134: Panel diagram information storage unit, 304, 504: Display unit, 305, 505: Input unit, 307: Imaging unit, 311, 514: Display control unit, 312: Imaging control unit, 313: Image management unit, 314: Image information notification unit, 511: Panel diagram information acquisition unit, 2119: Mounting unit position correspondence identification unit, 2120: Captured image notification unit, 2121, 2515: Panel diagram information update unit, 2512: Captured image acquisition unit, 2513: Composite image generation unit, 2516: Panel diagram information update request unit, NW1: Wide area network

Claims

1. A photographic image storage unit stores multiple images obtained by photographing the disc in multiple pre-set states, in association with disc identification information that identifies the disc. For each of the multiple captured images, a device mounting position estimation unit estimates the in-image position of a device mounting image, which is an image of at least one device mounting part included in the captured image, within the captured image. For each of the multiple captured images, the device mounting state estimation unit estimates whether the device is mounted on at least one of the device mounting sections included in the captured image, the state of the device, and the type of device mounted on the device mounting section. A device information reading unit reads device information contained in at least one of the device mounting part image and the device image, based on the position within the image of at least one of the device images and the type of at least one of the device, for each of the multiple captured images. A unit for estimating the positional relationship between multiple captured images, The system includes a panel diagram information generation unit that generates panel diagram information based on the position within the image of at least one of the equipment mounting parts, whether or not equipment is mounted on at least one of the equipment mounting parts and the type of equipment mounted on the equipment mounting part, equipment information and the positional relationship between a plurality of captured images. Board diagram information management system.

2. Display unit and Imaging unit, A display control unit that forms a shooting method notification image to inform the user of the shooting method when photographing the aforementioned disc and displays it on the display unit, The system further includes an image management unit that, after the shooting method notification image is displayed on the display unit and the shooting by the imaging unit is completed, stores the captured image information, which indicates the captured image obtained, in the captured image storage unit in association with image type information determined based on the shooting method. The control panel information management system according to claim 1.

3. The aforementioned image position relationship estimation unit extracts characteristic image regions contained in each of the multiple captured images and estimates the positional relationship of the multiple captured images based on the similarity of the feature vectors of the image regions in each of the multiple captured images. A control panel information management system according to claim 1 or 2.

4. The image management unit generates directory information for grouping and managing all the captured image information necessary for generating the panel diagram information of one panel. The control panel information management system according to claim 2.

5. Based on the device type estimated by the device mounting state estimation unit for each of the multiple captured images, the device count determination unit determines the number of devices included in each of the captured images, The system further includes a device use determination unit that determines the use of the device that was the subject of each of the multiple captured images, based on the number of identified devices. A control panel information management system according to claim 1 or 2.

6. The system further includes a shooting image splitting unit that generates shooting image information for each of a plurality of shooting images obtained by splitting a single shooting image containing images of multiple devices, from shooting image information that shows a single shooting image including images of multiple devices. A control panel information management system according to claim 1 or 2.

7. A device mounting part image identification unit identifies at least one of the device mounting part images included in the captured image, The system further includes a captured image determination unit that determines whether the captured image is suitable for generating the control panel information based on the number of pixels in each of the identified images of the device mounting part, A control panel information management system according to claim 1 or 2.

8. A mounting part position correspondence relationship identification unit that identifies the correspondence between the local coordinates of the equipment mounting part position in each of the captured images indicated by at least one captured image information and the global coordinates of the equipment mounting part in the entire panel, The system further comprises: a composite image generation unit that, based on the correspondence described above, identifies the state of the device attached to the device mounting unit in the captured image shown by the at least one captured image information; generates icon images representing the state of each device included in the captured image shown by the at least one captured image information based on the identified state of the device; and generates a composite image by combining the generated icon images with the captured image. A control panel information management system according to claim 1 or 2.

9. A photographic image storage unit stores multiple images obtained by photographing the disc in multiple pre-set states, in association with disc identification information that identifies the disc. For each of the multiple captured images, a device mounting position estimation unit estimates the in-image position of a device mounting image, which is an image of at least one device mounting part included in the captured image, within the captured image. For each of the multiple captured images, the device mounting state estimation unit estimates whether the device is mounted on at least one of the device mounting sections included in the captured image, the state of the device, and the type of device mounted on the device mounting section. A device information reading unit reads device information contained in at least one of the device mounting part image and the device image, based on the position within the image of at least one of the device images and the type of at least one of the device, for each of the multiple captured images. A unit for estimating the positional relationship between multiple captured images, The system includes a panel diagram information generation unit that generates panel diagram information based on the position within the image of at least one of the equipment mounting parts, whether or not equipment is mounted on at least one of the equipment mounting parts and the type of equipment mounted on the equipment mounting part, equipment information and the positional relationship between a plurality of captured images. Board diagram information management device.

10. The steps include: storing multiple captured images obtained by photographing the disc in a predetermined state in a captured image storage unit, in association with disc identification information that identifies the disc; For each of the multiple captured images, the step of estimating the in-image position within the captured image of at least one captured image of a device mounting part included in the captured image, For each of the multiple captured images, the steps include: estimating whether the device is attached to at least one of the device mounting parts included in the captured image, the state of the device, and the type of device attached to the device mounting part; For each of the multiple captured images, the step of reading device information included in at least one of the device mounting part image and the device image, based on the position within the image of at least one of the device images and the type of at least one of the device, A step of estimating the positional relationship between multiple aforementioned captured images, The process includes the step of generating control panel information based on the position within the image of at least one of the equipment mounting parts, whether or not equipment is mounted on at least one of the equipment mounting parts and the type of equipment mounted on the equipment mounting part, the equipment information, and the positional relationship between a plurality of captured images. Board diagram information management method.

11. Computers, A captured image storage unit stores multiple captured images obtained by photographing the disc in multiple pre-set states, in association with disc identification information that identifies the disc. For each of the multiple captured images, a device mounting position estimation unit estimates the in-image position of an image of a device mounting part, which is an image of at least one device mounting part included in the captured image, within the captured image. For each of the multiple captured images, an equipment mounting state estimation unit estimates whether the equipment is mounted on at least one of the equipment mounting parts included in the captured image, the state of the equipment, and the type of equipment mounted on the equipment mounting part. For each of the multiple captured images, a device information reading unit reads device information contained in at least one of the device mounting part images and the device image, based on the position of the device image of at least one of the devices within the image and the type of at least one of the devices. Image position relationship estimation unit that estimates the positional relationship between multiple captured images, A panel diagram information generation unit generates panel diagram information based on the position within the image of at least one of the equipment mounting parts, whether or not equipment is mounted on at least one of the equipment mounting parts and the type of equipment mounted on the equipment mounting part, the equipment information and the positional relationship between a plurality of captured images. A program designed to function as such.

Images