Panel diagram information management system, panel diagram information management device, panel diagram information management method and program

Abstract

Provided are a panel diagram information management system, a panel diagram information management device, a panel diagram information management method, and a program that can generate appropriate panel diagram information for managing equipment mounted on control panels, distribution boards, lighting panels, power panels, etc. [Solution] The panel diagram information management server 1 comprises an equipment mounting part position estimation unit 112 that estimates the position within the captured image of the equipment mounting part image for each captured image; an equipment mounting state estimation unit 113 that estimates whether an equipment is mounted on the equipment mounting part, the state of the equipment, and the type of equipment of the mounted equipment for each of the multiple captured images; an equipment information reading unit 114 that reads equipment information contained in at least one of the equipment mounting part image and the equipment image for each of the multiple captured images; a captured image positional relationship estimation unit 115 that estimates the positional relationship between the multiple captured images; and a panel diagram information generation unit 116 that generates panel diagram information.

Description

[Technical Field]

[0001] The present invention relates to a board plan information management system, a board plan information management device, a board plan information management method, and a program. [Background technology]

[0002] An information processing device has been proposed that includes an image recognition unit that recognizes an image of a two-dimensional ID code for identifying control devices contained in an image of a control panel captured by a camera, an equipment identification unit that identifies the model information and installation location of each control device based on the recognized equipment identification image, and a layout diagram generation unit that generates a layout diagram of each control device on the control panel based on the model information and installation location of each identified control device (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-215817 Summary of the Invention [Problem to be solved by the invention]

[0004] However, there are cases where a control panel is left without any devices attached to the device attachment section. In such cases, unless it is possible to grasp the state of the control panel, including whether or not a device is attached to the device attachment section and, if a device is attached, the state of the device, for example, its on / off state, it may be impossible to obtain an appropriate panel diagram for managing the devices connected to the control panel.

[0005] The present invention has been made in consideration of the above-mentioned reasons, and aims to provide a panel diagram information management system, panel diagram information management device, panel diagram information management method, and program that can generate appropriate panel diagram information for managing equipment installed in control panels, distribution boards, lighting panels, power panels, etc. [Means for solving the problem]

[0006] In order to achieve the above object, the board plan information management system according to the present invention comprises: a photographed image storage unit that stores a plurality of photographed images obtained by photographing a board in a plurality of preset states, in association with board identification information that identifies the board; a device mounting part position estimation unit that estimates, for each of the plurality of captured images, a position within the captured image of a device mounting part image that is a captured image of at least one device mounting part included in the captured image; a device wearing state estimation unit that estimates, for each of the plurality of captured images, whether the device is attached to at least one of the device mounting sections included in the captured image, the state of the device, and the device type of the device attached to the device mounting section; a device information reading unit that reads device information included in at least one of the device mounting unit image and the device image based on a position within the image of at least one of the devices and a type of at least one of the devices for each of the plurality of captured images; a captured image positional relationship estimation unit that estimates a positional relationship between the plurality of captured images; The device is provided with a panel diagram information generation unit that generates panel diagram information based on the position within the image of the device mounting unit image of at least one of the device mounting units, whether the device is mounted to at least one of the device mounting units and the device type of the device mounted to the device mounting unit, the device information, and the positional relationship between the multiple captured images. [Effects of the Invention]

[0007] According to the present invention, the device mounting state estimation unit estimates, for each of a plurality of captured images, whether a device is mounted in at least one device mounting section included in the captured image and the device type of the device mounted in the device mounting section. This makes it possible to generate panel diagram information that includes information on whether a device is mounted in a device mounting section, thereby generating panel diagram information appropriate for managing devices mounted in control panels, distribution boards, lighting panels, power panels, etc. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of a configuration of a board plan information management system according to an embodiment of the present invention. [Figure 2] 1 is a block diagram showing a hardware configuration of a panel diagram information management system according to an embodiment; [Figure 3] 1 is a block diagram showing a functional configuration of a board diagram information management system according to an embodiment; [Figure 4] 5A and 5B are diagrams illustrating an example of information stored in a captured image storage unit according to the embodiment. [Figure 5] FIG. 1A is a diagram showing an example of information stored in a local layout storage unit according to an embodiment, and FIG. 1B is a diagram showing an example of information stored in a global layout storage unit according to an embodiment. [Figure 6] FIG. 10 is a diagram illustrating an example of information stored in a panel diagram information storage unit according to an embodiment. [Figure 7] FIG. 2 is a sequence diagram illustrating the operation of the panel diagram information management system according to the embodiment. [Figure 8] FIG. 1A is a diagram showing an example of a photographed image of a board with the door closed, and FIG. 1B is a diagram showing an example of a photographed image of a board with the door open. [Figure 9] FIG. 10A is a diagram showing an example of a photographed image of a part of the board with the door open, and FIG. 10B is a diagram showing an example of a photographed image of another part of the board with the door open. [Figure 10] FIG. 10A is a diagram showing an example of a photographed image of a part of the board, and FIG. 10B is a diagram showing an example of a photographed image of another part of the board. [Figure 11] FIG. 10 is a diagram showing an example of a photographed image of a part of a board. [Figure 12] FIG. 2 is a sequence diagram illustrating the operation of the panel diagram information management system according to the embodiment. [Figure 13] 10 is a flowchart illustrating an example of the flow of a captured image information acquisition process executed by a terminal device according to an embodiment. [Figure 14]10 is a flowchart showing an example of the flow of a panel diagram information generation process executed by a panel diagram information generation server according to an embodiment. [Figure 15] (A) is a diagram showing an example of a photographed image of a portion of the board, (B) is a diagram showing a photographed image corresponding to area A71 in (A), (C) is a diagram showing a photographed image corresponding to area A72 in (A), and (D) is a diagram showing a photographed image corresponding to area A73 in (A). [Figure 16] FIG. 10 is a block diagram showing the functional configuration of a panel diagram information management system according to a modified example. [Figure 17] 10A and 10B are diagrams showing an example of a composite image displayed on a display unit of a terminal device according to a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0009] A panel diagram information management system according to an embodiment of the present invention will be described below with reference to the drawings. The panel diagram information management system according to this embodiment includes a captured image storage unit, a device mounting unit estimation unit, a device mounting state estimation unit, a device information reading unit, a captured image positional relationship estimation unit, and a panel diagram information generation unit. The captured image storage unit stores multiple captured images obtained by photographing a panel in multiple predefined states, in association with panel identification information that identifies the panel. The device mounting unit estimation unit estimates, for each of the multiple captured images, the position within the captured image of a device mounting unit image that is a captured image of at least one device mounting unit included in the captured image. The device mounting state estimation unit estimates, for each of the multiple captured images, whether a device is attached to at least one device mounting unit included in the captured image, the state of the device, and the device type of the device attached to the device mounting unit. The device information reading unit reads device information included in at least one device mounting unit image and / or device image based on the position within the image of at least one device image and the type of at least one device for each of the multiple captured images. The captured image positional relationship estimation unit estimates the positional relationship between the multiple captured images. The panel diagram information generation unit generates panel diagram information based on the position within the image of the device mounting unit image of at least one device mounting unit, whether or not a device is mounted in the at least one device mounting unit, the device type of the device mounted in the device mounting unit, the device information, and the positional relationship between the multiple captured images.

[0010] 1, the panel diagram information management system according to this embodiment includes a terminal device 3 carried by an operator who photographs a control panel, distribution panel, lighting panel, power panel, or other panel for which a panel diagram is to be created at the location where the panel is installed, a terminal device 5 carried by an administrator who manages the panel diagrams, and a panel diagram information management server 1. The terminal devices 3 and 5 and the panel diagram information management server 1 can communicate with each other via a wide area network NW1. The wide area network NW1 is, for example, the Internet.

[0011] The terminal device 3 is, for example, a smartphone, and as shown in FIG. 2 , includes a central processing unit (CPU) 301, a main memory 302, an auxiliary memory 303, a display 304, an input unit 305, a communication unit 306, an imaging unit 307, and a bus 309 connecting these components to one another. The main memory 302 has a volatile memory such as a random access memory (RAM) and is used as a work area for the CPU 301. The main memory 302 also has an image-only memory (not shown) that temporarily stores image information to be displayed on the display 304. The auxiliary memory 303 is a non-volatile memory such as a semiconductor flash memory and stores programs for the CPU 301 to execute various processes. The display 304 is a display device such as a liquid crystal display or an organic electroluminescence (EL) display. The input unit 305 is, for example, a transparent touchpad placed on top of the display 304. The communication unit 306 is connected to the local network NW3 and transmits information transferred from the CPU 301 to the board plan information management server 1 via the local network NW3 and the wide area network NW1, and transfers various information obtained from the board plan information management server 1 via the wide area network NW1 and the local network NW3 to the CPU 301.

[0012] The imaging unit 307 has a camera and a DMA (Direct Memory Access) controller, and after being started, the DMA controller transfers captured image information obtained by capturing an image with the camera to an image-dedicated memory in real time. Furthermore, when the imaging unit 307 is notified of imaging command information from the CPU 301, the imaging unit 307 notifies the CPU 301 of captured image information obtained by capturing an image at the timing when the imaging command information was notified.

[0013] The CPU 301 loads a program stored in the auxiliary storage unit 303 into the main storage unit 302 and executes it, thereby functioning as a display control unit 311, an image capture control unit 312, an image management unit 313, and an image information notification unit 314, as shown in FIG. 3. The auxiliary storage unit 303 shown in FIG. 2 also includes a captured image storage unit 331, as shown in FIG. 3. The captured image storage unit 331 stores captured image information indicating an image of at least a portion of a board captured by the image capture unit 307, location information indicating the location of the building in which the board is installed, installation position information indicating the installation position of the board within the building, and image type information indicating the type of the object being captured, in association with image identification information that identifies the captured image information. Here, the location information is information indicating, for example, the building name and address of the building to be captured, such as the name of the building, such as "XX Building," or an address, such as "XX City, XX Town, △-□." The installation position information is information indicating the floor number of the room in the building where the board is installed, the room number of the room, etc. The image type information is information that identifies the state of the board at the time of photographing (for example, whether the door is open or closed) and the part of the board that is to be photographed (for example, "the entire board," "a predetermined part of the board," etc.). The image identification information is information that identifies the captured image information, and includes, for example, the date and time of photographing.

[0014] When the user performs a board photographing preparation operation on the input unit 305 to photograph the board, the display control unit 311 accepts the board photographing preparation operation and, based on image type information (described later) notified from the image management unit 313, forms a photographing method notification image that notifies the user of the board photographing method, and displays the image on the display unit 304. Furthermore, when all photographed images necessary for generating board diagram information of the board to be photographed have been acquired and photographing completion notification information (described later) has been notified from the image management unit 313, the display control unit 311 forms an image transmission prompting image that prompts the user to transmit at least one piece of photographed image information obtained by photographing the board to the board diagram information management server 1, and displays the image on the display unit 304.

[0015] When a user performs an imaging operation on the input unit 305 to capture an image of the board to be photographed, the imaging control unit 312 accepts the imaging operation, generates imaging command information instructing the imaging unit 307 to capture an image of the board, and transfers the information to the imaging unit 307.

[0016] When the user inputs the location information and installation position information for the board to be photographed into the input unit 305 during the board photography preparation operation, the image management unit 313 stores the input location information and installation position information in the photographed image storage unit 331. After accepting the board photography preparation operation, the image management unit 313 sequentially notifies the display control unit 311 of at least one pre-set image type information. This image type information is determined based on the method of photographing the board. For example, different image type information is assigned to a photographed image of the entire board with the door closed, a photographed image of the entire board with the door open, and a photographed image of a portion of the board with the door open. Furthermore, after accepting the user's photographing operation, the image management unit 313 associates the transferred photographed image information obtained by photographing the board from the imaging unit 307 with the image identification information, the corresponding location information and installation position information, and the image type information most recently notified to the display control unit 311, and stores the transferred photographed image information in the photographed image storage unit 331. Thereafter, the image management unit 313 determines whether all of the captured image information necessary for generating the board diagram information of the board to be photographed has been acquired. Specifically, if the captured image storage unit 331 stores captured image information corresponding to all of at least one preset image type information, the image management unit 313 determines that all of the captured image information necessary for generating the board diagram information of the board to be photographed has been acquired, and notifies the display control unit 311 of a photographing completion notification. On the other hand, if the captured image storage unit 331 does not store captured image information corresponding to any of the at least one preset image type information, the image management unit 313 selects one of the image type information corresponding to the captured image information not stored in the captured image storage unit 331 and notifies the display control unit 311 of the selection.

[0017] When the user performs a captured image transmission operation on the input unit 305 to transmit captured image information to the panel plan information management server 1, the image information notification unit 314 accepts the captured image transmission operation and generates captured image notification information including the captured image information, location information, installation position information, and image type information stored in the captured image storage unit 331. Then, the image information notification unit 314 transmits the generated captured image notification information to the panel plan information management server 1.

[0018] The terminal device 5 is, for example, a personal computer and includes a display unit (not shown) and an input unit (not shown). The terminal device 5 also includes a board plan information storage unit (not shown) that stores board plan information acquired from the board plan information management server 1. Here, the display unit is a display device such as a liquid crystal display or an organic EL display. The input unit is an input device such as a keyboard. When an administrator performs a board plan information request operation via the input unit to request the board plan information management server 1 to send board plan information, the terminal device 5 accepts the board plan information request operation and generates board plan information request information. The terminal device 5 then transmits the generated board plan information request information to the board plan information management server 1, thereby acquiring board plan information notification information transmitted from the board plan information management server 1, extracting various information contained in the acquired board plan information notification information and storing it in the board plan information storage unit. The terminal device 5 then forms a board plan image based on the various information stored in the board plan information storage unit and displays it on the display unit.

[0019] Returning to FIG. 2, as shown in FIG. 2, the panel plan information management server 1 includes a CPU 101, a main memory unit 102, an auxiliary memory unit 103, a communication unit 106, and a bus 109 interconnecting these units. The CPU 101 is, for example, a multi-core processor. The main memory unit 102 has a volatile memory such as RAM and is used as a work area for the CPU 101. The auxiliary memory unit 103 is made up of a large-capacity non-volatile memory and stores programs for realizing various functions of the panel plan information management server 1. The communication unit 106 is connected to a wide area network NW1.

[0020] The CPU 101 reads out the programs stored in the auxiliary storage unit 103 into the main storage unit 102 and executes them, thereby functioning as an image information acquisition unit 111, an image area extraction unit 118, a device mounting unit position estimation unit 112, a device mounting state estimation unit 113, a device information reading unit 114, a captured image positional relationship estimation unit 115, a panel diagram information generation unit 116, and a panel diagram information notification unit 117, as shown in Fig. 3. The auxiliary storage unit 103 shown in Fig. 2 also includes a captured image storage unit 131, a local arrangement storage unit 132, a captured image arrangement storage unit 133, and a panel diagram information storage unit 134, as shown in Fig. 3. The captured image storage unit 131 stores captured image information, corresponding location information, installation position information, and image type information, in association with image identification information, as shown in Fig. 4, for example.

[0021] As shown in FIG. 5A, the local layout storage unit 132 stores the following information in association with image identification information: relative device mounting position information indicating the local coordinates of the device mounting position where the device is mounted in each captured image indicated by the captured image information; device status information indicating whether the device is mounted in the device mounting position and the status of the mounted device; device type information indicating the type of the mounted device; device image feature information indicating feature amounts of the image of the device portion; and device information indicating the device name, rated voltage, etc., written on a sticker or the like attached to the device. Examples of devices include switches such as main breakers and branch breakers mounted in the device mounting portion of the panel, meters, and programmable logic controllers. Examples of device information include the device model and the load name written on the nameplate of the device mounting portion. As shown in FIG. 5B, the captured image layout storage unit 133 stores image position information indicating the coordinates of a representative position of the captured portion of the entire panel corresponding to the captured image information in association with image identification information.

[0022] As shown in Figure 6, for example, the panel diagram information storage unit 134 stores absolute position information of the equipment mounting section indicating the global coordinates of the equipment mounting section on the entire panel, equipment status information indicating whether an equipment is mounted on the equipment mounting section and the status of the mounted equipment, equipment type information indicating the type of mounted equipment, equipment image feature information indicating the feature amounts of the image of the equipment part, and equipment information in association with image identification information.

[0023] 3 , upon acquiring captured image notification information transmitted from terminal device 3, image information acquisition unit 111 extracts captured image information, location information, installation position information, image type information, and image identification information included in the acquired captured image notification information, associates the extracted information with each other, and stores the information in captured image storage unit 131. Image region extraction unit 118 uses an image region extraction technique using a graph cut method to extract multiple characteristic image regions included in each captured image indicated by the captured image information stored in captured image storage unit 131. Then, image region extraction unit 118 notifies device mounting unit position estimation unit 112, device mounting state estimation unit 113, and device information reading unit 114 of image region information indicating each of the extracted image regions corresponding to each of the multiple captured images, and image region position information indicating the relative position coordinates of the image region within the entire captured image. In addition, the image area extraction unit 118 notifies the captured image positional relationship estimation unit 115 of the image area information and image area position information corresponding to each of the multiple pieces of captured image information, and the image type information stored in the captured image storage unit 131.

[0024] The device mounting part position estimation unit 112 estimates the intra-image position of the device mounting part image, which is a captured image of at least one device mounting part included in each captured image indicated by the plurality of pieces of captured image information stored in the captured image storage unit 131. Specifically, the device mounting part position estimation unit 112 estimates an image area corresponding to the device mounting part image from among the image areas indicated by each of the plurality of pieces of image area information notified by the image area extraction unit 118, using a device mounting part estimation model for estimating the image area corresponding to the device mounting part. Here, the device mounting part estimation model is represented by, for example, a convolutional neural network (CNN) and is a trained model for determining whether an image area is a device mounting part image. Then, based on the image area position information notified by the image area extraction unit 118, the device mounting part position estimation unit 112 identifies an intra-image position, which is the relative position coordinate of the device mounting part image in the entire captured image, and stores the device mounting part position information indicating the identified intra-image position in the local arrangement storage unit 132 in association with image identification information.

[0025] For each of a plurality of captured images, the device wearing state estimation unit 113 estimates whether a device is attached to at least one device wearing unit included in the captured image, the state of the device, and the type of the device attached to the device wearing unit. Specifically, the device wearing state estimation unit 113 references the device wearing unit position information stored in the local arrangement storage unit 132 and selects at least one image area adjacent to the device wearing unit image from among the image areas indicated by each of the plurality of image area information notified by the image area extraction unit 118. Next, the device wearing state estimation unit 113 uses a device estimation model for estimating the image area corresponding to the device attached to the device wearing unit and the state and type of the device to estimate an image area corresponding to the image of the device from among the at least one selected image area, and estimates the type and state of the device. Here, the device estimation model is represented, for example, by CNN and is a trained model for determining whether an image area is an image of a device and, if it is a device, the type and state of the device. This device estimation model is trained using training information including, for example, a combination of an image of a switch with its lever tilted to the on side and a training label indicating that the switch is on, and a combination of an image of a switch with its lever tilted to the off side and a training label indicating that the switch is off. This allows the device installation state estimation unit 113 to estimate the on / off state of a switch, even if, for example, no character indicating its on / off state is displayed on the switch. This device estimation model is also trained using training information including, for example, a combination of an image of a switch without a test button or the like and a training label indicating a circuit breaker (MCB), and a combination of an image of a switch with a test button or the like and a training label indicating an earth leakage breaker (ELB). This allows the device installation state estimation unit 113 to estimate the type of switch, for example, based on the presence or absence of an image of a test button or the like in the switch image. Then, the device attachment state estimation unit 113 stores device state information indicating whether the device is attached to the device attachment unit and the state of the attached device, and device type information indicating the type of device, in the local placement memory unit 132 in association with the device attachment unit position information of the corresponding device attachment unit.Furthermore, the device wearing state estimation unit 113 notifies the device information reading unit 114 of image area position information corresponding to the image area corresponding to the image of the device attached to the device attachment unit, which is notified by the image area extraction unit 118. Furthermore, the device wearing state estimation unit 113 estimates feature vectors for each image area corresponding to the estimated device image, for example, using a feature estimation model. Here, the device wearing state estimation unit 113 estimates feature vectors for each image area corresponding to the device attachment unit image or the image of the device attached to the device attachment unit, for example, using the feature estimation model. Here, the feature estimation model is represented by, for example, CNN, and is a trained model for determining feature vectors of image areas or captured images. Then, the device wearing state estimation unit 113 stores feature information indicating the estimated feature vectors in the local arrangement storage unit 132 in association with the corresponding device attachment unit position information.

[0026] The device information reading unit 114 reads device information included in at least one device image for each of the multiple captured images based on the in-image position and the type of at least one device. Specifically, the device information reading unit 114 references the device attachment unit position information stored in the local arrangement storage unit 132 and selects a device attachment unit image from the multiple image area information notified by the image area extraction unit 118. Furthermore, the device information reading unit 114 selects image area information corresponding to the image area position information notified by the device attachment state estimation unit 113 from the multiple image area information notified by the image area extraction unit 118. Next, the device information reading unit 114 reads the device information included in the selected image area using a character recognition model for recognizing characters included in the image area indicated by each of the selected image area information. Here, the character recognition model is represented by, for example, CNN and is a trained model for recognizing character strings indicating device information included in the image area. The device information reading unit 114 then stores the read device information in the local arrangement storage unit 132 in association with the device mounting unit position information of the corresponding device mounting unit. The device information reading unit 114 also selects image area information and image area position information of the captured image stored in the captured image storage unit 131 that corresponds to image type information indicating that the captured image is of the entire board with the door closed and is notified by the image area extraction unit 118, and identifies the image area information corresponding to the portion containing the board information based on the selected image area position information. The device information reading unit 114 then reads the board information included in the image area indicated by the identified image area information using the character recognition model. The device information reading unit 114 then stores the read board information in the board diagram information storage unit 134 in association with the corresponding location information and installation position information stored in the captured image storage unit 131.

[0027] The captured image positional relationship estimation unit 115 estimates the positional relationship between multiple captured images. Specifically, the captured image positional relationship estimation unit 115 selects image area information and image area position information of the captured images corresponding to image type information indicating that the captured image is an image of the entire board with the door open, as notified by the image area extraction unit 118, and estimates feature vectors of the image areas indicated by each of the selected image area information. The captured image positional relationship estimation unit 115 also estimates feature vectors of each captured image indicated by captured image information corresponding to image type information indicating that the captured image is an image of part of the board with the door open, as stored in the captured image storage unit 131. Here, the captured image positional relationship estimation unit 115 estimates the feature vectors of the image areas or the captured images using, for example, a feature estimation model. Here, the feature estimation model is represented, for example, by CNN, and is a trained model for determining the feature vectors of the image areas or the captured images. The captured image positional relationship estimation unit 115 then determines the positional relationship between the captured images indicated by the captured image information corresponding to the image type information indicating that the captured image is a portion of the board with the door open, based on the similarity, for example, Euclidean distance, between the feature vectors of each of the multiple image regions constituting the captured image of the entire board with the door open and the feature vector of the captured image of the portion of the board with the door open. Furthermore, the captured image positional relationship estimation unit 115 generates image position information indicating the coordinates of a representative position of the target portion of the entire board for each captured image of the portion of the board with the door open, based on the determined positional relationship, and stores this information in the captured image layout storage unit 133 in association with the image identification information of the captured image.

[0028] The panel diagram information generation unit 116 generates panel diagram information based on the position of at least one device mounting unit in the image of the device mounting unit, whether or not a device is mounted on at least one device mounting unit, the device status and device type of the device mounted on the device mounting unit, device information, and the positional relationship between multiple captured images. Specifically, the panel diagram information generation unit 116 calculates device mounting unit absolute position information indicating the global coordinates of the device mounting unit on the entire panel based on the device mounting unit position information and image identification information stored in the local layout storage unit 132 and the image position information and image identification information stored in the captured image layout storage unit 133. The panel diagram information generation unit 116 then associates the device status information, device type information, and device information stored in the local layout storage unit 132 with the location information and installation position information stored in the captured image storage unit 131 and the calculated device mounting unit absolute position information, and stores them in the panel diagram information storage unit 134. Here, the panel diagram information generation unit 116 selects two captured images in which the captured target portions are adjacent to each other, with reference to the image position information stored in the captured image layout storage unit 133. Then, the panel diagram information generation unit 116 refers to the feature amount information stored in the local layout storage unit 132, and identifies, if any, the device mounting unit images or device images included in the two selected captured images that have similar feature amount vectors indicated by the feature amount information, and stores only the combination of various information corresponding to either of the device mounting unit position information in the panel diagram information storage unit 134, assuming that the identified device mounting unit images or device images are images of the same device mounting unit or device.

[0029] When the panel plan information notification unit 117 acquires the panel plan information request information transmitted from the terminal device 5, it generates panel plan information notification information including various information stored in the panel plan information storage unit 134. Then, the panel plan information notification unit 117 transmits the generated panel plan information notification information to the terminal device 5 that is the sender of the panel plan information request information.

[0030] Next, the operation of the board diagram information management system according to this embodiment will be described with reference to Fig. 7. First, when a user carrying terminal device 3 performs the above-mentioned board photograph preparation operation on input unit 305 of terminal device 3, terminal device 3 accepts the board photograph preparation operation, forms the above-mentioned photographing method notification image, and displays it on display unit 304 (step S1). At this time, terminal device 3 stores in photographed image storage unit 331 the location information and installation position information of the board to be photographed that the user input in the above-mentioned board photograph preparation operation.

[0031] Next, when the user performs the aforementioned imaging operation using the input unit 305, the terminal device 3 accepts the imaging operation, images the board using the imaging unit 307, and stores the captured image information obtained by the imaging in the captured image storage unit 331 in association with the corresponding location information, installation position information, and image type information indicating the type of the captured image, along with image identification information that identifies the captured image information (step S2). Next, the terminal device 3 again forms a photography method notification image and displays it on the display unit 304 (step S3). Thereafter, when the user performs the aforementioned imaging operation using the input unit 305, the terminal device 3 accepts the imaging operation, images the board using the imaging unit 307, and stores the captured image information obtained by the imaging in association with the corresponding location information, installation position information, and image type information that indicates the type of the captured image, along with image identification information that identifies the captured image information, in the captured image storage unit 331 (step S4). Thereafter, the processes of steps S3 and S4 are repeated until all photographed image information required to generate board diagram information for the board to be photographed is acquired.

[0032] Here, for example, the terminal device 3 first displays on the display unit 304 a photographing method notification image notifying the user to photograph the entire panel with the door closed as shown in FIG. 8(A). When the user photographs the entire panel with the door closed, the terminal device 3 displays on the display unit 304 a photographing method notification image notifying the user to photograph the entire panel with the door open as shown in FIG. 8(B). When the user photographs the entire panel with the door open, the terminal device 3 displays on the display unit 304 a photographing method notification image notifying the user to photograph a portion of the panel near the main switch with the door open as shown in FIG. 9(A). When the user photographs the entire panel with the door open, the terminal device 3 displays on the display unit 304 a photographing method notification image notifying the user to photograph a portion of the panel near the branch switch with the door open as shown in FIG. 9(B). In this way, the terminal device 3 sequentially displays on the display unit 304 photographing method notification images notifying the user of the photographing method for acquiring photographed images required for creating panel diagram information.

[0033] 7, when the terminal device 3 determines that it has acquired the captured image information for all of the captured images required to generate the board plan information for the board to be photographed (step S5), it forms the aforementioned image transmission prompt image and displays it on the display unit 304 (step S6). Here, when the user performs the aforementioned captured image transmission operation on the input unit 305 of the terminal device 3, the terminal device 3 accepts the captured image transmission operation and generates captured image notification information including the captured image information, location information, installation position information, and image type information stored in the captured image storage unit 331 (step S7). The generated captured image notification information is then transmitted from the terminal device 3 to the board plan information management server 1 (step S8).

[0034] On the other hand, when the panel plan information management server 1 acquires the photographed image notification information, it extracts the photographed image information, location information, installation position information, image type information, and image identification information contained in the acquired photographed image notification information, and stores the extracted information in association with one another in the photographed image storage unit 131 (step S9). Next, the panel plan information management server 1 extracts a plurality of characteristic image areas contained in each photographed image for each photographed image indicated by the photographed image information stored in the photographed image storage unit 131 (step S10).

[0035] Next, the panel plan information management server 1 identifies an equipment mounting unit image of at least one equipment mounting unit from the extracted image area for each captured image indicated by the plurality of pieces of captured image information stored in the captured image storage unit 131, and estimates the intra-image position of the identified equipment mounting unit image within the captured image. Here, the panel plan information management server 1 estimates the intra-image positions of the equipment mounting unit images A41 and A43 in FIG. 9(B), for example. Then, as shown in FIG. 7, the panel plan information management server 1 stores device mounting unit position information indicating the estimated intra-image position in the local arrangement storage unit 132 in association with image identification information (step S11).

[0036] Then, for each of the multiple captured images, the panel plan information management server 1 estimates whether a device is attached to at least one device mounting section included in the captured image, the state of the device, and the device type of the device attached to the device mounting section. Here, the panel plan information management server 1 estimates the state and type of the device based on, for example, device images A42 and A44 in FIG. 9(B). Furthermore, if the device is a switch, the panel plan information management server 1 estimates the switch state to be "on" based on, for example, image A51 in FIG. 10(A), and the switch state to be "off" based on image A52 in FIG. 10(A). Alternatively, if the device is a switch, the panel plan information management server 1 estimates the switch state to be "on" based on, for example, image A61 in FIG. 10(B), and the switch state to be "off" based on image A62 in FIG. 10(B). Furthermore, if the device is a switch, the panel diagram information management server 1 estimates whether the switch is a circuit breaker (MCB) or an earth leakage breaker (ELB) based on the presence or absence of a button image portion BU63 in the captured image of FIG. 11. Then, as shown in FIG. 7, the panel diagram information management server 1 stores device status information indicating whether the device is mounted in a device mounting section and the state of the mounted device, and device type information indicating the type of device, in the local arrangement storage unit 132 in association with the device mounting section position information of the corresponding device mounting section (step S12). The panel diagram information management server 1 also estimates feature vectors for each image region corresponding to the image of the device mounted in the device mounting section. Then, the panel diagram information management server 1 stores feature information indicating the estimated feature vectors in association with the corresponding device mounting section position information in the local arrangement storage unit 132 (step S13).

[0037] Next, the panel plan information management server 1 reads device information included in at least one device image for each of the multiple captured images based on the in-image position and the type of at least one device. Here, the panel plan information management server 1 reads device information included in each of the image areas A41, A42, A43, and A44 included in the captured image of FIG. 9B, for example. Then, as shown in FIG. 7, the panel plan information management server 1 stores the read device information in the local arrangement storage unit 132 in association with the device mounting unit position information of the corresponding device mounting unit (step S14). The panel plan information management server 1 also selects image area information and image area position information for the captured image of the entire panel with the door closed, stored in the captured image storage unit 131, and identifies the image area information corresponding to the portion containing the panel information based on the selected image area position information. The panel plan information management server 1 then reads the panel information included in the image area indicated by the identified image area information. Here, the board plan information management server 1 reads the board information included in the image area A11 included in the photographed image of Fig. 8(A), for example. Then, as shown in Fig. 7, the board plan information management server 1 stores the read board information in the board plan information storage unit 134 in association with the corresponding location information and installation position information stored in the photographed image storage unit 131 (step S15).

[0038] Next, the panel plan information management server 1 estimates the positional relationship between the multiple captured images. Here, the panel plan information management server 1 estimates the feature vectors of each of the multiple image areas, including the image area A21 shown in FIG. 8(B), for example, and also estimates the feature vector of the captured image shown in FIG. 9(A). The panel plan information management server 1 then determines that the feature vectors of both are similar to each other and estimates the positional relationship of the captured portion of the captured image shown in FIG. 9(A) within the entire panel. Then, based on the identified positional relationship, the panel plan information management server 1 generates image position information indicating the coordinates of a representative position of the captured portion within the entire panel for each captured image of a portion of the panel with the door open, and stores the image position information in the captured image layout storage unit 133 in association with the image identification information of the captured image (step S16).

[0039] The panel plan information management server 1 then generates panel plan information based on the position of the device mounting unit image of at least one device mounting unit within the image, the presence or absence of a device mounted in at least one device mounting unit, the device status and device type of the device mounted in the device mounting unit, the device information, and the positional relationship between the multiple captured images.The panel plan information management server 1 then associates the device status information, device type information, and device information stored in the local arrangement storage unit 132, the location information and installation position information stored in the captured image storage unit 131, and the calculated device mounting unit absolute position information, and stores them in the panel plan information storage unit 134 (step S17).

[0040] Next, when the administrator performs the aforementioned operation to request board plan information on the input unit of the terminal device 5, the terminal device 5 accepts the operation to request board plan information and generates board plan information request information (step S18). The generated board plan information request information is then transmitted from the terminal device 5 to the board plan information management server 1 (step S19). Meanwhile, upon acquiring the board plan information request information, the board plan information management server 1 generates board plan information notification information including various information stored in the board plan information storage unit 134 in response to the request (step S20), as shown in FIG. 12. The board plan information notification unit 117 then transmits the generated board plan information notification information to the terminal device 5 that sent the board plan information request information. Meanwhile, upon acquiring the board plan information notification information, the terminal device 5 extracts the board plan information included in the acquired board plan information notification information and stores it in the board plan information storage unit (step S21). Next, the terminal device 5 forms a board plan image based on the board plan information stored in the board plan information storage unit and displays it on the display unit (step S22).

[0041] Next, the captured image information acquisition process executed by the terminal device 3 according to this embodiment will be described with reference to Fig. 13. This captured image information acquisition process is initiated when a program for executing the captured image information acquisition process is started in the terminal device 3 after the power to the terminal device 3 is turned on, and then the user performs the above-mentioned board photography preparation operation using the input unit 305. Here, when the image management unit 313 inputs the above-mentioned location information and installation position information for the board to be photographed using the input unit 305 during the board photography preparation operation, the image management unit 313 stores the input location information and installation position information in the captured image storage unit 331.

[0042] First, the display control unit 311 accepts the above-mentioned board photography preparation operation, and forms a photography method notification image based on the above-mentioned image type information notified from the image management unit 313, and displays it on the display unit 304 (step S201). Next, the imaging control unit 312 determines whether or not the above-mentioned imaging operation performed by the user on the input unit 305 has been accepted (step S202). Here, if the imaging control unit 312 determines that the imaging operation has not been accepted (step S202: No), the processing of step S201 is executed again.

[0043] On the other hand, when the imaging control unit 312 determines that it has received an imaging operation (step S202: Yes), it generates imaging command information instructing the imaging unit 307 to capture an image of the board and transfers it to the imaging unit 307. The imaging unit 307 then captures an image of the board at the timing when it is notified of the imaging command information, and transfers the captured image information obtained by capturing the image to the image management unit 313. The image management unit 313 then stores the transferred captured image information together with the image identification information in the captured image storage unit 331 in association with the corresponding location information and installation position information, and the image type information most recently notified to the display control unit 311 (step S203).

[0044] Next, the image management unit 313 determines whether all of the captured image information necessary for generating the board diagram information of the board to be photographed has been acquired (step S203). If the image management unit 313 determines that there is necessary captured image information that has not been acquired (step S203: NO), it selects one of the image type information corresponding to the captured image information that has not yet been acquired and notifies the display control unit 311. Then, the processing of step S201 is executed again. On the other hand, if the image management unit 313 determines that all of the captured image information necessary for generating the board diagram information of the board to be photographed has been acquired (step S203: Yes), it notifies the display control unit 311 of the photographing completion notification information. Then, the display control unit 311 forms the above-mentioned image transmission prompt image and displays it on the display unit 304 (step S204).

[0045] Next, the image information notification unit 314 determines whether or not the user's operation to send the captured image via the input unit 305 has been accepted (step S205). If the image information notification unit 314 determines that the user has not accepted the captured image sending operation within a preset waiting time after the image sending prompt image was displayed on the display unit 304 (step S205: No), the captured image information acquisition process ends. On the other hand, if the image information notification unit 314 determines that the user has accepted the captured image sending operation (step S205: Yes), the image information notification unit 314 generates captured image notification information including the captured image information, location information, installation position information, and image type information stored in the captured image storage unit 331. Then, the image information notification unit 314 transmits the generated captured image notification information to the panel plan information management server 1 (step S206), and the captured image information acquisition process ends.

[0046] Next, the board plan information management process executed by the board plan information management server 1 according to this embodiment will be described with reference to FIG. 14. The board plan information management process is initiated when a program for executing the board plan information management process is launched in the board plan information management server 1. First, the image information acquisition unit 111 determines whether or not captured image notification information transmitted from the terminal device 3 has been acquired (step S101). If the image information acquisition unit 111 determines that captured image notification information has not been acquired (step S101: No), the process of step S111, which will be described later, is executed. On the other hand, if the image information acquisition unit 111 determines that captured image notification information has been acquired (step S101: Yes), the image information acquisition unit 111 extracts captured image information, location information, installation position information, image type information, and image identification information included in the acquired captured image notification information, associates the extracted information with each other, and stores the extracted information in the captured image storage unit 131 (step S102). Next, image region extraction unit 118 extracts a plurality of characteristic image regions included in each captured image indicated by the captured image information stored in captured image storage unit 131 (step S103). Here, image region extraction unit 118 notifies device mounting unit position estimation unit 112, device mounting state estimation unit 113, and device information reading unit 114 of image region information indicating each of the extracted image regions corresponding to each of the plurality of captured images, and image region position information indicating the relative position coordinates of the image region within the entire captured image. Furthermore, image region extraction unit 118 notifies captured image positional relationship estimation unit 115 of the image region information and image region position information corresponding to each of the plurality of captured image information, and image type information stored in captured image storage unit 131.

[0047] Next, the device mounting part position estimation unit 112 estimates the intra-image position within the captured image of the device mounting part image, which is a captured image of at least one device mounting part included in each captured image indicated by the plurality of captured image information stored in the captured image storage unit 131. Then, based on the image area position information notified from the image area extraction unit 118, the device mounting part position estimation unit 112 specifies the intra-image position, which is the relative position coordinate of the device mounting part image within the entire captured image, and stores the device mounting part position information indicating the specified intra-image position in the local arrangement storage unit 132 in association with the image identification information (step S104).

[0048] Thereafter, the device wearing state estimation unit 113 estimates, for each of the multiple captured images, whether a device is attached to at least one device mounting unit included in the captured image, the state of the device, and the device type of the device attached to the device mounting unit.The device wearing state estimation unit 113 then associates device state information indicating whether the device is attached to the device mounting unit and the state of the attached device, and device type information indicating the type of device, in the local arrangement storage unit 132 (step S105).At this time, the device wearing state estimation unit 113 notifies the device information reading unit 114 of image area position information corresponding to the image area corresponding to the image of the device attached to the device mounting unit, which has been notified by the image area extraction unit 118.

[0049] Next, the device wearing state estimation unit 113 estimates feature vectors for each image region corresponding to the estimated device image. Then, the device wearing state estimation unit 113 stores feature information indicating the estimated feature vectors in association with the corresponding device wearing part position information in the local arrangement storage unit 132 (step S106).

[0050] Next, for each of the plurality of captured images, the device information reading unit 114 reads device information included in at least one device image based on the position within the image and the type of at least one device. Then, the device information reading unit 114 stores the read device information in the local arrangement storage unit 132 in association with device mounting unit position information of the corresponding device mounting unit (step S107).

[0051] The device information reading unit 114 then selects image area information and image area position information of the captured image stored in the captured image storage unit 131 that correspond to image type information indicating that the captured image is of the entire board with the door closed and is notified by the image area extraction unit 118, and identifies the image area information corresponding to the portion where the board information is written based on the selected image area position information.The device information reading unit 114 then uses the character recognition model described above to read the board information included in the image area indicated by the identified image area information.The device information reading unit 114 then stores the read board information in the board diagram information storage unit 134 in association with the corresponding location information and installation position information stored in the captured image storage unit 131 (step S108).

[0052] Next, the photographed image positional relationship estimation unit 115 estimates the positional relationship between the multiple photographed images. Then, based on the estimated positional relationship, the photographed image positional relationship estimation unit 115 generates image position information indicating the coordinates of a representative position of a photographed portion of the entire board in each photographed image of a part of the board, and stores the image position information in the photographed image layout storage unit 133 in association with the image identification information of the photographed image (step S109).

[0053] Next, the panel diagram information generation unit 116 generates panel diagram information based on the position of the device mounting unit image of at least one device mounting unit within the image, whether or not a device is mounted on at least one device mounting unit, the device status and device type of the device mounted on the device mounting unit, the device information, and the positional relationship between the multiple captured images.The panel diagram information generation unit 116 then associates the device status information, device type information, and device information stored in the local arrangement storage unit 132, the location information and installation position information stored in the captured image storage unit 131, and the calculated device mounting unit absolute position information, and stores them in the panel diagram information storage unit 134 (step S110).

[0054] Thereafter, the panel plan information notification unit 117 determines whether or not it has acquired panel plan information request information transmitted from the terminal device 5 (step S111). If the panel plan information notification unit 117 determines that it has not acquired the panel plan information request information (step S111: No), the processing of step S101 is executed again. On the other hand, if the panel plan information notification unit 117 determines that it has acquired the panel plan information request information (step S111: Yes), it generates panel plan information notification information including various information stored in the panel plan information storage unit 134. Then, the panel plan information notification unit 117 transmits the generated panel plan information notification information to the terminal device 5 that sent the panel plan information request information (step S112). Next, the processing of step S101 is executed again.

[0055] As described above, in the panel diagram information management system according to the present embodiment, the device mounting state estimation unit estimates, for each of a plurality of captured images, whether a device is mounted in at least one device mounting section included in the captured image and the device type of the device mounted in the device mounting section. This makes it possible to generate panel diagram information that includes information on whether a device is mounted in a device mounting section, thereby generating panel diagram information appropriate for managing devices mounted in control panels, distribution boards, lighting panels, power panels, etc.

[0056] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. For example, at least one of the device mounting part estimation model, device estimation model, character recognition model, and feature estimation model used in the panel diagram information management server 1 may be updated based on training data periodically or in response to a request from a user or administrator.

[0057] In an embodiment, when the image management unit 313 of the terminal device 3 accepts the aforementioned board photography preparation operation, it may generate directory information for grouping and managing all captured image information necessary for generating board diagram information for the board to be photographed. Here, when the image management unit 313 accepts the board photography preparation operation and then receives the first captured image information from the imaging unit 307, the image management unit 313 generates directory information indicating that the transferred captured image information is associated with group identification information that identifies a newly generated group, and stores the directory information in the captured image storage unit 331. Here, the group identification information may represent, for example, the date on which the photographing preparation operation was accepted. Then, after accepting the board photography preparation operation, until it is determined that all captured image information necessary for generating board diagram information for the board to be photographed has been acquired, the image management unit 313 updates the directory information each time captured image information is transferred from the imaging unit 307 so that the transferred captured image information is associated with the same group identification information. When the user performs the aforementioned photographed image transmission operation on the input unit 305 of the terminal device 3, the terminal device 3 accepts the photographed image transmission operation and generates photographed image notification information including directory information along with the photographed image information, etc., stored in the photographed image storage unit 331. The generated photographed image notification information is then transmitted from the terminal device 3 to the board plan information management server 1. Meanwhile, upon acquiring the photographed image notification information, the board plan information management server 1 extracts directory information along with the photographed image information, etc., contained in the acquired photographed image notification information, and stores the extracted information in the photographed image storage unit 131 in association with each other. Thereafter, when the administrator operates the input unit of the terminal device 5 to display the photographed image information stored in the photographed image storage unit 131 of the board plan information management server 1 on the display unit, the terminal device 5 first references the directory information and displays a folder image assigned with the name indicated by the aforementioned group identification information on the display unit. When the administrator operates the input unit to select a folder image, a file image indicating the photographed image information is displayed.

[0058] In addition, after receiving the operation to prepare for photographing the board, the image management unit 313 may recognize the name of the board from the image of the part of the photographed image indicated by the photographed image information that is initially transferred from the imaging unit 307, and generate group identification information based on the recognized name of the board.

[0059] In an embodiment, the panel plan information management server 1 may include an equipment number determination unit that determines the number of equipment included in each of the plurality of photographed images based on the equipment type estimated by the equipment installation state estimation unit 113 for each of the plurality of photographed images, and an equipment use determination unit that determines the use of the equipment included in each of the plurality of photographed images based on the number of identified equipment. Specifically, the panel plan information management server 1 may include a switch number determination unit that determines the number of images of switches included in each of the photographed images based on the equipment type of the equipment attached to the equipment installation unit estimated by the equipment installation state estimation unit 113 for each of the plurality of photographed images. The panel plan information management server 1 may also include a switch image type determination unit that determines whether each of the plurality of photographed images is an image of a main switch inserted in a main line or a branch switch inserted in a branch line based on the number of identified switches. Here, the switch image type determination unit may, for example, determine the image as a main switch if the number of images of the switch included in the photographed image is one, and determine the image as a branch switch if the number of images of the switch included in the photographed image is six or more. The circuit breaker image type discrimination unit may generate directory information for storing each piece of photographed image identification information that identifies the multiple pieces of photographed image information stored in the photographed image storage unit 131 in association with circuit breaker type information indicating the type of the identified circuit breaker, i.e., whether it is a main circuit breaker or a branch circuit breaker, and store the directory information in the photographed image storage unit 131.

[0060] In an embodiment, the board plan information management server 1 may further include a captured image division unit (not shown) that generates, from captured image information indicating a single captured image including images of multiple devices, captured image information indicating each of multiple captured images obtained by dividing the captured image. For example, for a captured image of an entire board as shown in FIG. 15(A), the captured image division unit generates captured image information indicating captured images corresponding to three image areas A71, A72, and A73 (see FIGS. 15(B) to 15(D)). Here, the captured image division unit may generate captured image information based on the positions of the device mounting units included in the captured image estimated by the device mounting unit position estimation unit 112 so that the number of device mounting units included in each of the multiple captured images generated by division is 24 or less (2 columns × 12 or less).

[0061] In an embodiment, the terminal device 3 may further include a device mounting part image identification unit (not shown) that identifies at least one device mounting part image corresponding to the device mounting part included in a captured image obtained by taking a photograph of the panel, and a captured image determination unit that determines whether the captured image is suitable for generating panel diagram information based on the pixel count of each of the identified at least one device mounting part image. Here, the captured image determination unit determines that the captured image is suitable for generating panel diagram information if the pixel count of the identified device mounting part image is equal to or greater than a predetermined pixel count. For example, the captured image is determined to be suitable for generating panel diagram information if the pixel count of the device mounting part image is equal to or greater than a predetermined pixel count threshold. Here, the pixel count threshold is set to, for example, 270,000 pixels. Note that, when multiple device mounting part images are identified, the captured image determination unit may determine whether the average, median, maximum, or minimum pixel count of each of the multiple device mounting part images is equal to or greater than the pixel count threshold. In addition, when the captured image determination unit determines that the captured image is not suitable for creating board diagram information, the display control unit 311 of the terminal device 3 may form a message image urging the user to take a photo of the board again and display it on the display unit 304.

[0062] In an embodiment, the panel diagram information management server may identify the correspondence between the local coordinates of the equipment mounting section position where the equipment is mounted in each of the photographed images indicated by at least one photographed image information and the global coordinates of the equipment mounting section on the entire panel, and generate photographed image information notification information including mounting section position correspondence information indicating the identified correspondence and the corresponding photographed image information, and send it to a terminal device held by the administrator.

[0063] A panel plan information management system according to this modified example includes a panel plan information management server 2001 as shown in Fig. 16, and a terminal device 2005 owned by an administrator. In Fig. 16, components similar to those in the embodiment are denoted by the same reference numerals as those in Fig. 3. The panel plan information management server 2001 shown in Fig. 16 includes functional units similar to those of the panel plan information management server 1 described in the embodiment, but some of these are omitted in Fig. 16. The hardware configurations of the panel plan information management server 2001 and the terminal device 5 are similar to those of the panel plan information management server 1 and the terminal device 5 described in the embodiment. The panel plan information management server 2001 includes an attachment unit position correspondence relationship identification unit 2119, a captured image notification unit 2120, and a panel plan information update unit 2121, in addition to the functional units described in the embodiment. When the image identification information corresponding to at least one piece of photographed image information to be transmitted from the photographed image notification unit 2120 to the terminal device 2005 is notified, the mounting unit position correspondence determination unit 2119 determines the correspondence between the local coordinates of the device mounting unit positions where the devices are mounted in each of the photographed images indicated by the photographed image information corresponding to each of the determined image identification information and the global coordinates of the device mounting unit on the entire board, based on the device mounting unit relative position information corresponding to each of the determined image identification information stored in the local arrangement storage unit 132 and the image position information corresponding to each of the determined image identification information stored in the photographed image arrangement storage unit 133. Then, the mounting unit position correspondence determination unit 2119 notifies the photographed image notification unit 2120 of the determined correspondence for each of the at least one piece of image identification information, i.e., mounting unit position correspondence information indicating a combination of the local coordinates and the global coordinates.

[0064] When the captured image notification unit 2120 acquires captured image request information (described later) transmitted from the terminal device 2005, it selects captured image information identified by at least one piece of image identification information included in the captured image request information from the captured image information stored in the captured image storage unit 131, and notifies the at least one piece of image identification information to the attachment unit position correspondence identification unit 2119. Then, when the captured image notification unit 2120 is notified of the above-mentioned attachment unit position correspondence information from the attachment unit position correspondence identification unit 2119, it generates captured image information notification information including the selected at least one piece of captured image information and the image identification information thereof, as well as the notified attachment unit position correspondence information, and transmits the generated information to the terminal device 2005.

[0065] When the panel plan information update unit 2120 acquires panel plan information update request information including updated panel plan information (described below) transmitted from the terminal device 2005, it extracts the panel plan information included in the acquired panel plan information update request information. Then, the panel plan information update unit 2120 uses the extracted panel plan information to update the panel plan information stored in the panel plan information storage unit 134.

[0066] The terminal device 2005 includes the display unit 504 and input unit 505 described in the embodiment. The terminal device 2005 also includes a CPU (not shown), a main memory unit (not shown), and an auxiliary memory unit (not shown). The CPU loads a program stored in the auxiliary memory unit into the main memory and executes it, thereby functioning as a panel diagram information acquisition unit 511, a captured image acquisition unit 2512, a composite image generation unit 2513, a display control unit 514, a panel diagram information update unit 2515, and a panel diagram information update request unit 2516. The auxiliary memory unit also includes a panel diagram information storage unit 531 and a captured image storage unit 2532. The panel diagram information storage unit 531 stores device mounting unit absolute position information indicating the global coordinates of the device mounting unit within the entire panel, device status information indicating whether a device is mounted in the device mounting unit and the status of the mounted device, device type information indicating the type of the mounted device, device image feature amount information indicating feature amounts of an image of the device part, and device information, in association with image identification information. The captured image storage unit 2532 stores at least one piece of captured image information in association with image identification information that identifies the captured image information and the above-mentioned attachment part position correspondence information that corresponds to the captured image indicated by the captured image information.

[0067] As explained in the embodiment, when the administrator performs the above-mentioned panel plan information request operation via the input unit 505, the panel plan information acquisition unit 511 accepts the panel plan information request operation, generates panel plan information request information, and transmits it to the panel plan information management server 1. As a result, the panel plan information acquisition unit 511 acquires panel plan information notification information transmitted from the panel plan information management server 1, extracts the panel plan information included in the acquired panel plan information notification information, and stores it in the panel plan information storage unit 531.

[0068] When the administrator performs a captured image request operation via the input unit 505, which includes at least one piece of image identification information and requests the panel plan information management server 2001 to transmit captured image information, the captured image acquisition unit 2512 accepts the captured image request operation, generates captured image request information, and transmits it to the panel plan information management server 1. As a result, the captured image acquisition unit 2512 acquires captured image information notification information transmitted from the panel plan information management server 1, extracts the captured image information, image identification information, and mounting unit position correspondence information contained in the acquired captured image information notification information, and stores the extracted information in the captured image storage unit 2532 in association with each other.

[0069] When the composite image generation unit 2513 receives a photographed image selection operation performed by the administrator via the input unit 505 to select photographed image information, it selects one piece of photographed image information selected by the photographed image selection operation and its corresponding mounting unit position correspondence information from the photographed image information stored in the photographed image storage unit 2532. Then, the composite image generation unit 2513 refers to the device mounting unit absolute position information stored in the panel diagram information storage unit 531 and the mounting unit position correspondence information stored in the photographed image storage unit 2532, and identifies the local coordinates of the device mounting unit in the photographed image indicated by the selected photographed image information. Furthermore, the composite image generation unit 2513 refers to the device status information stored in the panel diagram information storage unit 531, and identifies the status of the device mounted in the device mounting unit in the photographed image indicated by the selected photographed image information. The composite image generation unit 2513 then generates icon images representing the status of each device included in the captured image indicated by the selected captured image information based on the identified local coordinates and the device status, and generates a composite image by combining the generated icon images with the captured image indicated by the selected captured image information. The display control unit 514 causes the display unit 504 to display the composite image generated by the composite image generation unit 2513.

[0070] When the administrator performs a correction operation to correct the information stored in the panel plan information storage unit 531 via the input unit 505 while checking the composite image displayed on the display unit 504, the panel plan information update unit 2515 accepts the correction operation and updates the information stored in the panel plan information storage unit 531 using the corrected information specified by the administrator in the correction operation.

[0071] When the administrator performs a panel plan information update request operation via the input unit 505 to request that the information stored in the panel plan information storage unit 134 of the panel plan information management server 2001 be updated using information stored in the panel plan information storage unit 531, the panel plan information update request unit 2516 accepts the panel plan information update request operation and generates panel plan information update request information including the updated information stored in the panel plan information storage unit 531. The panel plan information update request unit 2516 then transmits the generated panel plan information update request information to the panel plan information management server 2001.

[0072] The terminal device 5 displays on the display unit 504 a composite image GA81 in which icon images BU81 and BU82 indicating the on / off state of each of the multiple switches are combined with a photographed image of a panel having multiple switches, as shown in Fig. 17, for example. Then, when the terminal device 5 is displaying the composite image GA81 on the display unit and the user performs the above-mentioned correction operation by switching the on / off state represented by the icon images BU81 and BU82 on the input unit of the terminal device 5, the device status information for each of the multiple switches stored in the panel diagram information storage unit 531, corresponding to the icon images BU81 and BU82 whose on / off state has been switched, is updated. That is, if the switch is in the on state, it is updated to the off state, and if it is in the off state, it is updated to the on state.

[0073] According to this configuration, the administrator can modify the various information stored in the panel diagram information storage unit 134, so that the various information stored in the panel diagram information storage unit 134 can be maintained as appropriate information.

[0074] In the embodiment, part of the functions of the board plan information management server 1 may be realized in the terminal device 3 or the terminal device 5.

[0075] Furthermore, the various functions of the panel plan information management server 1 and terminal devices 3, 5 according to the present invention can be realized using a normal computer system, rather than a dedicated system. For example, a program for executing the above operations may be stored on a non-transitory recording medium (such as a CD-ROM (Compact Disc Read Only Memory)) that can be read by a computer system and distributed to a computer connected to a network, and the panel plan information management server 1 and terminal device 3 that execute the above-described processes may be configured by installing the program in the computer system.

[0076] The method of providing the program to the computer is arbitrary. For example, the program may be uploaded to a bulletin board system (BBS) on a communication line and distributed to the computer via the communication line. The computer then launches the program and executes it under the control of an operating system (OS) in the same way as other applications. This allows the computer to function as the board plan information management server 1 and terminal devices 3 and 5 that execute the above-mentioned processes.

[0077] Although the embodiments and modifications of the present invention have been described above, the present invention is not limited to these. The present invention includes any combination of the embodiments and modifications, and any combination to which appropriate modifications have been made. [Industrial Applicability]

[0078] The present invention is suitable as a system for creating diagrams of switchboards, control panels, etc. [Explanation of symbols]

[0079] 1: Panel diagram information management server, 3, 5: Terminal device, 101, 301: CPU, 102, 302: Main memory unit, 103, 303: Auxiliary memory unit, 106, 306: Communication unit, 109, 309: Bus, 111: Image information acquisition unit, 112: Device mounting unit position estimation unit, 113: Device mounting state estimation unit, 114: Device information reading unit, 115: Captured image positional relationship estimation unit, 116: Panel information generation unit, 117: Panel information notification unit, 131, 331: Captured image storage unit, 132: Local arrangement storage unit, 133: Overall Layout storage unit, 134: panel diagram information storage unit, 304, 504: display unit, 305, 505: input unit, 307: imaging unit, 311, 514: display control unit, 312: imaging control unit, 313: image management unit, 314: image information notification unit, 511: panel diagram information acquisition unit, 2119: attachment unit position correspondence relationship identification unit, 2120: captured image notification unit, 2121, 2515: panel diagram information update unit, 2512: captured image acquisition unit, 2513: composite image generation unit, 2516: panel diagram information update request unit, NW1: wide area network

Claims

1. a photographed image storage unit that stores a plurality of photographed images obtained by photographing a board in a plurality of preset states, in association with board identification information that identifies the board; a device mounting part position estimation unit that estimates, for each of the plurality of captured images, a position within the captured image of a device mounting part image that is a captured image of at least one device mounting part included in the captured image; a device wearing state estimation unit that estimates, for each of the plurality of captured images, whether the device is attached to at least one of the device mounting sections included in the captured image, a state of the device, and a device type of the device attached to the device mounting section; a device information reading unit that reads device information included in at least one of the device mounting image and the device image based on a position within the image of the device image of at least one of the devices and a type of at least one of the devices for each of the plurality of captured images; a captured image positional relationship estimation unit that estimates a positional relationship between the plurality of captured images; a panel diagram information generating unit that generates panel diagram information based on the position within the image of the device mounting unit image of at least one of the device mounting units, whether the device is mounted on at least one of the device mounting units and the device type of the device mounted on the device mounting unit, the device information, and the positional relationship between the multiple captured images; Board map information management system.

2. A display unit; An imaging unit; a display control unit that forms a photography method notification image that notifies a user of a photography method for photographing the board and displays the image on the display unit; and an image management unit that, when the image capturing unit completes capturing an image after the image capturing method notification image is displayed on the display unit, stores captured image information indicating the captured image obtained by the capturing in the captured image storage unit in association with image type information determined based on the image capturing method. The panel diagram information management system according to claim 1 .

3. the captured image positional relationship estimation unit extracts a characteristic image region included in each of the plurality of captured images, and estimates a positional relationship between the plurality of captured images based on a similarity between feature vectors of the image region in each of the plurality of captured images; The panel diagram information management system according to claim 1 or 2.

4. the image management unit generates directory information for grouping and managing all of the photographed image information required to generate the board diagram information for one of the boards; The panel diagram information management system according to claim 2.

5. an equipment number determination unit that determines the number of the equipment included in each of the plurality of captured images based on the equipment types estimated for each of the plurality of captured images by the equipment wearing state estimation unit; and a device use determination unit that determines the use of the device that was the subject of each of the plurality of captured images based on the number of the identified devices. The panel diagram information management system according to claim 1 or 2.

6. a captured image division unit that generates, from captured image information indicating one captured image including images of a plurality of the devices, captured image information indicating each of a plurality of captured images obtained by dividing the captured image; The panel diagram information management system according to claim 1 or 2.

7. an equipment mounting part image specifying unit that specifies at least one of the equipment mounting part images included in the captured image; and a captured image determination unit that determines whether the captured image is suitable for generating the panel diagram information based on the number of pixels of each of the identified at least one device mounting portion image. The panel diagram information management system according to claim 1 or 2.

8. a mounting unit position correspondence specifying unit that specifies a correspondence between local coordinates of the device mounting unit position in each of the photographed images indicated by at least one photographed image information and global coordinates of the device mounting unit on the entire panel; and a composite image generation unit that specifies, based on the correspondence relationship, a state of the device attached to the device attachment unit in the captured image indicated by the at least one piece of captured image information, generates icon images representing the state of each device included in the captured image indicated by the at least one piece of captured image information based on the specified states of the devices, and generates a composite image by combining the generated icon images with the captured image. The panel diagram information management system according to claim 1 or 2.

9. a photographed image storage unit that stores a plurality of photographed images obtained by photographing a board in a plurality of preset states, in association with board identification information that identifies the board; a device mounting part position estimation unit that estimates, for each of the plurality of captured images, a position within the captured image of a device mounting part image that is a captured image of at least one device mounting part included in the captured image; a device wearing state estimation unit that estimates, for each of the plurality of captured images, whether the device is attached to at least one of the device mounting sections included in the captured image, a state of the device, and a device type of the device attached to the device mounting section; a device information reading unit that reads device information included in at least one of the device mounting image and the device image based on a position within the image of the device image of at least one of the devices and a type of at least one of the devices for each of the plurality of captured images; a captured image positional relationship estimation unit that estimates a positional relationship between the plurality of captured images; a panel diagram information generating unit that generates panel diagram information based on the position within the image of the device mounting unit image of at least one of the device mounting units, whether the device is mounted on at least one of the device mounting units and the device type of the device mounted on the device mounting unit, the device information, and the positional relationship between the multiple captured images; Board diagram information management device.

10. a step of storing a plurality of photographed images obtained by photographing the board in a plurality of preset states in a photographed image storage unit in association with board identification information for identifying the board; a step of estimating, for each of the plurality of captured images, a position within the captured image of an equipment mounting part image that is a captured image of at least one equipment mounting part included in the captured image; a step of estimating, for each of the plurality of captured images, whether the device is attached to at least one of the device mounting sections included in the captured images, a state of the device, and a device type of the device attached to the device mounting section; reading, for each of the plurality of captured images, device information included in at least one of the device mounting section image and the device image based on a position within the image of the device image of at least one of the devices and a type of at least one of the devices; estimating a positional relationship between the plurality of captured images; generating panel diagram information based on the position within the image of the device mounting section image of at least one of the device mounting sections, whether the device is mounted on at least one of the device mounting sections and the device type of the device mounted on the device mounting section, the device information, and the positional relationship between the plurality of captured images; Board diagram information management method.

11. Computer, a photographed image storage unit that stores a plurality of photographed images obtained by photographing the board in a plurality of preset states, in association with board identification information that identifies the board; a device mounting part position estimation unit that estimates, for each of the plurality of captured images, a position within the captured image of a device mounting part image that is a captured image of at least one device mounting part included in the captured image; a device wearing state estimation unit that estimates, for each of the plurality of captured images, whether the device is attached to at least one of the device mounting sections included in the captured image, a state of the device, and a device type of the device attached to the device mounting section; a device information reading unit that reads device information included in at least one of the device mounting portion image and the device image based on a position of the device image of at least one of the devices within the image and a type of the at least one device for each of the plurality of captured images; a captured image positional relationship estimation unit that estimates a positional relationship between the plurality of captured images; a panel diagram information generating unit that generates panel diagram information based on the position of the device mounting unit image of at least one of the device mounting units within the image, whether the device is mounted on at least one of the device mounting units and the device type of the device mounted on the device mounting unit, the device information, and the positional relationship between the multiple captured images; A program to function as a

Images