Information processing system and program

Abstract

To provide information for supporting mounting of components on a device by referring to components after having been removed from the device.SOLUTION: An operator photographs components 4 removed from a device 2 with a camera of a portable terminal 10. A server 20 analyzes a photographic image received from the portable terminal 10, creates a work object list 8 including the removed components 4 and the number of components, collates the work object list 8 with a component list which is obtained from each of previously prepared component removal work procedure manuals and includes components to be removed according to the removal procedure and the number of the components, and thereby specifies the removal work procedure manual. The server 20 reverses a removal procedure described in the removal work procedure manual, thereby creates a mounting procedure of the removed components 4, transmits the mounting procedure to the portable terminal 10, and thereby presents the mounting procedure to the operator.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing system and a program. [Background technology]

[0002] 2. Description of the Related Art In the maintenance of equipment, for example, various techniques have been proposed for assisting in the task of replacing parts that have been removed from the equipment.

[0003] For example, Patent Document 1 proposes a technique for assisting the user in returning a part to a device by capturing video of the process of removing the part from the device and presenting the captured video in reverse. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2021-180408 A [Patent Document 2] JP 2017-097856 A [Patent Document 3] JP 2019-163670 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the conventional technology, when returning parts to the equipment, the worker needs to watch a video image played in reverse, which may cause the worker to take time to recognize the order in which to return the parts to the equipment.

[0006] In addition, in the past, if a moving image could not be captured, it was not possible to provide information for returning the part to the device. In other words, if the user realizes that they failed to capture a moving image after removing the part from the device, there is a possibility that the part cannot be returned to the device.

[0007] An object of the present invention is to provide information that assists in installing a part in a device by referring to the part after it has been removed from the device. [Means for solving the problem]

[0008] The information processing system of the present invention is characterized in that it comprises a processor which acquires a photographed image of a part removed from an equipment, and uses the photographed image to select removal information corresponding to the part removed from the equipment from removal information which indicates a procedure for removing the part from the equipment and which is stored in a memory means, and presents installation procedure information which indicates a procedure for installing the part into the equipment which has been set by reversing the removal procedure for the part indicated by the selected removal information.

[0009] The processor also creates part information including identification information of the photographed parts and the number of the parts identified by analyzing the captured image, and selects, from the removal information stored in the storage means, removal information in which the identification information of the parts to be removed identified from the removal information and the number of the parts match the identification information of the parts and the number of the parts included in the part information, as removal information corresponding to the parts removed from the equipment.

[0010] The processor may also present the part information to a user in a modifiable manner, and when the presented part information is modified by the user, update the part information with the modified content.

[0011] Furthermore, when there is no removal information stored in the storage means that matches the identification information of the parts and the number of the parts included in the part information, the processor selects from the removal information stored in the storage means removal information in which the identification information of the parts to be removed and the number of the parts specified from the removal information are similar to the identification information of the parts and the number of the parts included in the part information, as a candidate removal information corresponding to the parts removed from the equipment, and presents the selected candidate removal information to a user to allow the user to select removal information corresponding to the parts removed from the equipment.

[0012] In addition, when there is a plurality of pieces of removal information stored in the memory means that match the identification information of the parts and the number of the parts included in the part information, the processor presents the plurality of pieces of removal information to the user as selection candidates, and allows the user to select the removal information corresponding to the parts removed from the equipment.

[0013] The processor may also display, as the installation procedure information, images of parts removed from the device, linked with the order in which the parts are to be installed in the device.

[0014] In addition, when the terminal device is displaying parts that have been removed from the equipment, the processor displays the installation procedure information by linking each part displayed by the terminal device to the order in which the parts are to be installed on the equipment.

[0015] The processor is also characterized in that it adds the order in which the parts are to be attached to the equipment to the image of each part contained in the acquired photographed image, and presents the photographed image after the installation order has been added to the image of each part as the installation procedure information.

[0016] The processor is also characterized in that it displays, as the installation procedure information, a part to be next installed in the device from among parts removed from the device in an identifiable manner.

[0017] The processor is also characterized in that it displays, as the installation procedure information, a part to be next installed on the device from among the parts removed from the device, and an installation position of the part on the device.

[0018] The program of the present invention enables a computer to perform the following functions: acquiring a photographed image of a part removed from an equipment; using the photographed image to select removal information corresponding to the part removed from the equipment from removal information stored in a storage means that indicates the procedure for removing the part from the equipment; and presenting installation procedure information that indicates the procedure for installing the part into the equipment that has been set by reversing the removal procedure for the part indicated by the selected removal information. Effect of the Invention

[0019] According to the invention recited in claim 1, it is possible to provide information for assisting in mounting a part on an apparatus by referring to the part after it has been removed from the apparatus.

[0020] According to the invention recited in claim 2, removal information can be selected from the removed parts and the number of the parts.

[0021] According to the invention recited in claim 3, even if the analysis result of the captured image is incorrect, the user can be prompted to correct the identification information of the parts removed from the device and the number of the parts.

[0022] According to the invention as recited in claim 4, the user can be made to select the removal information to be selected.

[0023] According to the invention as recited in claim 5, the user can be allowed to select the removal information to be selected.

[0024] According to the invention as recited in claim 6, it is possible to present to the user the order in which the parts removed from a device should be attached to the device.

[0025] According to the invention as recited in claim 7, the installation order can be linked and displayed for each part currently displayed on the terminal device.

[0026] According to the invention as recited in claim 8, the order of installation can be added to the image of each part included in the acquired photographed image, and the order of installation can be presented as installation procedure information.

[0027] According to the invention as recited in claim 9, the part that is to be installed next in the device can be presented to the user.

[0028] According to the invention as recited in claim 10, it is possible to present to the user the installation position of the next part to be installed in the device.

[0029] According to the invention recited in claim 11, it is possible to provide information for assisting in mounting a part on a device by referring to the part after it has been removed from the device. [Brief description of the drawings]

[0030] [Figure 1] 1 is a diagram showing an overall configuration of a work support system according to an embodiment of the present invention and a schematic diagram showing a basic process flow for presenting an installation procedure. [Diagram 2] 1 is a block diagram showing a configuration of a work support system according to an embodiment of the present invention; [Figure 3A] 10 is a flowchart showing a part installation procedure presentation process according to the present embodiment. [Figure 3B] 3B is a flowchart continuing from FIG. 3A. [Figure 4] 10A to 10C are diagrams illustrating a process for correcting a task target list obtained from a captured image in the present embodiment. [Diagram 5]FIG. 11 is a diagram showing an example of a screen display of installation procedure information displayed on a mobile terminal in the present embodiment. [Figure 6] 13A and 13B are diagrams showing other examples of screen display of installation procedure information displayed on a mobile terminal in the present embodiment. [Figure 7] 13A and 13B are diagrams showing other examples of screen display of installation procedure information displayed on a mobile terminal in the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

[0032] FIG. 1 is a diagram showing an overall configuration of a work support system according to this embodiment and a schematic diagram of a basic process flow for presenting an installation procedure. The work support system according to this embodiment is used to support maintenance work and the like of a device 2 installed in a facility such as a building. The device 2 is, for example, a multifunction peripheral. In the maintenance work, a plurality of parts 4 attached to the device 2 are removed, and the removed parts 4 are then replaced. Work to which the work support system according to this embodiment can be applied does not have to be limited to maintenance work, so long as it includes work to remove the parts 4 from the device 2 and then replace the parts 4.

[0033] In addition, the components 4a to 4d will be collectively referred to as "component 4" as described above when there is no need to distinguish between them.

[0034] The work support system in this embodiment includes a mobile terminal 10 and a server 20. The mobile terminal 10 is carried by a worker who performs maintenance work, and is brought to the work site for use. The mobile terminal 10 is assumed to be, for example, a smartphone or a tablet terminal. The mobile terminal 10 may be realized by a conventional general-purpose hardware configuration. That is, the mobile terminal 10 includes a CPU, a ROM, a RAM, a storage, a network interface provided as a communication means for communicating with the server 20, and a touch panel as a user interface. Furthermore, the mobile terminal 10 in this embodiment is equipped with a camera as a photographing means. Even if the mobile terminal 10 is not equipped with a photographing means, it is necessary for the mobile terminal 10 to have an interface for connecting the photographing means.

[0035] The work assistance system may include multiple mobile terminals 10. However, since each mobile terminal 10 only needs to have equivalent processing functions, the details of which will be described later, FIG. 1 illustrates only one mobile terminal 10.

[0036] When the server 20 receives a photographed image of the part 4 removed from the device 2 from the mobile terminal 10, the server 20 identifies a work procedure manual for removing the removed part 4, and creates a procedure for installing the part 4 in the device 2 by referring to the identified work procedure manual, and presents the procedure to the worker. The server 20 may be realized by a conventional general-purpose hardware configuration. That is, the server 20 includes at least a CPU, a ROM, a RAM, a storage, and a network interface provided as a communication means for communicating with the mobile terminal 10.

[0037] 2 is a block diagram showing the configuration of the work support system in this embodiment. As described above, the work support system in this embodiment has the mobile terminal 10 and the server 20. The mobile terminal 10 has an image capture unit 11, a user interface (UI) unit 12, a captured image transmission unit 13, and an inquiry response processing unit 14. Note that components that are not used in the description of this embodiment are omitted from the figure.

[0038] The photographing unit 11 is realized by a camera, and photographs in response to the operation of the worker. In this embodiment, the worker photographs the part 4 removed from the device 2 using the camera. The user interface unit 12 accepts user input from the touch panel and displays it on the touch panel. The installation procedure presenting unit 122 in the user interface unit 12 presents the installation procedure information transmitted from the server 20 to the worker, for example by displaying it on the touch panel. The photographed image transmitting unit 13 transmits the image photographed by the photographing unit 11 to the server 20. The inquiry response processing unit 14 responds to inquiries from the server 20, and replies to the server 20 with, for example, input instructions or selection results by the worker.

[0039] Each of the components 11 to 14 in the mobile terminal 10 is realized by the cooperative operation of a computer installed in the mobile terminal 10 and a program that runs on a CPU installed in the computer.

[0040] The server 20 has a photographed image receiving unit 21, a work target list creating unit 22, a work procedure manual identifying unit 23, an installation procedure information providing unit 24, and a work procedure manual storage unit 25. Note that components that are not used in the description of this embodiment are omitted from the drawing.

[0041] The captured image receiving unit 21 receives the captured images transmitted from the mobile terminal 10. The work target list creating unit 22 analyzes the captured images received by the captured image receiving unit 21 to determine the captured parts 4 and the number of each part 4, and creates a work target list as part information. In this embodiment, a part 4 is identified from other parts 4 by its identification information, and the name of the part 4 (hereinafter, "part name") is used as the identification information.

[0042] The work procedure manual storage unit 25 stores electronic data of a removal work procedure manual (hereinafter, simply referred to as a "work procedure manual") in which the procedure for removing the parts 4 from the device 2 is described. By referring to the removal procedure for the parts 4 described in the removal work procedure manual, it is possible to identify the parts 4 to be removed and the number of each part 4 to be removed. In the parts list prepared for each removal work procedure manual, the parts 4 to be removed and the number of each part 4 are set in association with each other. As will be described later, since the work target list is collated with the parts list corresponding to each removal work procedure manual, it is preferable that the work target list creation unit 22 creates the work target list with the same data configuration as the parts list.

[0043] The work procedure manual identification unit 23 collates the work target list created by the work target list creation unit 22 with the parts list corresponding to the work procedure manual storage unit 25 to identify the work procedure manual. That is, the work procedure manual identification unit 23 identifies a matching parts list based on the parts 4 removed from the device 2 and the number of each part 4, and identifies the work procedure manual corresponding to the identified parts list. Note that if the parts list is included as an attachment or the like to the removal work procedure manual, the parts list is utilized. If a parts list is not prepared in advance, for example, the work procedure manual identification unit 23 may create a parts list by subjecting the removal work procedure manual to character analysis processing.

[0044] An inquiry processing unit 231 included in the work procedure manual identification unit 23, when there is not only one work procedure manual that describes parts 4 and the number of parts 4 that match the parts 4 and the number of parts 4 included in the work target list, identifies the only one work procedure manual by, for example, inquiring of the worker.

[0045] Incidentally, if the removal procedure manual can be identified, the parts list can be identified. However, identifying the parts list does not necessarily mean that the removal procedure manual can be identified. This is because the parts 4 removed from the device 2 according to a certain removal procedure and the number of those parts 4 may match the parts 4 removed according to another removal procedure and the number of those parts 4. Note that in this embodiment, the explanation will be given on the assumption that the setting contents of the parts list obtained from the removal procedure manual are correct.

[0046] The installation procedure information providing unit 24 provides installation procedure information, including an installation procedure for installing the part 4 on the equipment 2 in accordance with the work procedure manual identified by the work procedure manual identifying unit 23, to a worker using the mobile terminal 10 that sent the captured image.

[0047] Each of the components 21 to 24 in the server 20 is realized by the cooperative operation of a computer installed in the server 20 and a program running on a CPU installed in the computer. Also, the work procedure manual storage unit 25 is realized by a HDD installed in the server 20. Alternatively, a RAM or, if possible, an external storage means may be used via a network.

[0048] Although the "information processing system" in this embodiment will be described as being configured with one server computer as an example, it may be configured with multiple computers.

[0049] Furthermore, the programs used in the present embodiment can be provided not only by communication means, but also by being stored in a computer-readable recording medium such as a CD-ROM or a USB memory. The programs provided from the communication means or recording medium are installed in a computer, and various processes are realized by the computer's CPU sequentially executing the programs.

[0050] Next, the operation of this embodiment will be described. For the sake of convenience, the following description will be given assuming that there is only one type of device 2 to be maintained. In other words, unless otherwise specified, the model is assumed to be specific.

[0051] In this embodiment, the worker brings the mobile terminal 10 to the installation location of the device 2 for the maintenance work of the device 2. Then, the worker removes the multiple types of parts 4a to 4d from the device 2 in a predetermined order during the maintenance work. The worker places the removed parts 4 appropriately near the device 2, such as on the floor, 6 without being aware of the order in which they were removed. Incidentally, there are multiple types of maintenance work for the device 2, and a removal work procedure manual is prepared in advance. The removal work procedure manual is stored in the procedure manual storage unit 25 as described above. Therefore, the worker may obtain the removal work procedure manual corresponding to the maintenance work to be performed by requesting the server 20 before or during the maintenance work, and may display the removal work procedure manual on the mobile terminal 10 and perform the work while referring to it during the maintenance work. Of course, the worker may perform the maintenance work without referring to the removal work procedure manual. When referring to the removal work procedure manual, the chapter, section, page, etc. in which the removal work procedure manual was referred to and the order of reference may be recorded in a log or the like. Whether or not a worker refers to a removal procedure manual, the worker is performing the removal work according to the removal procedure described in one of the removal procedure manuals. Alternatively, as a result of removing the part 4, the part 4 removed and the number of each part 4 should always match the parts to be removed and the number of each part described in at least one of the removal procedure manuals. In reality, there may be cases where they do not match, but this point will be discussed later.

[0052] 1 shows a schematic example in which a total of five parts 4 of four types have been removed from equipment 2. When a worker removes the parts 4 to be removed in the maintenance work from equipment 2, he or she photographs all of the removed parts 4. When the parts 4 are photographed, the photographed image transmission unit 13 transmits the photographed images to the server 20.

[0053] In this embodiment, for ease of explanation, an example will be described in which a single image is taken to capture all of the removed parts 4; however, since the server 20 only needs to be able to obtain images of all of the parts 4 removed from the device 2, it is sufficient that images of all of the parts 4 are ultimately obtained, even if the images are divided into multiple images taken multiple times.

[0054] Next, the part installation procedure presentation process in this embodiment, from when the server 20 acquires a photographed image of the part 4 removed from the device 2 to when the installation procedure for the part 4 is presented to the worker, will be described with reference to the flowcharts shown in FIGS. 3A and 3B.

[0055] When the captured image receiving unit 21 receives the captured image transmitted from the mobile terminal 10 and thereby acquires the captured image of the part (hereinafter also referred to as the "part to be removed" or simply "part") 4 removed from the equipment 2 (step S101), the work target list creation unit 22 analyzes the received captured image to extract the captured part 4 to be removed and create a work target list (step S102).

[0056] 1 shows an example of a work target list 8 created from captured images of all parts 4. The work target list 8 is part information related to the parts to be removed 4 identified by analyzing the captured images. The work target list 8 is created by associating part names, which serve as identification information for the parts to be removed 4, with the number of such parts.

[0057] The photographed parts 4 are recognized by the following process. Although the image may be a moving image, in this embodiment, a still image obtained by photographing with a camera is used. The process itself for recognizing the parts 4 from the photographed image may utilize existing technology, but a typical example is shown below.

[0058] First, the types of images to be handled may be two-dimensional images or three-dimensional images. In the case of two-dimensional images, comparison image data captured from multiple directions (e.g., from above, the front, the side, etc.) is prepared in advance for each part included in the device 2. Then, the image of the part 4 to be removed extracted from the captured image sent from the mobile terminal 10 is compared with the multiple pieces of comparison image data for each part to identify the type of the part 4 to be removed.

[0059] In the case of a three-dimensional image, three-dimensional shape data is generated for each part included in the device 2 and prepared in advance as comparison image data. The three-dimensional shape data may be created from point cloud data (e.g., each vertex and edge end point of a part, etc.) acquired using, for example, a depth sensor or Visual SLAM technology. The three-dimensional shape data is created, for example, by combining image data captured from multiple positions and in multiple postures.

[0060] Next, the work procedure manual specification unit 23 specifies the removal work procedure manual using the work target list 8 and obtains it from the work procedure manual storage unit 25, while collating the work target list 8 with the parts list (step S103). The removal work procedure manual describes the procedure for removing the parts 4 from the device 2, and by referring to the removal procedure, it is possible to specify the type of parts 4 to be removed and the number of parts to be removed. The parts list is set so that the parts to be removed and the number of parts to be removed are associated with each other, and is prepared for each removal work procedure manual. In this embodiment, the work target list 8 is created with the same data structure as the parts list to make comparison easier.

[0061] As described above, the work procedure manual identification unit 23 collates the work target list 8 with the parts list corresponding to each removal work procedure manual, and when there is a parts list that is completely identical to the work target list 8 (Y in step S104), and when there is only one completely identical parts list (Y in step S105), the work procedure manual identification unit 23 acquires the removal work procedure manual that corresponds to that parts list (step S108). "Completely identical" here means that both the parts that make up the parts list and the number of those parts match.

[0062] If a parts list that is completely identical to the work target list 8 exists but is not the only completely identical parts list (Y in step S104, N in step S105), that is, if multiple parts lists exist, the inquiry processing unit 231 in the work procedure manual identification unit 23 inquires of the worker about the order in which the parts 4 were removed by sending the removal procedures corresponding to each of the multiple parts lists to the mobile terminal 10 as selection candidates (step S106).

[0063] Incidentally, if the order in which parts are removed differs due to differences in the contents of the maintenance work, etc., a different removal work procedure manual will be prepared. In other words, when a removal work procedure manual is specified, the parts list is also specified. On the other hand, if the parts to be removed and their number are the same, the parts list will be the same even if the removal procedure is different. In other words, even if a parts list is specified, there may be multiple removal procedures that result in completely identical parts lists, so a single removal work procedure manual cannot be specified.

[0064] When the list of removal procedure candidates is sent from the server 20, the inquiry response processor 14 in the mobile terminal 10 displays the removal procedure candidates on a touch panel so that they can be selected, thereby presenting them to the worker.

[0065] Since the worker was performing the work of removing part 4 until just before, it is considered that the worker can select the removal procedure that he / she performed from the presented removal procedures. In this way, the worker selects the removal procedure that he / she performed from the presented candidate removal procedures. The inquiry response processing unit 14 replies with the removal procedure selected by the worker.

[0066] The inquiry processor 231 acquires the removal procedure manual corresponding to the removal procedure returned from the mobile terminal 10 in the above manner (step S107).

[0067] In this manner, when a single removal work procedure manual corresponding to a parts list that is completely identical to the work target list 8 can be selected from the removal work procedure manuals stored in the work procedure manual memory unit 25 (steps S107, S108), the installation procedure information providing unit 24 sets an installation procedure for the part 4 on the equipment 2 by reversing the removal procedure described in the removal work procedure manual, and provides the installation procedure information indicating the installation procedure by transmitting it to the mobile terminal 10 (step S109).

[0068] When the installation procedure is sent from the server 20, the installation procedure presenting unit 122 in the mobile terminal 10 presents the received installation procedure to the worker by, for example, displaying it on a touch panel.

[0069] The processing in the server 20 described above with reference to FIGS. 3A and 3B will now be described again with reference to FIG.

[0070] 1 illustrates two removal procedures P1 and P2. Although not illustrated, the parts list for removal procedure P1 shows that the number of parts A, B, and C is one, and the number of parts D is two, based on the description example. On the other hand, the parts list for removal procedure P2 shows that the number of parts A, B, C, and D is one, based on the description example. From this, it can be seen that the work target list 8 created from the images captured by the mobile terminal 10 is completely identical to the parts list corresponding to the removal work procedure manual in which the removal procedure P1 is described. As a result, the server 20 creates an installation procedure by reversing the removal procedure P1, and transmits the installation procedure to the mobile terminal 10, thereby presenting it to the worker.

[0071] Incidentally, the work target list 8 created from the captured image is created by analyzing the captured image. Therefore, depending on how the parts 4 are placed or arranged, it is not always possible to accurately extract the type and number of parts 4 to be removed from the captured image. In other words, a correct work target list 8 is not always created. Therefore, it is conceivable that the created work target list 8 may not be completely identical to any parts list.

[0072] Returning to FIG. 3A, if there is no parts list that is exactly identical to the work target list 8 (N in step S104), the inquiry processing unit 231 inquires of the worker whether the work target list 8 is correct by sending the work target list 8 to the mobile terminal 10 (step S110).

[0073] When the task target list 8 is sent from the server 20, the inquiry response processor 14 in the mobile terminal 10 displays the task target list 8 on a touch panel in an editable manner, thereby presenting it to the worker.

[0074] Since the worker places the parts 4 in the vicinity 6 of the equipment 2 himself, the worker can grasp the correct types of parts 4 to be removed and the number of each part 4 to be removed. Therefore, the worker refers to the work target list 8 displayed on the touch panel, and corrects any errors in the work target list 8. If the work target list 8 is corrected by the worker, the inquiry response processing unit 14 replies with the work target list 8 updated with the corrections. On the other hand, if the worker does not correct the presented work target list 8, it is assumed that there are no errors in the work target list 8. In this case, the inquiry response processing unit 14 replies that the work target list 8 was not corrected by the worker, in other words, the work target list 8 is correct.

[0075] The inquiry processor 231 receives a response from the mobile terminal 10. If a corrected task target list 8 is received (Y in step S111), the task target list 8 created from the captured image is updated with the corrected task target list 8 (step S113). Then, the process returns to step S103 and the above-described process is executed.

[0076] On the other hand, if a response is received from mobile terminal 10 indicating that work target list 8 is correct without being modified by the worker, the fact that there is no parts list that is exactly the same as work target list 8 even though work target list 8 is correct means that there is no removal work procedure manual on which an installation procedure can be created. Therefore, since installation procedure information provider 24 cannot create the installation procedure, it transmits a message to mobile terminal 10 indicating that the installation procedure cannot be provided, and notifies the worker of that fact (step S112).

[0077] The process when there is no parts list that is completely identical to the work target list 8, as described above with reference to FIGS. 3A and 3B, will now be described again with reference to FIG.

[0078] Fig. 4(a) shows an example of a photographed image of a part 4, Fig. 4(b) shows a work target list 8 obtained from the photographed image, and Fig. 4(c) shows the work target list 8 after it has been corrected by a worker. Fig. 4(d) shows part name information showing the relationship between the image of the part 4 shown in Fig. 4(a) and the part name for reference.

[0079] In step S102, if there is no parts list that is exactly the same as the work target list 8 created by analyzing the captured image, the work target list 8 is sent to the mobile terminal 10, and the worker is asked about it. Then, if the work target list 8 (FIG. 4(b)) presented to the worker contains an error, the worker corrects the error. The worker can easily check whether the work target list 8 is correct by comparing the actual parts 4 placed on the floor, for example, with the work target list 8 (FIG. 4(b)) displayed on the touch panel of the mobile terminal 10.

[0080] Comparing the work target lists 8 shown in Figures 4(b) and 4(c), we can see that the worker changed part F to part B, corrected the quantities of parts A and C, and also corrected part E to be deleted since it was mistakenly part C. In the work target list 8 shown in Figure 4(c), the parts that have been corrected are underlined for convenience.

[0081] In this way, if there is an error in the work target list 8 (FIG. 4(b)) that is automatically obtained by analyzing the captured image, the worker can be made to correct it. Then, by using the corrected work target list 8 (FIG. 4(c)) and comparing it again with the parts list (step S103), it is believed that an exactly identical parts list can be found.

[0082] However, workers do not always correctly fill in the work target list 8. It is possible that they do not correctly understand the part names, or confuse similar parts. In other words, it is possible that they are unable to obtain an accurate work target list 8, and as a result are unable to find the completely identical parts list.

[0083] In this case, in the above description, if a completely identical parts list does not exist even if the worker corrects the work target list 8, and if the worker does not correct the work target list 8 and as a result cannot identify a removal work procedure manual (N in step S104, N in step S111), the worker is notified that the installation procedure cannot be provided (step S112). However, before notifying, the work procedure manual identification unit 23 may refer to the work procedure manual storage unit 25 to search for a parts list similar to the latest work target list 8, select one or more parts lists extracted as the search result as candidates for the correct work target list 8, and present them to the worker. Note that the removal procedure described in the removal work procedure manual corresponding to the parts list may be presented instead of or together with the parts list. In this way, the worker may be informed that there may be an error in the work target list 8 that the worker once recognized as correct, and one or more parts lists that are candidates for correction of the work target list 8 may be presented to the worker.

[0084] When the worker selects one parts list from the candidate parts lists, the work procedure manual specification unit 23 regards the selected parts list as the corrected work target list 8 (Y in step S111), and can return to step S103. In this case, a completely identical parts list will always exist.

[0085] In addition, the similarity between the work target list 8 and the parts list can be determined by setting criteria for similarity in advance, such as when the part type configuration is the same but the number differs by about one, or when the part type differs by about one, and then extracting parts lists that are similar to the work target list 8 according to those criteria.

[0086] As described above, in this embodiment, it is possible to present to the worker the installation procedure for replacing the removed part 4. At this time, it is necessary to identify only one parts list that is completely identical to the work target list 8, and information that serves as a reference for the process of identifying that one may be obtained from the worker.

[0087] For example, the tool or jig used to remove the part 4 is photographed together with the part 4. Alternatively, if there is a plan to replace the part 4, the part 4 to be replaced is also photographed. In this embodiment, the description is given on the assumption that the part 4 removed from the device 2 is returned to its original state, but the removed part 4 may also be replaced. In this case, the "removed part" to be replaced should be read as the "part to be replaced" as appropriate. When replacing the removed part 4, it is necessary to take a good picture of the part 4 and arrange it so that the correspondence between the part 4 before and after the replacement can be understood and so that an error does not occur in the calculation of the number of parts. For example, a rule is set for the arrangement of the part 4, and text information such as a comment is added to or included in the photographed image. As a result, even if a single parts list cannot be identified only by comparing the work target list 8 and the parts list, it is easy to identify a single parts list by referring to additional information such as the photographed image of the tool or the like described above. In other words, information on the jig or replacement part can be used as information to support the process of narrowing down the parts list and identifying one. The mobile terminal 10 may be configured to transmit this support information to the server 20 separately, without including it in the captured image of the part 4, under instructions from the worker.

[0088] In the above description, the model of the device 2 to be maintained is limited to one. If there are multiple models of the device 2, the device 2 may be identified by adding information (e.g., a model code) for identifying the model of the device 2 to the captured image sent from the mobile terminal 10 to the server 20, or by sending the captured image of the device 2 together, so that the removal work instructions stored in the work instruction storage unit 25 can be narrowed down to the removal work instruction corresponding to the relevant model.

[0089] Incidentally, as exemplified in FIG. 1, the installation procedure information presented to the worker includes the installation procedure for the component 4. However, the form of the installation procedure information presented to the worker does not need to be limited to this. Basically, the information may be presented as text information as exemplified in FIG. 1, or a still image may be used when presenting text information. However, the installation procedure information may be presented as a moving image, or, if the mobile terminal 10 has a display function using AR (Augmented Reality) technology (hereinafter, "AR function"), the AR function may be used to present the installation procedure information.

[0090] Hereinafter, examples of displaying the mounting procedure information will be described with reference to the drawings. In Fig. 1, the mounting procedure information of the part 4 is displayed as text information, but it may be displayed diagrammatically as image information as exemplified in Figs. 5 to 7.

[0091] The processor in the mobile terminal 10 cooperates with the processor in the server 20 to control display on the touch panel in response to the installation procedure information and information display control instructions transmitted from the server 20. More specifically, the server 20 transmits the installation procedure information to the mobile terminal 10 and controls the mobile terminal 10 to display the installation procedure information, and the mobile terminal 10 displays the installation procedure information in response to the control by the server 20.

[0092] In this way, when the processor in the mobile terminal 10 and the processor in the server 20 work together to display the installation procedure information, the information processing system may be considered to be configured to include not only the server 20 but also the mobile terminal 10. However, for the sake of convenience, the control process in the server 20 may be omitted here, and the process in which the mobile terminal 10 displays the installation procedure information may be described.

[0093] Fig. 5 is a diagram showing an example of a screen display of installation procedure information displayed on a touch panel of the mobile terminal 10. As exemplified in Fig. 5, the installation procedure presenter 122 displays an image of each part 4 to be installed in the device 2, and associates the order in which the parts 4 are to be installed in the device 2 with the image. In the display example shown in Fig. 5, the installation order is displayed as circled numbers 31 arranged near the corresponding part 4. Of course, the display form of the installation order is only one example, and the installation order may be displayed in another form other than the circled numbers 31 as long as the order is known.

[0094] When the mobile terminal 10 uses the AR function to display the installation procedure information, the mobile terminal 10 uses a camera to capture an image of the parts 4 to be removed that are placed near the device 2. The image capture does not necessarily require the generation of captured image data. The installation procedure presenting unit 122 identifies the order in which the captured parts 4 should be installed from the installation procedure information received from the server 20, and displays the installation procedure by associating each part 4 with a circled number 31 as shown in FIG. 5.

[0095] On the other hand, when the mobile terminal 10 does not have an AR function or does not use the AR function, the installation procedure information providing unit 24 adds the order in which the parts 4 are to be installed on the device 2 according to the installation procedure to the image of each part 4 included in the captured image acquired by the photographing unit 11. Then, the installation procedure information providing unit 24 transmits the captured image after the installation order is added to the image of each part 4 to the mobile terminal 10 as installation procedure information. As a result, the installation procedure presenting unit 122 in the mobile terminal 10 can present the installation procedure information exemplified in FIG. 5 to the worker by displaying the captured image sent from the server 20.

[0096] Fig. 6 is a diagram showing another example of the screen display of the installation procedure information displayed on the touch panel of the mobile terminal 10. Fig. 5 shows an example of a display in which none of the parts 4 have been installed on the device 2 yet, while Fig. 6 shows an example of a display in which some of the removed parts 4 (parts B and C in Fig. 6) have already been installed on the device 2. The installation procedure presenting unit 122 then displays the next part 4 to be installed so that it can be identified. In the screen display example shown in Fig. 6, the next part 4 to be installed is presented to the worker by a display element 32 that combines the text information "Next" with an arrow.

[0097] When the mobile terminal 10 is equipped with an AR function, the mobile terminal 10 captures an image of the vicinity 6 of the device 2 with a camera, as in the case of Fig. 5. However, in this case, some of the parts 4 to be removed (parts B and C in Fig. 6) have already been installed, so the installation procedure presenting unit 122 identifies the next part 4 to be installed by comparing the installation procedure of the parts 4 in the installation procedure information with the parts 4 that have not yet been installed in the captured image, and uses the display element 32 to display the next part 4 to be installed in an identifiable manner.

[0098] The display form of the installation procedure information shown in Fig. 6 is basically realized by using the AR function. However, even if the mobile terminal 10 does not have the AR function or does not use the AR function, it is possible to display the installation procedure information shown in Fig. 6.

[0099] For example, the installation procedure information providing unit 24 transmits the captured image to the mobile terminal 10 in the same manner as in FIG. 5. However, the installation procedure corresponding to the circled number 31 is not presented, and the display element 32 is associated with the part 4 to be installed first. The installation procedure presenting unit 122 in the mobile terminal 10 displays the captured image sent from the server 20. When the worker installs the part 4 indicated by the displayed display element 32 on the device 2, the worker performs a predetermined operation to indicate that the part 4 has been installed. In response to the predetermined operation by the worker, the mobile terminal 10 transmits an information transmission request including the fact that the part 4 has been installed to the server 20. In response to the information transmission request from the mobile terminal 10, the installation procedure information providing unit 24 deletes the installed part 4 from the captured image, associates the display element 32 with the image of the part 4 to be installed next, and then transmits the captured image to the mobile terminal 10. The mobile terminal 10 and the server 20 cooperate to repeat the above process using the information transmission request, so that the screen display shown in FIG. 6 can be realized even when the mobile terminal 10 does not use the AR function. In the above process using an information transmission request, the installation procedure information providing unit 24 can be said to divide and transmit the installation procedure information.

[0100] In this manner, regardless of whether the mobile terminal 10 uses the AR function or not, it is possible to identifiably display the part 4 to be next attached to the device 2 from among the parts 4 removed from the device 2.

[0101] In Fig. 6, the next part 4 to be installed is displayed so as to be identifiable by the display element 32 consisting of the character "Next" and an arrow, but the display element 32 does not need to be limited to this form. For example, the next part 4 to be installed may be circled.

[0102] Fig. 7 is a diagram showing another example of a screen display of installation procedure information displayed on the touch panel of the mobile terminal 10. In the screen display example shown in Fig. 6, the display content presents the next part 4 to be installed to the device 2 from among the parts 4 to be installed. In the screen display example shown in Fig. 7, not only is the next part 4 to be installed to the device 2 from among the parts 4 removed from the device 2 presented to the worker, but the installation position of the next part 4 to be installed on the device 2 is also displayed. The screen display example shown in Fig. 7 can be formed relatively easily by using an AR function.

[0103] 7 shows an example of a screen display in which the left half of the touch panel of the mobile terminal 10 is a display area 33 of the device 2, and the right half is a display area 34 of the removable part 4. Of course, the ranges and layouts of the respective display areas 33, 34 are merely examples, and are not necessarily limited to the screen display example shown in FIG.

[0104] 7, the mobile terminal 10 needs to know an image of the device 2 and the installation position of the part 4 to be removed relative to the device 2. The image of the device 2 can be captured by using a camera and displayed in the display area 33. The installation position of the part 4 to be removed is the same as the removal position of the part 4. Incidentally, since the removal position of the part 4 can be specified by specifying the removal procedure manual, the mobile terminal 10 can obtain installation procedure information including the removal procedure manual from the server 20. Alternatively, the server 20 can specify the installation position of the part 4 that can be specified from the removal procedure manual, and the mobile terminal 10 can obtain installation procedure information including the specified installation position of the part 4 from the server 20.

[0105] The installation procedure presenter 122 in the mobile terminal 10 displays the installed part 4 by adding it to the image of the device 2. In the display using the AR function, even if the part 4 installed inside the device 2 cannot be visually recognized, the installation position of the part 4 can be specified from the removal procedure manual, so it is possible to virtually display it. The display area 34 of the part 4 to be removed can be displayed by processing using FIG. 6. In the screen display example shown in FIG. 7, the installation position 35 of the part 4 to be installed next on the device 2 is indicated by an arrow 36, instead of the display element 32 indicating the part 4 to be installed next in FIG. 6. As exemplified in FIG. 7, the installation position 35 of the part 4 is easy to imagine if it is drawn in the shape of the part 4 to be installed. In addition, the installation position 35 is indicated by a dashed line so that it is easy to distinguish it from the part 4 that has already been installed.

[0106] The example screen display shown in FIG. 7 can be formed relatively easily by utilizing the AR function; however, since the server 20 can identify the model of the device 2 from the removal procedure manual, it is possible to form the screen display shown in FIG. 7 without utilizing the AR function, for example, by transmitting and providing an image of the device 2 that has been prepared in advance to the mobile terminal 10.

[0107] In the above embodiments, the term "processor" refers to a processor in a broad sense, including general-purpose processors (e.g., CPU: Central Processing Unit, etc.) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.).

[0108] In addition, the operations of the processor in the above embodiments may be performed not only by one processor, but also by multiple processors located at physically separate locations working together. The order of the operations of the processor is not limited to the order described in the above embodiments, and may be changed as appropriate.

[0109] (Additional Note) (((1))) A processor is provided. The processor, Acquire images of the parts removed from the equipment, selecting removal information corresponding to the part removed from the device from removal information stored in a storage means, the removal information indicating a procedure for removing the part from the device, using the captured image; presenting installation procedure information indicating an installation procedure for the set device by reversing the removal procedure of the part indicated by the selected removal information; An information processing system comprising: (((2))) The processor, creating part information including identification information of the photographed parts and the number of the parts identified by analyzing the photographed image; selecting, from among the removal information stored in the storage means, removal information in which identification information of parts to be removed and the number of said parts match identification information of parts and the number of said parts included in the part information, as removal information corresponding to the parts removed from the device; 1. The information processing system according to claim 1, (((3))) The processor, presenting the part information to a user in a modifiable manner; When the presented part information is modified by a user, the part information is updated with the modified content. 2. The information processing system according to claim 1 , (((4))) The processor, if there is no removal information stored in the storage means that matches the identification information of the parts and the number of the parts included in the part information, selecting, from the removal information stored in the storage means, removal information in which the identification information of the parts to be removed and the number of the parts specified in the removal information are similar to the identification information of the parts and the number of the parts included in the part information, as a candidate for removal information corresponding to the parts removed from the device; presenting the selected candidates of removal information to a user and allowing the user to select removal information corresponding to the part removed from the device; 2. An information processing system according to claim 1, wherein the information processing system comprises: (((5))) The information processing system described in (((2))) or (((3))) is characterized in that, when there is a plurality of pieces of removal information stored in the storage means that match the part identification information and number of parts included in the part information, the processor presents the plurality of pieces of removal information to a user as selection candidates and allows the user to select the removal information corresponding to the part that has been removed from the equipment. (((6))) The information processing system of any one of (((1))) to (((5))), wherein the processor displays, as the installation procedure information, images of parts removed from the equipment, linked to the order in which the parts are to be installed on the equipment. (((7))) The information processing system described in (((6))) is characterized in that, when the terminal device is displaying parts that have been removed from the equipment, the processor displays the installation procedure information by linking each part displayed by the terminal device to the order in which the parts are to be installed on the equipment. (((8))) The processor, adding an order in which the parts are to be attached to the device to the images of the parts included in the acquired photographed images; presenting the photographed images after adding the installation order to the images of the components as the installation procedure information; 2. The information processing system according to claim 1 , wherein: (((9))) The information processing system according to any one of ((1)) to (((5))), wherein the processor displays, as the installation procedure information, a part to be next installed on the equipment from among parts removed from the equipment in a manner that enables the part to be identified. (((10))) The information processing system described in any one of ((1))) to (((5))) is characterized in that the processor displays, as the installation procedure information, the next part to be installed on the equipment from among the parts removed from the equipment, and the installation position of the part on the equipment. (((11))) On the computer, Ability to capture images of parts removed from equipment; a function of selecting, using the captured image, removal information corresponding to the part removed from the device from removal information stored in a storage means, the removal information indicating a procedure for removing the part from the device; a function of presenting installation procedure information indicating an installation procedure for the set device by reversing the removal procedure of the part indicated by the selected removal information; A program to achieve this.

[0110] According to the invention described in (((1))), it is possible to provide information for assisting in installing a part back into a device by referring to the part after it has been removed from the device. According to the invention described in (((2))), removal information can be selected from the removed parts and the number of those parts. According to the invention described in (((3))), even if the analysis result of the captured image is incorrect, the user can be made to correct the identification information of the parts removed from the device and the number of the parts. According to the invention described in (((4))), it is possible to allow the user to select the removal information to be selected. According to the invention described in (((5))), it is possible to allow the user to select the removal information to be selected. According to the invention described in (((6))), it is possible to present to a user the order in which each part removed from a device should be attached to the device. According to the invention described in (((7))), the installation order can be linked and displayed for each part currently displayed on the terminal device. According to the invention described in (((8))), the order in which parts are to be attached can be added to the images of the parts included in the acquired photographed images, and the order can be presented as attachment procedure information. According to the invention described in (((9))), the next part to be installed in the device can be presented to the user. According to the invention described in (((10))), it is possible to present to a user the installation position of the next part to be installed in the device. According to the invention described in (((11))), it is possible to provide information for assisting in installing a part back into a device by referring to the part after it has been removed from the device. [Explanation of symbols]

[0111] 2 Equipment, 4 Parts, 10 Mobile terminal, 11 Photography unit, 12 User interface (UI) unit, 13 Photographed image transmission unit, 14 Response processing unit, 20 Server, 21 Photographed image receiving unit, 22 Work target list creation unit, 23 Work procedure manual identification unit, 24 Procedure information provision unit, 25 Work procedure manual storage unit, 122 Installation procedure presentation unit, 231 Inquiry processing unit.

Claims

1. A processor is provided. The processor, Acquire photographs of the parts removed from the equipment, selecting removal information corresponding to the part removed from the device from removal information stored in a storage means, the removal information indicating a procedure for removing the part from the device, using the captured image; presenting installation procedure information indicating an installation procedure for the set device by reversing the removal procedure of the part indicated by the selected removal information; An information processing system comprising:

2. The processor, creating part information including identification information of the photographed parts and the number of the parts identified by analyzing the photographed image; selecting, from among the removal information stored in the storage means, removal information in which identification information of parts to be removed and the number of said parts match identification information of parts and the number of said parts included in the part information, as removal information corresponding to the parts removed from the device; 2. The information processing system according to claim 1 .

3. The processor, presenting the part information to a user in a modifiable manner; When the presented part information is modified by a user, the part information is updated with the modified content.

3. The information processing system according to claim 2.

4. The processor, if there is no removal information stored in the storage means that matches the identification information of the parts and the number of the parts included in the part information, selecting, from the removal information stored in the storage means, removal information in which the identification information of the parts to be removed and the number of the parts specified in the removal information are similar to the identification information of the parts and the number of the parts included in the part information, as a candidate for removal information corresponding to the parts removed from the device; presenting the selected candidates of removal information to a user and allowing the user to select removal information corresponding to the part removed from the device; 3. The information processing system according to claim 2.

5. The information processing system according to claim 2, characterized in that when there is a plurality of pieces of removal information stored in the memory means that match the part identification information and number of parts included in the part information, the processor presents the plurality of pieces of removal information to a user as selection candidates and allows the user to select the removal information corresponding to the parts removed from the equipment.

6. The information processing system according to claim 1 , wherein the processor displays, as the installation procedure information, images of parts removed from the device, associated with an order in which the parts are to be installed in the device.

7. The information processing system according to claim 6, characterized in that, when the terminal device is displaying parts that have been removed from the equipment, the processor displays, as the installation procedure information, each part displayed by the terminal device linked to the order in which the parts are to be installed on the equipment.

8. The processor, adding an order in which the parts are to be attached to the device to the images of the parts included in the acquired photographed images; presenting the photographed images after adding the installation order to the images of the components as the installation procedure information; 7. The information processing system according to claim 6.

9. 2. The information processing system according to claim 1, wherein the processor displays, as the installation procedure information, a part to be next installed in the device from among parts removed from the device in a manner that allows the part to be specified.

10. The information processing system according to claim 1, wherein the processor displays, as the installation procedure information, the next part to be installed on the device from among the parts removed from the device and the installation position of the part on the device.

11. On the computer, Ability to capture images of parts removed from equipment; a function of selecting, using the captured image, removal information corresponding to the part removed from the device from removal information stored in a storage means, the removal information indicating a procedure for removing the part from the device; a function of presenting installation procedure information indicating an installation procedure for the set device by reversing the removal procedure of the part indicated by the selected removal information; A program to achieve this.

Images