Mobile terminal device and maintenance support program

Abstract

Provided is a portable terminal device or the like that prevents incorrect operations during maintenance and inspection work of equipment in a plant. 【Solution means】The portable terminal device includes an imaging unit 10 that images equipment and parts, a display unit 20 that displays imaging information, a storage unit 30 that stores operation manual information including equipment identification information and part identification information, a learning model 42 in which the shape and part identification information of parts have been learned, an image recognition unit 41 that recognizes part identification information described on or near parts of the equipment to be operated, an object identification unit 43 that identifies the equipment to be operated based on the equipment identification information recognized by the image recognition unit and the name of the equipment to be operated registered in the operation manual information, and also identifies the part to be operated based on the name of the part to be operated and the result of determining the shape of the part in the equipment to be operated imaged by the imaging unit using the learning model, and a display control unit 44 that at least displays the identification result by the object identification unit on the display unit.

Description

Technical Field

[0001] The present invention relates to a portable terminal device and the like for preventing misoperations in maintenance and inspection work of equipment in a plant.

Background Art

[0002] For example, substation equipment such as that in a power station is required to operate continuously and stably. Therefore, in order to maintain a highly reliable state, the equipment such as substation equipment is regularly maintained and inspected by maintenance personnel (operators).

[0003] In recent years, due to the sophistication of substation equipment and the like, the types of equipment to be handled have increased, and the work has become more complex. Therefore, it has become important to prevent human errors such as misrecognition of equipment and parts to be operated by maintenance personnel. Therefore, at present, when performing work, two or more operators work together, and when one operator performs work, the other monitors, or the other certifies the fingertip confirmation of one operator, and the work is carried out under such regulations to prevent errors as much as possible. However, increasing the number of operators or being too careful in work to prevent human errors will reduce work efficiency. From the above, in maintenance and inspection work, a support system that can improve work efficiency while preventing human errors by operators is desired.

[0004] For example, in a conventional operation and maintenance system, the operation instruction means identifies the inspection target part by relatively obtaining the position of the inspection target part based on the two-dimensional coordinates from the position of the identifier attached to the inspection target equipment recognized by the equipment identification means, and for the identified inspection target part, recognizes the character displayed on the inspection target part of the inspection target equipment photographed by the photographing unit as the name of the inspection target part on which the operation is to be performed, and based on the recognized name of the inspection target part and the name of the inspection target part shown in the operation procedure manual, determines whether the inspection target part on which the operation is to be performed is the correct inspection target part shown in the operation procedure manual (see, for example, Patent Document 1).

Prior Art Documents

Patent Document

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, in the operation and maintenance system disclosed in the above Patent Document 1, since the position of the inspection target part is relatively obtained based on the two-dimensional coordinates from the position of the identifier attached to the inspection target device, it is necessary to register in advance the positions of all inspection target parts with respect to the identifier, which requires a huge amount of working time and is not efficient.

[0007] The present invention has been made to solve the above problems, and it is an object of the present invention to provide a portable terminal device and a program that do not require prior registration of the arrangement of parts mounted on equipment in a plant, improve the efficiency of equipment maintenance and inspection work, and prevent misrecognition of parts by an operator.

Means for Solving the Problems

[0008] The mobile terminal device according to the present invention includes imaging means for imaging devices and components, display means for displaying imaging information including still images and moving images imaged by the imaging means, storage means for storing operation manual information of an operation target device including device identification information for identifying the device and component identification information of components mounted on the device, a learning model in which the shape of the component and the component identification information have been learned, image recognition means for recognizing the device identification information based on the imaging information and recognizing the component identification information described on or near the component of the operation target device, based on the device identification information recognized by the image recognition means and the name of the device that is the operation target registered in the operation manual information, identifying the operation target device, and based on the component identification information recognized by the image recognition means, the name of the operation target component that is the operation target registered in the operation manual information, and the result of determining the shape of the component in the operation target device imaged by the imaging means using the learning model, target identification means for identifying the operation target component, and display control means for at least displaying the identification result by the target identification means on the display means.

[0009] Thus, in the present invention, the operation target component to be operated is identified by the target identification means based on the component identification information recognized by the image recognition means, the name of the operation target component that is the operation target registered in the operation manual information, and the result of determining the shape of the component in the operation target device imaged by the imaging means using the learning model. Therefore, the operation target component can be identified only from the name and shape of the component, and it is not necessary to register the arrangement of components for each device in advance, so that the introduction cost can be suppressed. In addition, since it is possible to perform operations and confirmations in the work alone, it is possible to prevent human errors without increasing the number of personnel.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Embodiments for Carrying Out the Invention

[0011] (The First Embodiment of the Present Invention) Hereinafter, a mobile terminal device and an operation support program according to the first embodiment of the present invention will be described with reference to FIGS. 1 to 5. The mobile terminal device according to the present embodiment realizes a function of preventing incorrect operations due to human errors, such as when performing maintenance or repair on equipment installed in, for example, a power plant or a substation.

[0012] FIG. 1 is a diagram showing the concept of an operation support system using the mobile terminal device according to the present embodiment. The mobile terminal device 10 according to the present embodiment images a plurality of devices X (X1, X2, ···, Xn) installed in, for example, a power plant or a substation and parts Y (Y1, Y2, ···, Yn) mounted on the device X, and determines whether the device X or the part Y to be operated is legitimate according to an operation manual, and displays the determination result using AR (augmented reality) technology. An operator proceeds with the operation while checking the determination result, thereby eliminating the occurrence of incorrect operations.

[0013] As shown in FIG. 1, the mobile terminal device 10 is, for example, a tablet-type mobile terminal. An operator refers to the operation manual information displayed on the display of the mobile terminal device 10, identifies the operation target device X1 and the operation target part Y1 corresponding to the operation procedure to be performed next, and overlays an object representing the identification result on the video captured by the camera and displays it on the display. The operator surely executes the operation according to the operation manual information while checking the object representing the identification result displayed on the display.

[0014] In the present embodiment, as shown in FIG. 1, the case where a tablet-type mobile terminal is used as the mobile terminal device 10 will be described. However, a smartphone, a notebook computer, a Google-type wearable terminal, or the like may be used.

[0015] FIG. 2 is a functional block diagram showing the configuration of the mobile terminal device according to the present embodiment. In FIG. 2, the mobile terminal device 10 includes an imaging unit 20 that images at least an operation target including device X and component Y with a camera or the like, and a storage unit 30 having an operation-related information storage unit 31 that stores operation manual information in which work procedures for maintenance and repair are registered. The control unit 40 uses the imaging information captured by the imaging unit 20 and the operation-related information stored in the storage unit 30 to identify operation target devices X1, operation target components Y1, etc., and execute other processes. The display unit 50 includes a display that displays the video captured by the imaging unit 20 and displays the calculation results of the control unit 40 and the like. Further, although details will be described later, the learning data for the learning model 42 in the control unit 40 to perform machine learning is stored in an external storage device as the learning data unit 60.

[0016] Note that the processing of the control unit 40 in FIG. 2 is realized by the operation support program causing the CPU of the computer to function as each processing unit. Further, the storage unit 30 is a data unit stored in the memory, hard disk, etc. of the computer.

[0017] The imaging unit 20 images at least device X and component Y mounted on the device X according to the operation of the operator. More specifically, the operator referring to the operation manual information images device identification information including the device name described on the nameplate of device X (the plate-like body on which the device name is described above the front of device X in FIG. 1), and component identification information including the component name described on or near component Y.

[0018] Note that the device identification information may be the device name described on the nameplate as described above, or identification information such as a barcode or QR code (registered trademark) attached to the device X. Similarly, the component identification information may be identification information such as a barcode or QR code (registered trademark) attached in the vicinity of component Y.

[0019] Further, the imaging unit 20 may be provided integrally with the mobile terminal device 10, or may be detachably provided with the mobile terminal device 10. When the imaging unit 20 is provided separately from the mobile terminal device 10, the imaging information captured by the imaging unit 20 may be configured to be transmitted to the mobile terminal device 10 by the communication function.

[0020] The control unit 40 includes at least an image recognition unit 41, a learning model 42, an object identification unit 43, and a display control unit 44. The image recognition unit 41 performs image processing on the imaging information including the still image and the moving image captured by the imaging unit 20, and recognizes device identification information and component identification information. As described above, this image recognition unit 41 includes a character recognition unit 45 that recognizes the character string of the device name described on the nameplate of device X or the character string of the component name described in the vicinity of component Y or the like. The character recognition unit 45 can recognize characters by, for example, OCR (Optical Character Recognition).

[0021] The learning model 42 is a learning model that identifies the name of the component Y from the shape of the component Y included in the imaging information captured by the imaging unit 20. As described above, this learning model 42 is obtained by machine learning (for example, deep learning, etc.) of the learning data stored in the learning data unit 60. The learning data stored in the learning data unit 60 is, for example, the shape image information 61 of the component Y captured from various angles and positions and the name information 62 of the component Y. By performing machine learning on these, it becomes possible to generate a learning model 42 that identifies the component Y included in the imaging information based on the imaging information input from the imaging unit 20.

[0022] The target identification unit 43 determines whether the device X and component Y included in the imaging information are the operation target device X1 and operation target component Y1 according to the operation manual information, from the device identification information and component identification information recognized by the image recognition unit 41, the component Y identified by the learning model 42, and the operation manual information stored in the operation-related information storage unit 31 of the storage unit 30. Specifically, in the operation manual information, it is determined whether the name of the operation target device X1 registered in the procedure to be performed matches the device identification information recognized by the image recognition unit 41 (the device name of the device X described on the captured nameplate). If they match, it is determined that the captured device X is the operation target device X1.

[0023] Also, when the operation target device X1 is identified, it is determined whether the name of the operation target component Y1 registered in the procedure of the operation manual information matches the component identification information recognized by the image recognition unit 41 (the name of the component Y described on the captured component Y or in its vicinity). At the same time, it is also determined whether the name of the operation target component Y1 in the operation manual information matches the name of the component Y identified by the learning model 42. If both match, it is determined that the captured component Y is the operation target component Y1.

[0024] The display control unit 44 acquires the operation manual information stored in the operation-related information storage unit 31 and displays it on the display unit 50. FIG. 3 is a diagram showing an example of the display mode when the operation manual information is displayed in the mobile terminal device according to the present embodiment. In FIG. 3, an operation menu M showing the operation procedures (No.), names (operation targets), contents (operation contents), states, etc. of the operation target device X1 and operation target component Y1 is displayed on the display unit 50. The operator executes the operations according to the operation contents described in the operation contents while identifying the operation target device X1 and operation target component Y1 according to the procedures of the operation menu M displayed on the display unit 20.

[0025] Note that the operation manual information may be displayed as the operation menu M only for the procedures of the operations to be performed hereafter and the procedures of the operations currently being performed, or only the procedures of the operations to be performed hereafter and the procedures of the operations currently being performed among a series of procedures may be displayed in a manner different from other procedures.

[0026] Further, the display control unit 44 generates image information such as a virtual object representing the result of the target identification unit 43 identifying the operation target device X1 or the operation target component Y1, and superimposes and displays it on the imaging information on the display unit 50. FIG. 4 is a diagram showing an example of a display mode when the operation target component Y1 is specified in the mobile terminal device according to the present embodiment. As shown in FIG. 4, when the target identification unit 43 identifies the operation target component Y1, a virtual object 70 representing a component area including at least the outer edge of the shape of the identified operation target component Y1 is generated, and the virtual object 70 is displayed at the position of the corresponding operation target component Y1. In addition, for example, when the operation target device X1 is specified and a plurality of devices X are displayed on the display unit 50, a virtual object (not shown) representing a device area including at least the outer edge of the shape of the operation target device X1 may be generated and the virtual object may be displayed at the position of the corresponding operation target device X1.

[0027] The storage unit 30 has an operation-related information storage unit 31 in which the operation manual information described above is stored. This operation manual information may be downloaded in advance only the operation manual information necessary for the maintenance and repair to be performed hereafter from a server device, cloud, etc. and stored in the operation-related information storage unit 31, or all the operation manual information may be stored in the mobile terminal device 10.

[0028] Next, the operation of the mobile terminal device according to the present embodiment will be described. FIG. 5 is a flowchart showing the operation of the mobile terminal device according to the present embodiment. First, as a preliminary preparation, the display control unit 44 reads out operation manual information as shown in FIG. 3 from the operation-related information storage unit 31 and displays it on the display unit 50. Thereafter, the operator will identify and operate the operation target device X1 and the operation target part Y1 according to the procedure of the operation manual information displayed on the display unit 50.

[0029] Hereinafter, for example, a case where the device name of the operation target device X1 is "Maintenance Panel A" and the part name of the operation target part Y1 is "8R" will be described as a specific example. First, the imaging unit 10 images the nameplate of the device X as shown in FIG. 1 (S1). The character recognition unit 45 of the image recognition unit 41 recognizes the character string "Maintenance Panel A" described on the nameplate included in the imaging information (S2). The target identification unit 43 determines whether the device name recognized by the character recognition unit 45 matches the device name described in the procedure of the operation manual information acquired from the operation-related information storage unit 31 (S3). If they do not match, S1 to S3 are repeated until the device name of the legitimate operation target device X1 along the operation manual information is recognized in the imaging information.

[0030] If they match in S3, assuming that the legitimate operation target device X1 along the operation manual information is recognized, the process proceeds to the identification of the following operation target part Y1. The imaging unit 10 images the part Y mounted on the operation target device X1 (S4). The character recognition unit 45 recognizes the part name "8R", which is the part identification information described on the part Y or in its vicinity, by image processing (S5). The learning model 42 identifies the type and name of the part Y from the shape of the part Y included in the imaging information (S6).

[0031] The target identification unit 43 determines whether the part name recognized by the character recognition unit 45 matches the operation target part Y1 registered in the operation manual information, and whether the part Y identified by the learning model 41 from the shape of the part Y based on the imaging information matches the part Y that is the operation target registered in the operation manual information (S7). If they do not match, steps S4 to S6 are repeatedly executed until the name and shape of the legitimate operation target part Y1 in accordance with the operation manual information are recognized. If they match in S7, it is determined that the imaged part Y is the legitimate operation target part Y1 in accordance with the operation manual information (S8), and the identification of the operation target part Y1 is terminated.

[0032] By the above processing, when the target identification unit 43 identifies the operation target part Y1, for example, a virtual object 70 as shown in FIG. 4 is displayed. The operator confirms that the operation target part Y1 is identified by the virtual object 70, and operates the "8R" of the operation target part Y1 to the "off" state according to the work content described in the menu M. Then, the operator presses the operation completion button as shown in FIG. 4 to end the work on the operation target device X1, "Maintenance Panel A". Note that instead of the operation completion button, the operation may be completed by the operator's utterance of "operation completed" through voice recognition.

[0033] As described above, in the mobile terminal device 10 according to the present embodiment, the target identification unit 43 identifies the operation target part Y1 based on the part identification information recognized by the image recognition unit 42 and the name of the operation target part Y1 registered in the operation manual information, and the result of determining the shape of the part Y in the operation target device X1 imaged by the imaging unit 10 using the learning model 41. Therefore, the operation target part Y1 can be identified only from the name and shape of the part Y, and it is not necessary to register in detail the arrangement of the part Y for each device X in advance, and the introduction cost can be significantly reduced.

[0034] In addition, when using a tablet or smartphone as the mobile terminal device 2, there is no need to wear it on the head like the goggle type. Therefore, maintenance and repair work can be carried out without feeling discomfort caused by wearing heavy objects. Also, since the mobile terminal device 10 only needs to be directed towards the equipment X or the component Y when necessary, the work can be carried out while directly visually recognizing with one's own eyes, and the work can proceed smoothly.

[0035] (Second Embodiment of the Present Invention) The mobile terminal device and the operation support program according to the second embodiment of the present invention will be described with reference to FIGS. 6 and 7. The mobile terminal device 10 according to the present embodiment calculates and manages the position information in the three-dimensional virtual space so that only the component Y mounted on the work target device X1 becomes the processing target when specifying the work target component Y1. Note that the description overlapping with the first embodiment in this embodiment is omitted.

[0036] FIG. 6 is a functional block diagram showing the configuration of the mobile terminal device according to the present embodiment. In FIG. 6, the difference from FIG. 2 in the first embodiment is that it includes a position specifying unit 46 that specifies the spatial coordinates in the three-dimensional virtual space of the operation target device X1 specified by the target specifying unit 43.

[0037] The position specifying unit 46 specifies the spatial coordinates of the operation target device X1 in the three-dimensional virtual space specified by the target specifying unit 43. Specifically, for example, when the power of the mobile terminal device 10 is turned "ON", the position of the mobile terminal device 10 is set as the origin in the three-dimensional virtual space. Then, as the operator moves and the mobile terminal device 10 moves, and when the nameplate of the device X is imaged according to the operation menu M and the device X is recognized as the operation target device X1, for example, based on an external signal (not shown) obtained by an acceleration sensor mounted on the mobile terminal device 10 or a focus function of the imaging unit 10, the relative positional relationship between the mobile terminal device 10 and the operation target device X1 in the real space is obtained, and the spatial coordinates of the operation target device X1 in the three-dimensional virtual space are calculated. In the operation-related information storage unit 31, the shape, size, etc. of each device X are stored, and the spatial region of the entire operation target device X1 in the three-dimensional virtual space is specified.

[0038] Further, the image recognition unit 41 subjects only the component Y imaged by the imaging unit 10 within the spatial region in the three-dimensional virtual space of the operation target device X1 specified by the position specifying unit 46 to image processing, and recognizes the component identification information. That is, by not subjecting the imaging information outside the spatial region occupied by the operation target device X1 to image processing, it is possible to reduce the processing load.

[0039] Next, the operation of the mobile terminal device according to the present embodiment will be described. FIG. 7 is a flowchart showing the operation of the mobile terminal device according to the present embodiment. The processes from S1 to S3 are the same as those in the case of FIG. 5 in the first embodiment. When they match in S3, the position specifying unit 46 calculates the coordinates of the spatial region of the operation target device X1 in the three-dimensional virtual space (S4). Then, the imaging unit 10 images an arbitrary component Y (S5). It is determined whether the position of the imaged component Y in the three-dimensional virtual space is within the spatial region of the operation target device X1 calculated in S4 (S6). If it is outside the spatial region of the operation target device X1, the process returns to S5 and an arbitrary component Y is imaged again until a component within the spatial region of the operation target device X1 is imaged.

[0040] If it is within the spatial region of the target device X1 during operation at S6, the processes after S7 are executed. Since the processes from S7 to S10 are the same as the processes from S5 to S8 in FIG. 5, the description thereof is omitted.

[0041] As described above, in the mobile terminal device 10 according to the present embodiment, only the component Y included in the spatial region in the three-dimensional virtual space of the target device X1 specified by the position specifying unit 46 is targeted for image processing. Therefore, even when there are a plurality of target devices X1 having the same type and shape, only the component Y mounted on the target device X1 becomes the processing target, and it is possible to surely prevent an incorrect operation by the operator.

[0042] (Third Embodiment of the Present Invention) The mobile terminal device and the operation support program according to the third embodiment of the present invention will be described with reference to FIGS. 8 to 10. After the target device X1 and the target component Y1 are specified in the mobile terminal device according to the present embodiment, immediately before the operator actually performs an operation, the target device X1 and the target component Y1 are confirmed with the fingertips, and it is determined whether this fingertip confirmation is correct. Further, after an operation is executed on the target component Y1, the operator confirms the component Y that has actually been operated with the fingertips, and determines whether this fingertip confirmation is correct. Note that descriptions overlapping with the above-described embodiments are omitted in the present embodiment.

[0043] FIG. 8 is a functional block diagram showing the configuration of the mobile terminal device according to the present embodiment. The difference from FIG. 2 in the first embodiment in FIG. 8 is that the image recognition unit 41 includes a gesture recognition unit 47 that recognizes a gesture including the body shape (hereinafter referred to as fingertips) of the operator, and based on the recognized gesture information, determines whether the target component Y1 has been confirmed with the fingertips, that is, whether the operator recognizes the target component Y1.

[0044] The gesture recognition unit 47 performs image processing on the imaging information captured by the imaging unit 10 and recognizes the fingertip region of the operator. The processing of the gesture recognition unit 47 can utilize a conventionally known image recognition technique.

[0045] When at least a part of the outer edge of the fingertip area recognized by the gesture recognition unit 47 is within or near (within a predetermined distance) the virtual object 70 as shown in FIG. 4, the gesture determination unit 48 determines that the operator is recognizing the legitimate operation target part Y1.

[0046] FIG. 9 is a diagram showing an example of a display when gesture determination processing is performed in the mobile terminal device according to the present embodiment. As shown in FIG. 9, the fingertip of the operator is recognized by the gesture recognition unit 47, and its outer edge (shown by a thick dotted line in FIG. 9) is specified. When a part of this outer edge overlaps the virtual object 70 representing the operation target part Y1 or is within a predetermined distance, the gesture determination unit 48 determines that the operator has surely recognized the operation target part Y1. The determination result can be made explicit, for example, by making a blinking display when it can be determined that the recognition is surely made, and not making a blinking display (for example, when the fingertip is separated from the virtual object 70 by a predetermined distance or more) when it is determined that the recognition is not made, so that the display content and display mode are made different for recognition / non-recognition. The operator proceeds to the next operation after confirming such a determination result.

[0047] Note that the above gesture recognition unit 47 and gesture determination unit 48 are executed in the following two cases. The first case is, as described above, the confirmation time immediately before the operator actually executes an operation on the operation target part Y1 when the operation target part Y1 of the operation target device X1 is specified by the mobile terminal device 10. That is, after the operation target device X1 and the operation target part Y1 are specified by the mobile terminal device 10, when the operator actually performs an operation on the operation target part Y1, the mobile terminal device 10 is once placed horizontally or the like, and the line of sight of the operator is cut off from the display unit 10. In order to eliminate the possibility that the operator's recognition deviates from the operation target part Y1 during that time, the above finger-pointing confirmation is performed immediately before the operation. When it can be determined that the operator is recognizing the legitimate operation target part Y1, this fact is displayed on the display unit 10.

[0048] Also, the second case is when, after the operator actually performs an operation on the component Y1 to be operated on, it is checked whether the component Y that was operated on is the legitimate component Y1 to be worked on. That is, in the unlikely event that the component Y operated on by the operator is not the component Y1 to be worked on, it becomes possible to minimize damage by taking prompt action. That is, by identifying the component Y actually operated on immediately after the operation on the component Y by fingertip confirmation, it can be determined whether an operation has been performed on the legitimate component Y1 to be operated on, and the next procedure can be proceeded with with confidence.

[0049] Next, the operation of the mobile terminal device according to the present embodiment will be described. FIG. 10 is a flowchart showing the operation of the mobile terminal device according to the present embodiment. In FIG. 10, the processes from S1 to S8 are the same as those in the case of FIG. 5 in the first embodiment. In S8, when the component Y1 to be operated on is recognized, the display control unit 44 displays a virtual object 70 representing a component area including at least the outer edge of the shape of the component Y1 to be operated on on the display unit 50 (S9). The gesture recognition unit 47 recognizes the outer edge of the fingertip area of the operator based on the imaging information captured by the imaging unit 10 (S10).

[0050] The gesture determination unit 48 determines whether at least a part of the outer edge of the fingertip area of the operator is within or near the virtual object 70 displayed in S9 (S11). If at least a part of the outer edge of the fingertip area is not within or near the virtual object 80, the process of S10 is repeated until the component Y1 to be operated on is confirmed by fingertip confirmation. If at least a part of the outer edge of the fingertip area is within or near the virtual object 80, it is assumed that the operator can recognize the legitimate component Y1 to be operated on, and this fact is displayed on the display unit 50 (S12), and the process ends.

[0051] In addition, in the case of the above-described second case, immediately after the operator actually operates on the operation target part Y1, the same processing as that from S9 to S12 is executed. However, in S11, if at least a part of the outer edge of the fingertip area is not within or near the virtual object 80, a warning is displayed on the display unit 10 indicating that the operated part Y may not be the operation target part Y1, and the process is terminated.

[0052] Also, in FIG. 9 above, the fingertip confirmation is such that the back of the hand is imaged. However, fingertip confirmation may be performed so that the fingerprints and palm prints of the operator can be imaged by the mobile terminal device 10, and the control unit 40 of the mobile terminal device 10 may have a function of performing fingerprint and palm print authentication. That is, by authenticating that the person performing the fingertip confirmation is undoubtedly the responsible operator and then performing the determination of the fingertip confirmation, more accurate operations can be realized, and it is also possible to clarify the location of responsibility in case of any problems.

[0053] As described above, in the mobile terminal device 10 according to the present embodiment, when at least a part of the outer edge of the operator's body is within or near the virtual object 70 based on the gesture information, the gesture determination unit 48 determines that the operator recognizes the legitimate operation target part Y1, and completes the specific process of the operation target part Y1. Therefore, in addition to specifying the operation target part Y1 by the mobile terminal device 2, the operator's recognition is involved, and it is possible to more reliably prevent an incorrect operation of the operation target part Y1.

[0054] In addition, for the imaging information after the operation, when at least a part of the outer edge of the operator's body is within or near the virtual object 70, the gesture determination unit 48 determines that the operator has operated the legitimate operation target part Y1, and if not, it is displayed that there is a possibility that the wrong part Y has been operated, so that the operator can immediately grasp the possibility of an incorrect operation of the part Y that has been operated.

[0055] In addition, as another embodiment, the storage unit 30 may store captured images before and after the operation of the component Y1 to be operated by the operator, compare the captured images before and after the operation of the component Y1 to be operated, and include an operation determination unit that determines whether the procedure based on the operation manual information of the component Y1 to be operated has been completed. That is, by comparing the images before and after the operation of the component Y1 to be operated, if there is no difference, it can be determined that the operation has not been performed, and if there is a difference, it can be determined that the operation has been performed. Also, by combining with the above-described finger-pointing confirmation process, it is possible to compare the images before and after the operation for the image in the finger-pointing confirmation state and determine whether there is a deviation in the operator's recognition before and after the operation.

Explanation of Signs

[0056] 10 Portable terminal device 20 Imaging unit 30 Storage unit 31 Operation-related information storage unit 40 Control unit 41 Image recognition unit 42 Learning model 43 Target identification unit 44 Display control unit 45 Character recognition unit 46 Position identification unit 47 Gesture recognition unit 48 Gesture determination unit 50 Display unit 60 Learning data unit 61 Shape image information 62 Name information 70 Virtual object

Claims

1. An imaging means for imaging the device and the component; a display means for displaying image information including still images and moving images captured by the imaging means; a storage means for storing operation manual information of an operation target device to be operated, the operation manual information including device identification information for identifying the device and part identification information of parts mounted on the device; a learning model that has learned the shape of the part and the part identification information; an image recognition means for recognizing the device identification information based on the imaging information and recognizing the part identification information written on or near a part of the device to be operated; a target specifying means for specifying the target device based on the device identification information recognized by the image recognition means and the name of the device to be operated that is registered in the operation manual information, and for specifying the target part to be operated based on the part identification information recognized by the image recognition means and the name of the target part to be operated that is registered in the operation manual information, and a result of determining, using a learning model, the shape of the part in the target device imaged by the imaging means; a position identification means for identifying a position coordinate in a three-dimensional virtual space of the device to be operated identified by the target identification means; a display control means for at least displaying a result of the identification by the target identification means on the display means, The image recognition means recognizes only the part mounted on the device imaged by the imaging means when position coordinates in the three-dimensional virtual space of the device imaged by the imaging means match position coordinates in the three-dimensional virtual space of the device to be operated. A mobile terminal device comprising:

2. An imaging means for imaging devices and components; a display means for displaying image information including still images and moving images captured by the imaging means; a storage means for storing operation manual information of an operation target device to be operated, the operation manual information including device identification information for identifying the device and part identification information of parts mounted on the device; a learning model that has learned the shape of the part and the part identification information; an image recognition means for recognizing the device identification information based on the imaging information and recognizing the part identification information written on or near a part of the device to be operated; a target specifying means for specifying the target device based on the device identification information recognized by the image recognition means and the name of the device to be operated that is registered in the operation manual information, and for specifying the target part to be operated based on the part identification information recognized by the image recognition means and the name of the target part to be operated that is registered in the operation manual information, and a result of determining, using a learning model, the shape of the part in the target device imaged by the imaging means; a display control means for at least displaying a result of the identification by the target identification means on the display means, the display control means displays a virtual object representing a part area including at least an outer edge of a shape of the operation target part identified by the target identification means; the image recognition means acquires gesture information including a body shape of the operator based on the imaging information; and a completion confirmation means for determining that the operator has properly identified the operation target part and completing the identification of the operation target part when at least a part of the outer periphery of the body of the operator is within or near the virtual object based on the gesture information. A mobile terminal device comprising:

3. 3. The mobile terminal device according to claim 2, When at least a part of an outer periphery of the body of the operator is within or near the virtual object with respect to image information captured after the operator operates the operation target part, the completion confirmation means determines that the operator has legitimately operated the operation target part and completes the operation based on the operation manual information. A mobile terminal device comprising:

4. 3. The mobile terminal device according to claim 1, the storage means stores image information before and after an operation of the operation target part by an operator; The apparatus further comprises an operation determination unit that determines whether or not an operation based on the operation manual information of the operation target part has been completed based on image information before and after the operation of the operation target part. A mobile terminal device comprising:

5. A learning model that has learned the shapes of parts mounted on a device and part identification information that identifies the parts; an image recognition means for recognizing device identification information for identifying the device based on image information captured by an imaging means, and for recognizing the part identification information written on or near a part of the device to be operated; a target identification means for identifying the target device based on the device identification information recognized by the image recognition means, and a name of the device to be operated that is registered in operation manual information for the target device, the name including the device identification information for identifying the device stored in a storage means and the part identification information of a part mounted on the device, and for identifying the part based on the part identification information recognized by the image recognition means and the name of the target part that is to be operated that is registered in the operation manual information, and a result of determining, using a learning model, the shape of the part in the target device imaged by the imaging means; a position identification means for identifying a position coordinate in a three-dimensional virtual space of the operation target device identified by the target identification means; causing a computer to function as a display control means for at least displaying a result of the identification by the target identification means on a display means; The image recognition means recognizes only the part mounted on the device imaged by the imaging means when position coordinates in the three-dimensional virtual space of the device imaged by the imaging means match position coordinates in the three-dimensional virtual space of the device to be operated. A maintenance support program that features:

6. A learning model that has learned the shapes of parts mounted on the equipment and part identification information that identifies the parts; an image recognition means for recognizing device identification information for identifying the device based on image information captured by an imaging means, and for recognizing the part identification information written on or near a part of the device to be operated; a target identification means for identifying the target device based on the device identification information recognized by the image recognition means, and a name of the device to be operated that is registered in operation manual information for the target device, the name including the device identification information for identifying the device stored in a storage means and the part identification information of a part mounted on the device, and for identifying the part based on the part identification information recognized by the image recognition means and the name of the target part that is to be operated that is registered in the operation manual information, and a result of determining, using a learning model, the shape of the part in the target device imaged by the imaging means; causing a computer to function as a display control means for at least displaying a result of the identification by the target identification means on a display means; the display control means displays a virtual object representing a part area including at least an outer edge of a shape of the operation target part identified by the target identification means; the image recognition means acquires gesture information including a body shape of the operator based on the imaging information; and causing a computer to function as a completion confirmation unit that completes the identification of the operation target part by determining that the operator has properly identified the operation target part when at least a part of the outer periphery of the body of the operator is within or near the virtual object based on the gesture information. A maintenance support program that features:

Images