Mobile terminal device and maintenance support program

Abstract

To provide a mobile terminal device or the like that prevents erroneous operation during maintenance and inspection work on equipment in plants.SOLUTION: The mobile terminal device includes: an imaging unit 10 that images equipment and components; a display unit 20 that displays the image information; a memory unit 30 that stores operation manual information including equipment identification information and component identification information; a learning model 42 that has learned the shapes and component identification information of components; an image recognition unit 41 that recognizes component identification information written on components of the equipment to be operated or in the vicinity thereof; a target identification unit 43 that identifies the equipment to be operated based on the equipment identification information recognized by the image recognition unit and the name of the equipment to be operated that is registered in the operation manual information, and identifies the component to be operated based on the name of the component to be operated and the result of determining, using the learning model, the shape of the component in the equipment to be operated that has been imaged by the imaging unit; and a display control unit 44 that at least displays the identification result by the target identification unit on the display unit.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a mobile terminal device or the like that prevents erroneous operation during maintenance and inspection work of equipment in a plant. [Background technology]

[0002] For example, substation equipment at power stations is required to operate continuously and stably. Therefore, to maintain high reliability, maintenance personnel (workers) regularly inspect and maintain substation equipment and other devices.

[0003] In recent years, the sophistication of substation equipment and other equipment has led to an increase in the variety of equipment handled and the complexity of maintenance work. Therefore, it is important to prevent human error, such as maintenance personnel misidentifying the equipment or parts they are operating. Currently, measures are taken to minimize errors by having two or more workers accompany each worker during work, with one person monitoring the other while the other works, or by having the other person confirm the other's fingertip check. However, increasing the number of workers or being overly cautious in their work in an attempt to prevent human error reduces work efficiency. For these reasons, there is a need for a support system that can prevent human error while improving work efficiency in maintenance and inspection work.

[0004] For example, in a conventional operation and maintenance system, an operation instruction means identifies the part to be inspected by relatively determining the position of the part to be inspected based on two-dimensional coordinates from the position of an identifier attached to the equipment to be inspected recognized by the equipment identification means, and then recognizes the characters displayed on the part to be inspected of the equipment to be inspected photographed by the photographing unit as the name of the part to be inspected to be operated, and determines whether the part to be inspected to be operated is the correct part to be inspected as indicated in the operation procedure manual based on the recognized name of the part to be inspected and the name of the part to be inspected indicated in the operation procedure manual (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 6884911 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the operation and maintenance system disclosed in Patent Document 1 determines the relative position of the parts to be inspected based on two-dimensional coordinates from the position of an identifier attached to the equipment to be inspected, so it is necessary to register the positions of all parts to be inspected relative to the identifier in advance, which requires a huge amount of work time and is not efficient.

[0007] The present invention has been made to solve the above-mentioned problems, and aims to provide a mobile terminal device and program that eliminates the need to register in advance the location of parts installed on equipment in a plant, improves the efficiency of equipment maintenance and inspection work, and prevents operators from misidentifying parts. [Means for solving the problem]

[0008] a learning model that has learned the shape of the part and the part identification information; an image recognition means that recognizes the device identification information based on the imaging information and recognizes the part identification information written on or near the part of the device to be operated; an object identification means that identifies the device to be operated based on the device identification information recognized by the image recognition means and the name of the device to be operated registered in the operation manual information, and that identifies the object part to be operated based on the part identification information recognized by the image recognition means and the name of the device to be operated registered in the operation manual information, and based on the part identification information recognized by the image recognition means, the name of the object part to be operated registered in the operation manual information, and a result of determining the shape of the part in the device to be operated imaged by the imaging means using the learning model; and a display control means that displays at least the result of identification by the object identification means on the display means.

[0009] In this way, in the present invention, the target part to be operated is identified by the target identification means based on the part identification information recognized by the image recognition means, the name of the target part to be operated registered in the operation manual information, and the result of determining the shape of the part in the target device imaged by the imaging means using a learning model. Therefore, the target part can be identified based only on the name and shape of the part, eliminating the need to register the layout of parts for each device in advance, thereby reducing implementation costs. Furthermore, since operation and confirmation during work can be performed by one person, human error can be prevented without increasing the number of personnel. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram showing the concept of an operation assistance system using a mobile terminal device according to a first embodiment of the present invention. [Figure 2] 1 is a functional block diagram showing a configuration of a mobile terminal device according to a first embodiment of the present invention. [Figure 3] 3 is a diagram showing an example of a display mode when operation manual information is displayed on the mobile terminal device according to the first embodiment of the present invention. FIG. [Figure 4] 4 is a diagram showing an example of a display mode when an operation target part is specified in the mobile terminal device according to the first embodiment of the present invention. FIG. [Figure 5] 5 is a flowchart showing the operation of the mobile terminal device according to the first embodiment of the present invention. [Figure 6] FIG. 10 is a functional block diagram showing the configuration of a mobile terminal device according to a second embodiment of the present invention. [Figure 7] 10 is a flowchart showing the operation of a mobile terminal device according to a second embodiment of the present invention. [Figure 8] FIG. 10 is a functional block diagram showing a configuration of a mobile terminal device according to a third embodiment of the present invention. [Figure 9] FIG. 11 is a diagram showing an example of a display when a gesture determination process is performed in a mobile terminal device according to a third embodiment of the present invention. [Figure 10] 10 is a flowchart showing the operation of a mobile terminal device according to a third embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] (First embodiment of the present invention) A mobile terminal device and an operation assistance program according to a first embodiment of the present invention will be described below with reference to Figures 1 to 5. The mobile terminal device according to this embodiment realizes a function to prevent erroneous operations due to human error when performing maintenance on equipment installed in, for example, a power plant or a substation.

[0012] 1 is a diagram illustrating the concept of an operation support system using a mobile terminal device according to this embodiment. The mobile terminal device 10 according to this embodiment captures images of multiple devices X (X1, X2, . . . , Xn) installed in, for example, a power plant or a substation, and components Y (Y1, Y2, . . . , Yn) mounted on the devices X, and determines whether the devices X and components Y to be operated are legitimate according to the operation manual, and displays the determination results using AR (augmented reality) technology. A maintenance worker can proceed with the operation while checking the determination results, thereby eliminating the occurrence of operational errors.

[0013] 1, the mobile terminal device 10 is, for example, a tablet-type mobile terminal, and the worker refers to the operation manual information displayed on the display of the mobile terminal device 10, identifies the operation target device X1 or operation target part Y1 that corresponds to the operation procedure to be performed, and an object representing the identification result is displayed on the display superimposed on the image captured by the camera. The worker reliably performs the operation in accordance with the operation manual information while checking the object representing the identification result displayed on the display.

[0014] In this embodiment, as shown in FIG. 1, a tablet-type mobile terminal is used as the mobile terminal device 10, but a smartphone, a notebook computer, a goggle-type wearable terminal, or the like may also be used.

[0015] 2 is a functional block diagram showing the configuration of a mobile terminal device according to this embodiment. In FIG. 2, the mobile terminal device 10 includes an imaging unit 20 that captures images of at least operation targets, including a device X and a component Y, using a camera or the like; a storage unit 30 having an operation-related information storage unit 31 that stores operation manual information, including operation procedures for maintenance and other operations; a control unit 40 that uses the image information captured by the imaging unit 20 and the operation-related information stored in the storage unit 30 to identify the operation target device X1, the operation target component Y1, etc., and perform other processing; and a display unit 50 that displays the images captured by the imaging unit 20 and the results of calculations performed by the control unit 40. Furthermore, as will be described in detail later, learning data used by a learning model 42 in the control unit 40 for machine learning is stored in an external storage device as a learning data unit 60.

[0016] 2 is realized by the operation assistance program causing the CPU of the computer to function as each processing unit. The storage unit 30 is a data unit stored in the memory, hard disk, etc. of the computer.

[0017] The imaging unit 20, in accordance with the operator's operation, images at least the device X and the component Y mounted on the device X. More specifically, the operator, referring to the operation manual information, images the device identification information including the device name and the like written on the nameplate of the device X (a plate-like body on which the device name is written, attached to the upper front surface of the device X in FIG. 1 ) and the component identification information including the component name and the like written on the component Y or in its vicinity.

[0018] The device identification information may be the device name written on the nameplate as described above, or may be identification information such as a barcode or QR code (registered trademark) attached to the device X. Similarly, the part identification information may be identification information such as a barcode or QR code (registered trademark) attached near the part Y.

[0019] Furthermore, the imaging unit 20 may be provided integrally with the mobile terminal device 10, or may be provided detachably from the mobile terminal device 10. When the imaging unit 20 is provided separately from the mobile terminal device 10, imaging information captured by the imaging unit 20 may be configured to be transmitted to the mobile terminal device 10 via a communication function.

[0020] The control unit 40 has at least an image recognition unit 41, a learning model 42, an object identification unit 43, and a display control unit 44. The image recognition unit 41 processes image information including still images and moving images captured by the imaging unit 20, and recognizes device identification information and component identification information. As described above, the image recognition unit 41 has a character recognition unit 45 that recognizes the character string of the device name written on the nameplate of device X and the character string of the component name written on or near component Y. The character recognition unit 45 is capable of recognizing characters, for example, by OCR (Optical Character Recognition).

[0021] The learning model 42 is a learning model that identifies the name of the part Y from the shape of the part Y included in the imaging information captured by the imaging unit 20. As described above, this learning model 42 is obtained by machine learning (e.g., deep learning) of the learning data stored in the learning data unit 60. The learning data stored in the learning data unit 60 is, for example, shape image information 61 obtained by capturing images of the part Y from various angles and positions and name information 62 of the part Y. By machine learning these, it is possible to generate a learning model 42 that identifies the part Y included in the imaging information based on the imaging information input from the imaging unit 20.

[0022] The target identification unit 43 determines whether the device X or part Y included in the captured image information is the device X1 or part Y1 to be operated according to the operation manual information, based on the device identification information and part identification information recognized by the image recognition unit 41, the part Y identified by the learning model 42, and the operation manual information stored in the operation-related information storage unit 31 of the storage unit 30. Specifically, the target identification unit 43 determines whether the name of the device X1 to be operated, which is registered in the operation manual information as the procedure to be performed, matches the device identification information recognized by the image recognition unit 41 (the device name of the device X written on the imaged nameplate), thereby determining that the captured device X is the device X1 to be operated.

[0023] Furthermore, when the operation target device X1 is identified, it is determined whether the name of the operation target part Y1 registered in the procedure of the operation manual information matches the part identification information recognized by the image recognition unit 41 (the name of the imaged part Y or part Y written in its vicinity). At the same time, it is determined whether the name of the operation target part Y1 in the operation manual information matches the name of the part Y identified by the learning model 42. Then, if both match, it is determined that the imaged part Y is the operation target part Y1.

[0024] The display control unit 44 acquires the operation manual information stored in the operation-related information storage unit 31 and displays it on the display unit 50. FIG. 3 is a diagram showing an example of a display mode when the operation manual information is displayed on the mobile terminal device according to this embodiment. In FIG. 3, the display unit 50 displays an operation menu M showing the operation procedure (No.), name (task object), content (task content), status, etc. of the operation target device X1 and the operation target part Y1. The worker follows the procedure in the operation menu M displayed on the display unit 20 to identify the operation target device X1 and the operation target part Y1 and perform the operation in accordance with the work content described in the work content.

[0025] The operation manual information may be configured so that only the procedure for the operation to be performed or the procedure for the operation currently being performed is displayed as the operation menu M, or so that only the procedure for the operation to be performed or the procedure for the operation currently being performed among a series of procedures is displayed in a manner different from the other procedures.

[0026] Furthermore, the display control unit 44 generates image information such as a virtual object representing the result of the target device X1 or the target part Y1 being identified by the target identification unit 43, and displays the image information on the display unit 50 superimposed on the captured image information. FIG. 4 is a diagram showing an example of a display mode when the target device Y1 is identified in the mobile terminal device according to this embodiment. As shown in FIG. 4, when the target device identification unit 43 identifies the target device Y1, a virtual object 70 representing a part area including at least the outer edge of the shape of the identified target device Y1 is generated, and the virtual object 70 is displayed at the position of the corresponding target device Y1. Alternatively, for example, when the target device X1 is identified and multiple devices X are displayed on the display unit 50, a virtual object (not shown) representing a device area including at least the outer edge of the shape of the target device X1 may be generated, and the virtual object may be displayed at the position of the corresponding target device X1.

[0027] As described above, the storage unit 30 has an operation-related information storage unit 31 in which operation manual information is stored. This operation manual information may be stored in the operation-related information storage unit 31 by downloading only the operation manual information required for the maintenance to be performed from a server device, a cloud, or the like before performing the maintenance, or all of the operation manual information may be stored in the mobile terminal device 10.

[0028] Next, the operation of the mobile terminal device according to this embodiment will be described. Fig. 5 is a flowchart showing the operation of the mobile terminal device according to this embodiment. First, as a preliminary step, the display control unit 44 reads out operation manual information such as that shown in Fig. 3 from the operation-related information storage unit 31 and displays it on the display unit 50. Thereafter, the operator follows the procedure in the operation manual information displayed on the display unit 50 to identify and operate the operation target device X1 and operation target part Y1.

[0029] The following describes a specific example in which the device name of the operation target device X1 is "Maintenance Panel A" and the part name of the operation target part Y1 is "8R." First, the imaging unit 10 captures an image of the nameplate of device X as shown in FIG. 1 (S1). The character recognition unit 45 of the image recognition unit 41 recognizes the character string "Maintenance Panel A" written on the nameplate included in the image information (S2). The target identification unit 43 determines whether the device name recognized by the character recognition unit 45 matches the device name described in the procedure of the operation manual information acquired from the operation-related information storage unit 31 (S3). If they do not match, S1 through S3 are repeated until the valid device name of the operation target device X1 according to the operation manual information is recognized in the image information.

[0030] If there is a match in S3, the operation target device X1 is recognized as a legitimate device in accordance with the operation manual information, and the process proceeds to identifying the operation target part Y1 below. The imaging unit 10 captures an image of the part Y mounted on the operation target device X1 (S4). The character recognition unit 45 recognizes the part name "8R", which is part identification information written on or near the part Y, by image processing (S5). The learning model 42 identifies the type and name of the part Y from the shape of the part Y included in the image information (S6).

[0031] The target identification unit 43 determines whether the part name recognized by the character recognition unit 45 matches the operation target part Y1 registered in the operation manual information, and whether the part Y identified by the learning model 41 from the shape of the part Y based on the image capture information matches the part Y to be operated registered in the operation manual information (S7). If they do not match, steps S4 to S6 are repeated until the name and shape of the valid operation target part Y1 in accordance with the operation manual information are recognized. If they match in S7, the imaged part Y is determined to be the valid operation target part Y1 in accordance with the operation manual information (S8), and identification of the operation target part Y1 is terminated.

[0032] When the target identification unit 43 identifies the operation target part Y1 through the above process, a virtual object 70 such as that shown in FIG. 4 is displayed. The worker confirms that the operation target part Y1 has been identified by the virtual object 70, and operates "8R" of the operation target part Y1 to the "off" state in accordance with the work content described in the menu M. Then, the worker presses the operation completion button as shown in FIG. 4 to end the work on the operation target device X1 "maintenance panel A." Note that instead of the operation completion button, the operation may be completed by the worker uttering "operation completed" using voice recognition.

[0033] As described above, in the mobile terminal device 10 according to this embodiment, the target part Y1 to be operated is identified by the target identification unit 43 based on the part identification information recognized by the image recognition unit 42, the name of the target part Y1 to be operated registered in the operation manual information, and the result of the learning model 41 determining the shape of the part Y in the target device X1 imaged by the imaging unit 10. This means that the target part Y1 to be operated can be identified based only on the name and shape of the part Y, eliminating the need to register in advance in detail the arrangement of the part Y for each device X, thereby significantly reducing implementation costs.

[0034] Furthermore, when a tablet or smartphone is used as the mobile terminal device 2, there is no need to wear it on the head like goggles, so maintenance work can be performed without the discomfort of wearing a heavy object. Also, since the mobile terminal device 10 only needs to be pointed at the device X or component Y when necessary, the work can be performed while directly viewing it with one's own eyes, allowing the work to proceed smoothly.

[0035] (Second embodiment of the present invention) A mobile terminal device and an operation assistance program according to a second embodiment of the present invention will be described with reference to Fig. 6 and Fig. 7. The mobile terminal device 10 according to this embodiment calculates and manages position information in a three-dimensional virtual space so that, when specifying a work target part Y1, only part Y mounted on the work target device X1 is targeted for processing. Note that in this embodiment, descriptions that overlap with those of the first embodiment will be omitted.

[0036] Fig. 6 is a functional block diagram showing the configuration of a mobile terminal device according to this embodiment. Fig. 6 differs from Fig. 2 in the first embodiment in that it includes a position identification unit 46 that identifies the spatial coordinates in the three-dimensional virtual space of the operation target device X1 identified by the target identification unit 43.

[0037] The position identification unit 46 identifies the spatial coordinates in the three-dimensional virtual space of the operation target device X1 identified by the object identification unit 43. Specifically, for example, when the power of the mobile terminal device 10 is turned "ON," the position of the mobile terminal device 10 is set as the origin in the three-dimensional virtual space. Then, as the mobile terminal device 10 moves along with the worker's movement, an image of the nameplate of the device X is captured according to the operation menu M. When the device X is recognized as the operation target device X1, the relative positional relationship between the mobile terminal device 10 and the operation target device X1 in the real space is determined using, for example, an acceleration sensor mounted on the mobile terminal device 10 or an external signal (not shown) obtained by a focus function of the imaging unit 10, and the spatial coordinates of the operation target device X1 in the three-dimensional virtual space are calculated. The operation-related information storage unit 31 stores the shape, size, etc. of each device X, and the spatial region of the entire operation target device X1 in the three-dimensional virtual space is identified.

[0038] Furthermore, the image recognition unit 41 performs image processing on only the spatial region in the three-dimensional virtual space of the target device X1, where the part Y imaged by the imaging unit 10 is identified by the position identification unit 46, and recognizes the part identification information. That is, the processing load can be reduced by not processing image information outside the spatial region occupied by the target device X1.

[0039] Next, the operation of the mobile terminal device according to this embodiment will be described. Fig. 7 is a flowchart showing the operation of the mobile terminal device according to this embodiment. The processes from S1 to S3 are the same as those in Fig. 5 of the first embodiment. If a match is found in S3, the position identification unit 46 calculates the coordinates of the spatial area of ​​the operation target device X1 in the three-dimensional virtual space (S4). Then, the imaging unit 10 images an arbitrary part Y (S5). It is determined whether the position of the imaged part Y in the three-dimensional virtual space is within the spatial area of ​​the operation target device X1 calculated in S4 (S6). If it is outside the spatial area of ​​the operation target device X1, the process returns to S5, where the arbitrary part Y is imaged again, and this process is repeated until a part within the spatial area of ​​the operation target device X1 is imaged.

[0040] If the location is within the spatial region of the operation target device X1 in S6, the processing from S7 onwards is executed. The processing from S7 to S10 is the same as the processing from S5 to S8 in Fig. 5, and therefore a description thereof will be omitted.

[0041] As described above, in the mobile terminal device 10 according to the present embodiment, only the part Y included in the spatial region in the three-dimensional virtual space of the operation target device X1 identified by the position identification unit 46 is subject to image processing. Therefore, even if there are multiple operation target devices X1 of the same type and shape, only the part Y mounted on the operation target device X1 is subject to processing, which can reliably prevent erroneous operation by the operator.

[0042] (Third embodiment of the present invention) A mobile terminal device and an operation assistance program according to a third embodiment of the present invention will be described with reference to Figs. 8 to 10. In the mobile terminal device according to this embodiment, after the operation target device X1 and the operation target part Y1 are identified, the operator checks the operation target device X1 and the operation target part Y1 with his / her fingertip just before actually performing the operation, and determines whether this fingertip check is correct. Furthermore, after performing an operation on the operation target part Y1, the operator checks the part Y that was actually operated with his / her fingertip, and determines whether this fingertip check is correct. Note that in this embodiment, descriptions that overlap with the above embodiments will be omitted.

[0043] Fig. 8 is a functional block diagram showing the configuration of a mobile terminal device according to this embodiment. Fig. 8 differs from Fig. 2 in the first embodiment in that the image recognition unit 41 includes a gesture recognition unit 47 that recognizes gestures including the body shape (hereinafter referred to as fingertips) of the worker, and a gesture determination unit 48 that determines whether the operation target part Y1 is confirmed as a fingertip, i.e., whether the worker recognizes the operation target part Y1, based on the recognized gesture information.

[0044] The gesture recognition unit 47 recognizes the area of ​​the worker's fingertip by processing the image information captured by the imaging unit 10. The processing by the gesture recognition unit 47 can utilize a conventionally known image recognition technology.

[0045] The gesture determination unit 48 determines that the operator recognizes the valid operation target part Y1 when at least a part of the outer edge of the fingertip area recognized by the gesture recognition unit 47 is within or in the vicinity (within a predetermined distance from) the virtual object 70 as shown in FIG. 4 .

[0046] FIG. 9 is a diagram showing an example of a display when performing gesture determination processing in the mobile terminal device according to this embodiment. As shown in FIG. 9, the gesture recognition unit 47 recognizes the operator's fingertip and identifies its outer edge (shown by a thick dotted line in FIG. 9). If a part of this outer edge overlaps with or is within a predetermined distance from the virtual object 70 representing the operation target part Y1, the gesture determination unit 48 determines that the operator has reliably recognized the operation target part Y1. The determination result may be clearly indicated by different display contents or display modes depending on whether the recognition is confirmed or not, such as by flashing if the recognition is determined to be reliably confirmed, or not flashing if the recognition is not determined to be reliably confirmed (for example, if the fingertip is away from the virtual object 70 by a predetermined distance or more). The operator proceeds to the next operation after confirming such a determination result.

[0047] The gesture recognition unit 47 and the gesture determination unit 48 are executed in the following two cases. The first case is, as described above, when the operation target part Y1 of the operation target device X1 is identified by the mobile terminal device 10 and a confirmation is made immediately before the worker actually performs an operation on the operation target part Y1. That is, after the operation target device X1 and the operation target part Y1 are identified by the mobile terminal device 10, when the worker actually performs an operation on the operation target part Y1, the worker may temporarily set the mobile terminal device 10 aside, and the worker's line of sight may be cut off from the display unit 10. In order to eliminate the possibility that the worker's recognition may deviate from the operation target part Y1 during this time, the pointing confirmation is performed immediately before the operation, and if it is determined that the worker recognizes the correct operation target part Y1, a message to that effect is displayed on the display unit 10.

[0048] The second case is when, after the worker actually performs an operation on the operation target part Y1, it is necessary to confirm whether the operated part Y is the legitimate operation target part Y1. In other words, if the part Y operated by the worker is not the operation target part Y1, it is possible to minimize the damage by taking prompt action. In other words, by identifying the actually operated part Y by fingertip confirmation immediately after the operation on part Y, it is possible to determine whether the operation was performed on the legitimate operation target part Y1, and proceed to the next step with confidence.

[0049] Next, the operation of the mobile terminal device according to this embodiment will be described. Fig. 10 is a flowchart showing the operation of the mobile terminal device according to this embodiment. In Fig. 10, the processes from S1 to S8 are the same as those in Fig. 5 of the first embodiment. When the operation target part Y1 is recognized in S8, the display control unit 44 displays a virtual object 70 representing a part area including at least the outer edge of the shape of the operation target part Y1 on the display unit 50 (S9). The gesture recognition unit 47 recognizes the outer edge of the worker's fingertip area based on the imaging information captured by the imaging unit 10 (S10).

[0050] The gesture determination unit 48 determines whether at least a part of the outer edge of the worker's fingertip area is within or near the virtual object 70 displayed in S9 (S11), and if at least a part of the outer edge of the fingertip area is not within or near the virtual object 80, the process of S10 is repeated until the operation target part Y1 is confirmed as the fingertip. If at least a part of the outer edge of the fingertip area is within or near the virtual object 80, it is determined that the worker has recognized the valid operation target part Y1, and a message to that effect is displayed on the display unit 50 (S12), and the process ends.

[0051] In the second case described above, immediately after the operator actually performs an operation on the operation target part Y1, the same processing as S9 to S12 described above is executed. However, if at least a part of the outer edge of the fingertip area is not within or near the virtual object 80 in S11, a warning is displayed on the display unit 10 to the effect that the operated part Y may not have been the operation target part Y1, and the processing is terminated.

[0052] 9 above, fingertip confirmation is performed so that the back of the hand is imaged, but fingertip confirmation may be performed so that the mobile terminal device 10 can image the worker's fingerprint or palm print, and the control unit 40 of the mobile terminal device 10 may have a function for authenticating fingerprints and palm prints. In other words, by authenticating that the person performing fingertip confirmation is definitely the worker in charge and then certifying the fingertip confirmation, more accurate operation can be achieved and it becomes possible to clarify who is responsible if something goes wrong.

[0053] In this way, in the mobile terminal device 10 according to this embodiment, if at least a part of the outer edge of the worker's body is within or near the virtual object 70 based on the gesture information, the gesture determination unit 48 determines that the worker recognizes the valid operation target part Y1 and completes the process of identifying the operation target part Y1. Therefore, in addition to the identification of the operation target part Y1 by the mobile terminal device 2, the worker's recognition is involved, which makes it possible to more reliably prevent erroneous operation of the operation target part Y1.

[0054] Furthermore, if the gesture determination unit 48 determines from the image information after the operation that at least a part of the outer edge of the worker's body is within or near the virtual object 70, it determines that the worker has operated the correct part Y1 to be operated, and if this is not the case, it displays a message indicating that the wrong part Y may have been operated, thereby enabling the worker to immediately grasp the possibility of an erroneous operation of the part Y operated by the worker.

[0055] In another embodiment, the storage unit 30 may store captured images before and after the operator operates the operation target part Y1, and may include an operation determination unit that compares the captured images before and after the operator operates the operation target part Y1 to determine whether the procedure based on the operation manual information for the operation target part Y1 has been completed. That is, by comparing the images before and after the operation of the operation target part Y1, it can be determined that no operation has been performed if there is no difference, and that the operation has been performed if there is a difference. Furthermore, by combining this with the above-mentioned pointing confirmation process, it is also possible to compare the images of the fingertip confirmation state before and after the operation and determine whether there is any discrepancy in the operator's recognition before and after the operation. [Explanation of symbols]

[0056] 10. Portable terminal device 20 Imaging unit 30 Storage section 31 Operation-related information storage unit 40 Control Unit 41 Image Recognition Unit 42 Learning Model 43 Target Identification Department 44 Display control unit 45 Character recognition section 46 Location identification part 47 Gesture Recognition Unit 48 Gesture determination unit 50 Display section 60 Learning Data Section 61 Shape image information 62 Name information 70 Virtual Objects

Claims

1. an imaging means for imaging the device and the component; a display means for displaying image information including still images and moving images captured by the imaging means; a storage means for storing operation manual information of an operation target device to be operated, the operation manual information including device identification information for identifying the device and part identification information of parts mounted on the device; a learning model that has learned the shape of the part and the part identification information; an image recognition means for recognizing the device identification information based on the image information and recognizing the part identification information written on or near a part of the device to be operated; an object specifying means for specifying the operation target device based on the device identification information recognized by the image recognition means and the name of the device to be operated registered in the operation manual information, and for specifying the operation target part based on the part identification information recognized by the image recognition means and the name of the operation target part to be operated registered in the operation manual information, and a result of determining, using a learning model, the shape of the part in the operation target device imaged by the imaging means; a display control means for at least displaying the result of the identification by the target identification means on the display means; A mobile terminal device characterized by the above.

2. 2. The mobile terminal device according to claim 1, a position specifying means for specifying a position coordinate in a three-dimensional virtual space of the operation target device specified by the target specifying means, The image recognition means recognizes only the part mounted on the device imaged by the imaging means when the position coordinates in the three-dimensional virtual space of the device imaged by the imaging means match the position coordinates in the three-dimensional virtual space of the device to be operated. A mobile terminal device characterized by the above.

3. 2. The mobile terminal device according to claim 1, the display control means displays a virtual object representing a part area including at least an outer edge of the shape of the operation target part identified by the target identification means; the image recognition means acquires gesture information including a body shape of the operator based on the imaging information; and a completion confirmation means for determining that the operator has properly identified the operation target part and completing the identification of the operation target part when at least a part of the outer periphery of the operator's body is within or near the virtual object based on the gesture information. A mobile terminal device characterized by the above.

4. 4. The mobile terminal device according to claim 3, When at least a part of the outer edge of the body of the operator is within or near the virtual object with regard to image information captured after the operator operates the operation target part, the completion confirmation means determines that the operator has operated the operation target part legitimately and completes the operation based on the operation manual information. A mobile terminal device characterized by the above.

5. 2. The mobile terminal device according to claim 1, the storage means stores image information before and after the operation of the operation target part by the operator; The device further includes an operation determination unit that determines whether or not the operation of the operation target part based on the operation manual information has been completed based on image information before and after the operation of the operation target part. A mobile terminal device characterized by the above.

6. a learning model that has learned the shapes of parts mounted on the equipment and part identification information that identifies the parts; an image recognition means for recognizing the device identification information for identifying the device based on image information captured by the imaging means, and for recognizing the part identification information written on or near a part of the device to be operated; an object specifying means for specifying the operation target device based on the device identification information recognized by the image recognition means and the name of the device to be operated registered in the operation manual information of the operation target device, which includes device identification information for identifying the device stored in a storage means and part identification information of parts mounted on the device, and for specifying the operation target part based on the part identification information recognized by the image recognition means and the name of the operation target part to be operated registered in the operation manual information, and a result of determining, using a learning model, the shape of the part in the operation target device imaged by the imaging means; causing the computer to function as display control means for at least displaying the result of identification by the target identification means on the display means; A maintenance support program that features:

Images