Work assistance device and work assistance method

Abstract

A work assistance device (100) is provided with a detection unit (111) which compares a three-dimensional model of an object subject to work against at least one of four items acquired from a sensor (183), the items being three-dimensional space information relating to the object, three-dimensional space information relating to a peripheral object around the object, an image of the object, and an image of the peripheral object, to detect the position and direction of the object, and detect an abnormality related to the object or peripheral object on the basis of the image of the object or peripheral object. The work assistance device (100) is also provided with an assistance unit (112) that calculates the position of the object or peripheral object having an abnormality, and displays a marker of the position.

Description

Work support device and work support method

[0001] The present invention relates to a work support device and a work support method for supporting a work.

[0002] When performing maintenance work on facilities and equipment, maintenance personnel are required to find abnormalities (problems) in the facilities and equipment. Examples of abnormalities include oil leaks from oil tanks and (traces of) leaked oil. When an abnormality is discovered, maintenance personnel must take a photo of the abnormality with a camera and record the facility or equipment where it was discovered, as well as the location. However, this type of maintenance work can sometimes lead to problems, such as overlooking an abnormality or recording it inaccurately.

[0003] One method for supporting maintenance work is to use a head-mounted display (HMD) equipped with a see-through display. The head-mounted display can support maintenance work by informing maintenance personnel of maintenance procedures or displaying facility or equipment manuals. Furthermore, by using augmented reality (AR) technology or mixed reality (MR) technology, the head-mounted display can point out the target part / component of the facility or equipment, which can improve the efficiency of maintenance work. The method described in Patent Document 1 is a method for inspecting an object using augmented reality, which acquires three-dimensional measurement data of the object and displays the measurement data and a camera image of the object.

[0004] US Patent Application Publication No. 2016 / 284079

[0005] The method described in Patent Document 1 requires the preparation of markers (AR markers) to acquire three-dimensional measurement data, which is time-consuming. Patent Document 1 also does not include any description of anomaly detection. It is desirable for head-mounted displays to assist in and improve the efficiency of work not only in maintenance work but also in work involving various objects, such as product manufacturing work. The present invention has been made in light of this background, and aims to provide a work assistance device and a work assistance method that enable efficient work assistance.

[0006] In order to solve the above-mentioned problems, the work assistance device of the present invention includes a detection unit that detects the position and orientation of an object by comparing at least one of four pieces of three-dimensional spatial information of an object that is the target of work, obtained from a sensor, three-dimensional spatial information of surrounding objects around the object, an image of the object, and an image of the surrounding objects with a three-dimensional model of the object, and detects an abnormality related to the object or the surrounding objects based on the image of the object or the surrounding objects, and a assistance unit that calculates the position of the object or the surrounding object where the abnormality is located, and displays a marker for that position.

[0007] According to the present invention, it is possible to provide a work assistance device and a work assistance method that enable efficient work assistance. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments.

[0008] 1 is a flowchart showing a procedure for maintenance work according to the present embodiment. FIG. 1 is a diagram showing a pointer displayed on a transmissive display of a work support device according to the present embodiment, and a view of transformer equipment as a scene seen through the transmissive display. FIG. 2 is a diagram showing a message and a pointer displayed on a transmissive display according to the present embodiment, and transformer equipment as a scene seen through the transmissive display. FIG. 3 is a diagram showing a message displayed on a transmissive display according to the present embodiment, and transformer equipment as a scene seen through the transmissive display. FIG. 4 is a diagram showing a message, an area, and three-dimensional axes displayed on a transmissive display according to the present embodiment, and transformer equipment as a scene seen through the transmissive display. FIG. 5 is a diagram showing a message, a marker displayed on a transmissive display according to the present embodiment, and transformer equipment as a scene seen through the transmissive display. FIG. 6 is a functional block diagram of a work support device according to the present embodiment. FIG. 7 is a flowchart of work support processing according to the present embodiment. FIG. 8 is a flowchart of support processing corresponding to an operation of checking for oil leakage as an abnormality in a conservator according to the present embodiment. FIG. 9 is a diagram for explaining an image area to be detected according to the present embodiment. FIG. 10 is a flowchart of support processing corresponding to an operation of recording numerical values ​​indicated by instruments according to the present embodiment.

[0009] <Overview of Work Support Device> A work support device in a mode (embodiment) for carrying out the present invention will be described below. The work support device is a head-mounted display equipped with a see-through display device. The work support device detects the position of the maintenance worker wearing the device and indicates the direction of the facility or equipment to be maintained, thereby guiding the maintenance worker to the facility or equipment. In this case, the work support device calculates the positional relationship and orientation between the maintenance worker and the maintenance object by comparing images captured by a camera or a 3D point cloud acquired by a depth sensor with a 3D model of the maintenance object. In the following, the maintenance object (the object to be maintained) is assumed to be a transformer facility.

[0010] The work support device displays maintenance procedures when a maintenance worker arrives at a transformer facility. One maintenance task for transformer facilities is to check for oil leaks in the conservator (oil tank). The work support device guides the maintenance worker to the location of the conservator or points out the conservator (see Figure 3 below), instructing the maintenance worker to check for oil leaks or signs of oil leaks in the conservator or its surrounding area. At this time, the work support device detects oil leaks in camera images, particularly in images that are in the line of sight of the maintenance worker, and prompts the maintenance worker to check. When the maintenance worker confirms an oil leak, the work support device records the image and location of the oil leak. The work support device also displays a marker at the location of the oil leak.

[0011] This type of work support device reduces the time and effort required for maintenance personnel to refer to and record maintenance procedure manuals. Furthermore, by detecting abnormalities (oil leaks) using the work support device, it reduces the number of abnormalities that are overlooked. This ultimately improves the efficiency of maintenance work.

[0012] <<Overview of Maintenance Work>> Before describing the configuration of the work assistance device 100 (see FIG. 7 described later), an overview of the maintenance work will be described. FIG. 1 is a flowchart showing the procedure of the maintenance work according to this embodiment. With reference to FIG. 1, an overview of the procedure of the maintenance work performed by a maintenance worker wearing the work assistance device 100, which is a head-mounted display, on a maintenance target will be described.

[0013] In step S11, the maintenance person moves to the installation location of the facility or equipment to be maintained, using the pointer 511 (see FIG. 2, which will be described later). In the following description, it is assumed that the maintenance target is a transformer facility.

[0014] 2 is a diagram showing a pointer 511 displayed on a transmissive display 181 (see FIG. 7 described later) of the work assistance device 100 according to this embodiment, and a view of a transformer facility 400 as seen through the transmissive display 181. The work assistance device 100 is equipped with the transmissive display 181. A maintenance worker sees a view ahead seen through the transmissive display 181 and an image displayed on the transmissive display 181 superimposed on each other. It is assumed that the image appears three-dimensional due to binocular stereoscopic vision.

[0015] The work support device 100 is equipped with an accelerometer and a gyroscope, and constantly (periodically) detects its own position and orientation. Even if the maintenance worker moves or turns his / her head, the work support device 100 displays a pointer 511 on the transmissive display 181 so that the pointer 511 points to the transformer facility 400.

[0016] Returning to FIG. 1 , the explanation of the maintenance work procedure will continue. When the maintenance worker moves in front of the maintenance target (transformer equipment 400), he performs step S12 as the next work. The maintenance worker may perform step S12 after uttering "arrived" or "start work." In step S12, the maintenance worker starts the process of repeating the work of step S13. In step S13, the maintenance worker performs the work in accordance with the work instructions displayed on the transmissive display 181 (see message 521 in FIG. 3 described later). The work will be explained below using the example of checking for oil leakage as an abnormality in the conservator.

[0017] 3 is a diagram showing a message 521 and a pointer 522 displayed on the transmissive display 181 according to this embodiment, and the transformer equipment 400 as a scene seen through the transmissive display 181. The message 521 instructs the maintenance worker to check for oil leaks in the conservator 410 and to record the location of any oil leaks that may be present. The pointer 522 points to the conservator 410 as the target of the work. The maintenance worker looks at the conservator 410 and below it to check for oil leaks. The following explanation will be continued assuming that there is an oil leak 523 (a trace of an oil leak).

[0018] 4 is a diagram showing a message 531 displayed on the transmissive display 181 according to this embodiment, and the transformer facility 400 as a scene seen through the transmissive display 181. The work support device 100 detects and photographs an oil leak 523. The message 531 asks the maintenance worker whether to record the oil leak 523. If the maintenance worker utters "No," the work support device 100 does not record anything. If the maintenance worker utters "Yes," the work support device 100 displays a message 541 (see FIG. 5, described later).

[0019] 5 is a diagram showing a message 541, an area 542, and a three-dimensional axis 543 displayed on the transmissive display 181 according to this embodiment, and the transformer facility 400 as a scene seen through the transmissive display 181. The message 541 asks the maintenance worker whether it is appropriate to record the captured image of the area 542 as an image of an abnormality and to record the position indicated by the three-dimensional axis 543 as the location of the abnormality. If the maintenance worker utters "Yes," the work assistance device 100 stores the image of the oil leak 523 shown in the area 542, the position indicated by the three-dimensional axis 543, the date and time of confirmation, and the like as records related to the conservator 410 of the transformer facility 400. If the maintenance worker utters "No," the work assistance device 100 does not record.

[0020] FIG. 6 is a diagram showing a message 551 and a marker 552 displayed on the transmissive display 181 according to this embodiment, and the transformer facility 400 as a scene seen through the transmissive display 181. The message 551 indicates that an image of the oil leak 523 as an abnormality and the position indicated by the marker 552 have been stored as a record of the maintenance work related to the conservator 410. The position indicated by the marker 552 is recorded as a relative position as seen from the transformer facility 400, for example. Even if the maintenance worker moves or changes the direction of his or her head, the work assistance device 100 displays the marker 552 on the transmissive display 181 so that the marker 552 is located at the recorded position (the position indicated by the three-dimensional axis 543 in FIG. 5 ). In this way, the work assistance device 100 detects the oil leak 523 as an abnormality and records the position of the oil leak 523.

[0021] Returning to Figure 3, a case where there is no oil leak 523 will be described. If no oil leak 523 is found and the maintenance worker utters "No abnormalities," the work support device 100 stores the fact that there were no abnormalities along with the date and time of confirmation as a record related to the conservator 410 of the transformer equipment 400. If the maintenance worker finds an oil leak 523 that is not detected by the work support device 100, he utters "Anomaly found, take a photo" and indicates the area to be photographed. The subsequent work is as described with reference to Figures 5 and 6.

[0022] The maintenance personnel may also record any comments they notice, such as the cause of an oil leak. For example, the maintenance personnel may say "comment" to instruct the system to record the audio that follows, or to convert it into text and record it. The maintenance personnel may also say "take a photo" to instruct the system to record images captured by the camera.

[0023] The above describes the maintenance work related to an abnormality of oil leakage from the conservator 410. Maintenance work related to other abnormalities includes, for example, damage, cracks, and discoloration of the bushing 420 (see FIG. 2). The work assistance device 100 assists in the maintenance work of an abnormality in the bushing 420, as well as the oil leakage 523, by pointing to the bushing 420 (see pointer 522 in FIG. 3), detecting the abnormality (see message 531 in FIG. 4), and recording it.

[0024] The abnormality may be other abnormalities, such as rust on the transformer equipment 400. In addition to abnormalities, the work assistance device 100 reads and records the displayed values ​​and values ​​indicated by the instruments. The work assistance device 100 may also photograph the instruments and record the images.

[0025] 7 is a functional block diagram of a work assistance device 100 according to this embodiment. The work assistance device 100 is a computer also known as a head-mounted display. The work assistance device 100 includes a control unit 110, a storage unit 120, a transmissive display 181, a speaker 182, a sensor 183, an operation device 184, and a communication device 185.

[0026] The transmissive display 181 is capable of displaying three-dimensional CG images. A maintenance worker wearing the work assistance device 100, which is a head-mounted display, sees a CG-imaged object as if it were three-dimensionally present in the physical world seen through the transmissive display 181. The sensor 183 includes a microphone, a camera, a depth sensor, an accelerometer, and a gyroscope. The camera includes a camera that captures an image in front of the work assistance device 100, which is a head-mounted display, as well as a camera that detects the line of sight of the maintenance worker wearing the work assistance device 100. The camera may be an infrared camera that detects the pupil.

[0027] The depth sensor measures the shape of an object in front of the work assistance device 100 and outputs it as three-dimensional spatial information (three-dimensional point cloud information). The position (relative position from a reference point) and orientation of the work assistance device 100 can be calculated based on the measurement results of the accelerometer and gyroscope. The sensor 183 may further include a GNSS (Global Navigation Satellite System) antenna and a geomagnetic sensor. The position and orientation of the work assistance device 100 can be calculated based on information from these sensors. The operation device 184 is, for example, a button or dial for operating the work assistance device 100. The communication device 185 is, for example, a device for wireless communication.

[0028] <Work Support Device: Storage Unit> The storage unit 120 is configured to include storage devices such as a ROM (Read Only Memory), a RAM (Random Access Memory), and a flash memory. The storage unit 120 stores a maintenance information database 130, a recognition model 121, and a program 128. The program 128 includes a description of the work support process (see FIG. 8, which will be described later).

[0029] <Storage Unit: Maintenance Information Database> The maintenance information database 130 stores information related to maintenance targets such as facilities and equipment, such as names, positions of maintenance targets, images, 3D models, maintenance procedures, and maintenance records. The maintenance information database 130 may store not only information related to the facilities and equipment themselves that are maintenance targets, but also information related to parts / portions that make up the maintenance targets, such as names, relative positions from the maintenance targets, images, 3D models, maintenance procedures, and maintenance records. The maintenance information database 130 may also store map information of the area around the maintenance targets.

[0030] <Storage Unit: Recognition Model> The recognition model 121 is a machine learning model used to detect (recognize, detect) a maintenance target, a component / part of the maintenance target, or an abnormality, and is, for example, a model of deep learning technology. The recognition model 121 may be a plurality of recognition models 121 according to the detection target. For example, there may be a recognition model 121 for each detection target (recognition target, detection target) for the maintenance target, the component / part of the maintenance target, and the abnormality. By preparing a recognition model 121 for each detection target, the amount of hardware resources required for detection processing can be reduced, and improvements in processing speed and detection accuracy can be expected.

[0031] <<Work Support Device: Control Unit>> The control unit 110 is configured to include a CPU (Central Processing Unit) and is equipped with a detection unit 111, a support unit 112, and a gaze detection unit 113. The control unit 110 may be configured to include a processor such as a GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), or ASIC (Application Specific Integrated Circuit). The control unit 110 may also include a processor that performs processing (holographic processing) to make the image displayed on the transmissive display 181 appear three-dimensional.

[0032] <Work Support Device: Detection Unit> The detection unit 111 detects a maintenance target based on information acquired by the sensor 183, and acquires the position and orientation of the maintenance target relative to the work support device 100. For example, the detection unit 111 acquires the position and orientation of the maintenance target by comparing three-dimensional spatial information (three-dimensional point cloud information) acquired by a depth sensor provided in the sensor 183 with a three-dimensional model of the maintenance target stored in the maintenance information database 130.

[0033] Furthermore, the detection unit 111 may detect the maintenance target using the recognition model 121 based on an image acquired by a camera provided in the sensor 183, and may detect the position and orientation of the maintenance target based on the image by comparing it with a three-dimensional model. Furthermore, the detection unit 111 may combine the three-dimensional model with three-dimensional space information and the image to detect the maintenance target and acquire the relative position and orientation.

[0034] In this way, the detection unit 111 detects the position and orientation of the maintenance target (transformer facility 400) by comparing three-dimensional spatial information and images of the maintenance target (transformer facility 400) and its surrounding objects (for example, the ground) with the three-dimensional model. In addition, the detection unit 111 detects parts / portions that constitute the maintenance target in the same way as the maintenance target, and obtains their relative positions and orientations.

[0035] The detection unit 111 also uses the recognition model 121 based on the image acquired by the camera to detect an abnormality related to the maintenance target. An example of an abnormality is an oil leak 523 (see FIGS. 3 to 6 ) in a conservator (oil tank) that constitutes the transformer equipment that is the maintenance target. The detection unit 111 detects the oil leak 523 (oil stain) on the main body of the conservator 410 or on the ground below the conservator 410, and calculates its position. Note that this position is a position in three-dimensional space and can be calculated based on the position and orientation of the work support device 100 and the position of the abnormality (oil leak 523) in the image. This position may be calculated by the support unit 112, which will be described later, instead of the detection unit 111.

[0036] When detecting an oil leak 523, the detection unit 111 may narrow down the images to be detected to images of the main body of the conservator 410 or the ground below the conservator 410. This is because oil leaks 523 are often found around the conservator 410. The range of images to be detected may be narrowed down depending on the type of abnormality. In this way, the detection unit 111 can efficiently detect abnormalities. In addition, the detection unit 111 detects parts provided in the maintenance target and acquires their status. For example, the detection unit 111 detects instruments provided in the maintenance target and reads the values ​​indicated by the needles or the displayed values, or detects valves and acquires the open / closed status.

[0037] As described above, the work assistance device 100 includes a detection unit 111 that detects the position and orientation of an object by comparing at least one of the following four pieces of information acquired from the sensor 183: three-dimensional spatial information of the object that is the work target (see transformer facility 400), three-dimensional spatial information of surrounding objects (e.g., the ground) around the object, an image of the object, and an image of the surrounding objects with a three-dimensional model of the object. Furthermore, the detection unit 111 detects an abnormality (see oil leak 523) related to the object or the surrounding objects based on the image of the object or the surrounding objects.

[0038] As described above, the abnormality is an abnormality related to a part of the object (see the conservator 410). The detection unit 111 detects the abnormality based on an image of the part and the surrounding area of ​​the part (the ground below the conservator 410).

[0039] <Work Support Device: Support Unit> The support unit 112 supports maintenance work by displaying a pointer indicating the facility, device, or part / section to be maintained, indicating the maintenance procedure, and acquiring records. For example, the support unit 112 displays a pointer indicating the facility, device, or part / section (see pointer 511 in FIG. 2 and pointer 522 in FIG. 3 ) on the transmissive display 181 based on the position of the facility, device, or part / section acquired by the detection unit 111. Furthermore, for example, the support unit 112 displays maintenance procedures and confirmation messages (see messages 521 and 531 in FIGS. 3 and 4 ). The support unit 112 also calculates the position of the abnormality and displays a marker 552 (see FIG. 6 ) indicating the position.

[0040] The support unit 112 changes the positions and orientations of the displayed pointers 511, 522 and markers 552 according to the position and orientation of the work support device 100 acquired by the accelerometer and gyroscope. In addition, the support unit 112 records, in the maintenance information database 130, images and positions indicating abnormalities acquired by the detection unit 111, numerical values ​​of instruments read by the detection unit 111, and the like.

[0041] <Work Support Device: Gaze Detection Unit> The gaze detection unit 113 calculates the gaze (direction of gaze) of the maintenance worker based on the positions of the iris and pupil captured by the camera. More specifically, the gaze detection unit 113 calculates the gaze when the maintenance worker is checking an abnormality or checking instruments, and outputs the calculation result to the detection unit 111. The detection unit 111 detects abnormalities and instruments by narrowing down the image to be subjected to detection processing to the image around the gaze point, rather than the entire image captured by the camera. In this way, the detection unit 111 can efficiently detect abnormalities.

[0042] As described above, the work assistance device 100 includes the assistance unit 112 that calculates the position of an object (target object, see the transformer facility 400) or a surrounding object (for example, the ground) that has an abnormality (see the oil leak 523) and displays a marker 552 at that position (a marker 552 indicating that position). The assistance unit 112 displays a pointer 522 (see FIG. 3 ) that points to a part of the object where an abnormality may occur (see the conservator 410).

[0043] The work assistance device 100 includes a transmissive display 181. A sensor 183 acquires the position and orientation of the work assistance device 100. The assistance unit 112 changes the position and orientation of a marker 552 displayed in accordance with the position and orientation acquired by the sensor 183, and displays the marker 552 on the transmissive display 181. The assistance unit 112 records the position and image of the abnormality. The work assistance device 100 includes a gaze detection unit 113 that detects the gaze of a worker wearing the work assistance device 100.

[0044] <Work Support Processing> Fig. 8 is a flowchart of the work support processing according to this embodiment. With reference to Fig. 8, the work support processing of the work support device 100 corresponding to the maintenance work procedure shown in Fig. 1 will be described. In the following, it is assumed that the maintenance target is the transformer facility 400 (see Fig. 2).

[0045] In step S21, the support unit 112 guides the maintenance worker to the facility / equipment that is the maintenance target. For example, the support unit 112 may calculate the relative position of the maintenance target as seen by the maintenance worker based on the positional relationship between the maintenance worker and the maintenance target, and may display a graphic (e.g., an arrow) indicating the direction of the maintenance target on the transmissive display 181 to guide the maintenance worker. Furthermore, when the distance between the maintenance worker and the maintenance target is equal to or less than a predetermined value, or when it is determined that the maintenance worker can see the maintenance target, the support unit 112 may display a pointer 511 (see FIG. 2) to guide the maintenance worker.

[0046] The location of the maintenance target and map information about the surrounding area are stored in the maintenance information database 130. The location and orientation of the maintenance worker before starting guidance may be set by the maintenance worker, or may be calculated based on the output results of the GNSS antenna and geomagnetic sensor provided in the work assistance device 100.

[0047] In step S22, the detection unit 111 detects the maintenance target and acquires its relative position and orientation. Steps S21 and S22 may be executed simultaneously in parallel, or step S22, which is the process of detecting the maintenance target, may be started from the start of guidance. Step S22 may also be started after it is determined that the distance to the maintenance target is equal to or less than a predetermined value or that the maintenance target is in a visible position.

[0048] In step S23, when the support unit 112 receives the maintenance worker's utterance of "arrived" or "start work" (see step S11 in FIG. 1) via the microphone provided in the sensor 183, it obtains the maintenance procedure for the maintenance target stored in the maintenance information database 130. In step S24, the support unit 112 repeats the process of step S25 for each maintenance procedure obtained in step S23. In step S25, the support unit 112 executes support processing according to the maintenance procedure. The following describes the support processing corresponding to the work of checking for oil leakage as an abnormality in the conservator 410 described using FIGS. 3 to 6.

[0049] 9 is a flowchart of support processing corresponding to the work of checking for oil leakage as an abnormality in the conservator 410 according to this embodiment. In step S31, the support unit 112 displays a message 521 (see FIG. 3 ) and a pointer 522. At the time of step S31, the position and orientation of the transformer equipment 400, which is the maintenance target as seen from the work support device 100, have already been calculated, and the position and orientation of the conservator 410 can also be calculated. The detection unit 111 can also detect the conservator 410, which is a part / portion that constitutes the maintenance target, and can calculate its position and orientation. The support unit 112 displays the pointer 522 based on the calculated position and orientation of the conservator 410.

[0050] In step S32, the gaze detection unit 113 detects the gaze (gaze direction) of the maintenance worker. In step S33, the detection unit 111 detects an oil leak 523 (see FIG. 3 ). More specifically, the detection unit 111 detects the oil leak 523 using a recognition model 121 based on an image acquired by a camera included in the sensor 183 that captures an image in front of the work support device 100. Note that the recognition model 121 may be the recognition model 121 used to detect an abnormality related to the conservator 410, particularly the oil leak 523, instead of the recognition model 121 used to detect the maintenance target (transformer equipment 400) in step S22 (see FIG. 8 ). When the detection unit 111 detects the oil leak 523, the support unit 112 displays a message 531 (see FIG. 4 ).

[0051] The detection unit 111 may narrow down the area to be detected in the image acquired by the camera. FIG. 10 is a diagram illustrating the image area to be detected according to this embodiment. A dashed line 573 indicates the line of sight (direction of the line of sight) detected by the line of sight detection unit 113. The detection unit 111 detects the oil leak 523 by narrowing down the field of view to an area 571 (field of view) that is within a predetermined angle 572 in the direction of the dashed line 573 (line of sight). The detection unit 111 may refer to three-dimensional spatial information acquired by a depth sensor included in the sensor 183, and detect the oil leak 523 by narrowing down the field of view to an area within the area 571 that is near the transformer facility 400 or the conservator 410.

[0052] As described above, the detection unit 111 detects an abnormality (see oil leak 523) based on an image within the field of view (see area 571) indicated by a predetermined angle 572 centered on the line of sight direction (see dashed line 573).

[0053] Returning to Fig. 9, the explanation of the support process will be continued. In step S34, if the maintenance person utters "yes" and confirms the oil leak 523 (step S34 → YES), the support unit 112 proceeds to step S35. If the maintenance person utters "no" (step S34 → NO), the support unit 112 returns the process to step S33. These "yes" and "no" are the "yes" and "no" uttered by the maintenance person described in Fig. 4.

[0054] In step S35, the support unit 112 identifies an area 542 (see FIG. 5 ) of the oil leak 523 detected by the detection unit 111, and displays a message 541, a frame indicating the area 542, and a three-dimensional axis 543. The support unit 112 calculates the position in three-dimensional space indicated by the three-dimensional axis 543 based on the relationship between the positions and orientations of the work support device 100 and the transformer equipment 400, including the ground.

[0055] In step S36, if the maintenance person utters "yes" and confirms the location of oil leak 523 (step S36→YES), the support unit 112 proceeds to step S37. If the maintenance person utters "no" (step S36→NO), the support unit 112 returns the process to step S35. These "yes" and "no" are the "yes" and "no" uttered by the maintenance person described in FIG. 5.

[0056] In step S37, the support unit 112 stores the image of the area 542, the position of its center, and the like in the maintenance information database 130 as a record of the maintenance work related to the conservator 410. The support unit 112 displays a message 551 and a marker 552 (see FIG. 6) at the center position of the area 542.

[0057] The above describes the support process of the work support device 100 corresponding to the task of checking for oil leakage 523 as an abnormality related to the conservator 410. As another support process, a support process corresponding to the task of recording values ​​indicated by instruments such as a voltmeter, oil level gauge, and oil level thermometer will be described. Figure 11 is a flowchart of the support process corresponding to the task of recording values ​​indicated by instruments according to this embodiment.

[0058] In step S41, the support unit 112 displays a message. This message instructs the maintenance worker to record the numerical values ​​of each meter equipped in the transformer equipment 400 and confirm that the values ​​are within the normal range. In step S42, the gaze detection unit 113 detects the gaze (direction of gaze) of the maintenance worker. In step S43, the detection unit 111 detects the meter and reads the numerical value. The process of detecting the meter is the same as the process of detecting the oil leak 523. When the detection unit 111 reads the numerical value, the support unit 112 displays a message confirming whether the numerical value is correct or not.

[0059] In step S44, if the maintenance technician utters "yes" and confirms the value (step S44→YES), the support unit 112 proceeds to step S46. If the maintenance technician utters "no" (step S44→NO), the support unit 112 proceeds to step S45. The maintenance technician utters "no" followed by the correct value.

[0060] In step S45, the support unit 112 acquires the numerical value spoken by the maintenance technician and corrects the numerical value read in step S43. In step S46, the support unit 112 stores and records the numerical value in the maintenance information database 130. Next, the support unit 112 compares the numerical value with a normal range stored in the maintenance information database 130 and displays a message including the comparison result.

[0061] If the value is outside the normal range, the maintenance technician investigates the cause and instructs the work support device 100 to record the investigation results. The maintenance technician may, for example, say "comment" to instruct the work support device 100 to record the subsequent audio, or to convert the audio into text and record it. The maintenance technician may also say "take a photo" to instruct the work support device 100 to record an image (video) captured by the camera. The support unit 112 stores and records the audio, the text converted from the audio, and the captured image in the maintenance information database 130 in accordance with the maintenance technician's instructions. The recording process of such comments and images (video) is not limited to the task of recording the values ​​indicated by instruments, but is also applicable to other tasks, including the task of checking for an oil leak as an abnormality.

[0062] When a maintenance worker specifies a position by looking at the object or pointing at it, the support unit 112 calculates and records the position of the gaze or pointed location. The gaze is calculated by the gaze detection unit 113. The pointed direction can be calculated based on an image or three-dimensional space information. The support unit 112 calculates the position by determining the intersection of the object (such as the transformer equipment 400 or the ground) with the gaze or pointed direction.

[0063] <Features of the Work Support Device> The work support device 100 displays a pointer 511 that guides the maintenance worker to the maintenance target (see the transformer facility 400 in FIG. 2 ) and a pointer 522 that points to the part to be maintained (see the conservator 410 in FIG. 3 ). This allows the maintenance worker to easily understand the maintenance target and part.

[0064] The work support device 100 detects an abnormality (see oil leak 523) related to the maintenance target, calculates the position of the abnormality, and displays a marker 552 (see FIG. 6 ). This position is a position in three-dimensional space, and even if the position or orientation of the work support device 100 changes, the work support device 100 displays the marker 552 as if it were in the same position. Furthermore, the work support device 100 records images and positions of the abnormality. By using such a work support device 100, maintenance personnel can reduce the amount of time spent overlooking abnormalities and recording abnormalities. Furthermore, since the parties involved can hold discussions while referring to the records, discussions can proceed efficiently.

[0065] When detecting an abnormality, the work assistance device 100 narrows the range of the image to be detected to an area 571 (see FIG. 10 ) in the vicinity of the line of sight, or to the part related to the abnormality and its surroundings (for example, the conservator 410 and the ground below it). Narrowing the image range to be detected in this way makes it possible to reduce the computational resources required for detection processing, and enables high-speed, high-precision detection processing even in the work assistance device 100, which is a head-mounted display.

[0066] <<Modification: Display Device>> In the above embodiment, the display device provided in the work assistance device 100 is the transmissive display 181, but is not limited to this. The work assistance device 100 (assistance unit 112) may display a view in front of the work assistance device 100 captured by a camera on the display, and may also display a pointer 522 and messages 521 and 531.

[0067] Although several embodiments of the present invention have been described above, these embodiments are merely illustrative and do not limit the technical scope of the present invention. The present invention can take on various other embodiments, and various modifications such as omissions and substitutions can be made without departing from the spirit of the present invention. These embodiments and their modifications are included within the scope and spirit of the invention described in this specification, etc., and are also included in the invention described in the claims and their equivalents.

[0068] 100 Work support device 111 Detection unit 112 Support unit 113 Gaze detection unit 121 Recognition model 128 Program 130 Maintenance information database 181 Transparent display 183 Sensor 400 Transformer equipment (object) 410 Conservator (part) 523 Oil leak (abnormality) 511, 522 Pointer 543 Three-dimensional axis (position) 552 Marker 571 Area (field of view) 572 Angle 573 Dashed line (gaze)

Claims

1. A work support device comprising: a detection unit that detects the position and orientation of an object by comparing at least one of the following four pieces of information obtained from a sensor: 3D spatial information of the object that is the subject of work, 3D spatial information of surrounding objects around the object, an image of the object, and an image of the surrounding objects with a 3D model of the object; and detects abnormalities related to the object or the surrounding objects based on the image of the object or the surrounding objects; and a support unit that calculates the position of the object or the surrounding object where the abnormality is located, and displays a marker for that position.

2. A work support device as described in claim 1, further comprising a gaze detection unit that detects the gaze of a worker wearing the work support device, the detection unit detecting the abnormality based on an image within a field of view indicated at a predetermined angle centered on the gaze direction.

3. The work support device according to claim 1, wherein the abnormality is an abnormality relating to a part of the object, and the detection unit detects the abnormality based on an image of the part and an image of the surrounding area of ​​the part.

4. The work support device according to claim 1, wherein the support unit displays a pointer indicating a part of the object where the abnormality may occur.

5. A work support device as described in claim 1, further comprising a transmissive display, wherein the sensor acquires a position and orientation of the work support device, and the support unit displays the marker on the transmissive display by changing the position and orientation of the marker displayed according to the position and orientation acquired by the sensor.

6. The work support device according to claim 1, wherein the support unit records the position of the abnormality and an image.

7. A work support method in which a work support device executes the steps of: detecting the position and orientation of an object by comparing at least one of four items obtained from a sensor, namely, three-dimensional spatial information of an object that is the subject of work, three-dimensional spatial information of surrounding objects around the object, an image of the object, and an image of the surrounding objects, with a three-dimensional model of the object; detecting an abnormality related to the object or the surrounding objects based on the image of the object or the surrounding objects; and calculating the position of the object or the surrounding object where the abnormality is present, and displaying a marker for that position.

Images