Work support device and work support method

Abstract

To make it possible to efficiently support work.SOLUTION: A work support device 100 is equipped with a detection unit 111 that detects the position and orientation of an object by comparing a three-dimensional model of the object with at least one of the four pieces of three-dimensional spatial information on the object being a work object which is acquired from a sensor 183, three-dimensional spatial information on surrounding objects around the object, an image of the object, and an image of the surrounding objects and that detects abnormalities related to the object or the surrounding objects based on the image of the object or the surrounding objects. The work support device 100 further includes a support unit 112 that calculates the position of the object or the surrounding object having an abnormality and displays a marker of the position. The work support device 100 may further include a gaze detection unit 113 that detects the gaze of a worker wearing the work support device 100. The detection unit 111 may detect an abnormality based on an image within a field of view indicated by a predetermined angle centered on the line-of-sight direction.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to a work assistance device and a work assistance method for assisting a work. [Background technology]

[0002] When performing maintenance work on facilities and equipment, maintenance personnel are required to find abnormalities (problems) in the facilities and equipment. Examples of abnormalities include oil leaks from oil tanks and (traces of) leaked oil. When a maintenance personnel finds an abnormality, they must take a picture of the abnormal area with a camera and record the facility or equipment where they found it, as well as the location. However, problems can arise with this type of maintenance work, such as overlooking an abnormality or recording it inaccurately.

[0003] One method to support maintenance work is to use a head mounted display (HMD) equipped with a see-through display. The head mounted display can support maintenance work by informing maintenance personnel of maintenance procedures and displaying facility or equipment manuals. Furthermore, by using Augmented Reality (AR) or Mixed Reality (MR) technology, the head mounted display can point out the part or component of the facility or equipment that is the target of the work, which can improve the efficiency of maintenance work. The method described in Patent Document 1 is a method for inspecting an object using augmented reality, which acquires three-dimensional measurement data of the object and displays the measurement data and a camera image of the object. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] US Patent Application Publication No. 2016 / 284079 Summary of the Invention [Problem to be solved by the invention]

[0005] In the method described in Patent Document 1, in order to obtain three-dimensional measurement data, it is necessary to prepare a marker (AR marker), which is time-consuming. Furthermore, Patent Document 1 does not contain any description related to abnormality detection. It is desirable for head-mounted displays to support work and improve the efficiency not only in maintenance work but also in work on various objects, such as product manufacturing work. The present invention has been made in view of the above background, and an object of the present invention is to provide a work support device and a work support method that enable efficient work support. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, the work support device of the present invention is equipped with a detection unit that detects the position and orientation of an object by comparing at least one of four pieces of three-dimensional spatial information of an object that is the subject of work obtained from a sensor, three-dimensional spatial information of surrounding objects around the object, an image of the object, and an image of the surrounding objects with a three-dimensional model of the object, and detects an abnormality related to the object or the surrounding objects based on the image of the object or the surrounding objects, and a support unit that calculates the position of the object or the surrounding object where the abnormality is located, and displays a marker for that position. Effect of the Invention

[0007] According to the present invention, it is possible to provide a work support device and a work support method that enable efficient work support. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0008] [Figure 1] 4 is a flowchart showing a procedure of a maintenance operation according to the embodiment. [Diagram 2] 1 is a diagram showing a pointer displayed on a transmissive display of the work support device according to the present embodiment, and a transformer facility as a scene seen through the transmissive display. FIG. [Diagram 3] A diagram showing a message and a pointer displayed on the transmissive display in this embodiment, and a transformer facility as a scene seen through the transmissive display. [Figure 4] 1 is a diagram showing a message displayed on a transmissive display according to this embodiment, and a transformer facility as a scene seen through the transmissive display. FIG. [Diagram 5] 1 is a diagram showing a message, an area, and a three-dimensional axis displayed on a transmissive display according to this embodiment, and a transforming facility as a scene seen through the transmissive display. [Figure 6] 1 is a diagram showing a message and a marker displayed on a transmissive display according to this embodiment, and a transformer facility as a scene seen through the transmissive display. [Figure 7] 1 is a functional block diagram of a work support device according to an embodiment of the present invention. [Figure 8] 10 is a flowchart of a work support process according to the present embodiment. [Figure 9] 11 is a flowchart of a support process corresponding to an operation of checking for oil leakage as an abnormality of the conservator according to the present embodiment. [Figure 10] 5A and 5B are diagrams for explaining an image region that is a detection target according to the present embodiment. [Figure 11] 11 is a flowchart of a support process corresponding to a task of recording values ​​indicated by instruments according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] <Overview of the work support device> A work support device in a form (embodiment) for carrying out the present invention will be described below. The work support device is a head-mounted display equipped with a see-through display device. The work support device detects the position of the maintenance worker who wears the device and indicates the direction of the facility or equipment to be maintained, thereby guiding the maintenance worker to the facility or equipment. At this time, the work support device calculates the positional relationship and orientation between the maintenance worker and the maintenance object by comparing an image captured by a camera or a 3D point cloud acquired by a depth sensor with a 3D model of the maintenance object. In the following, the maintenance object (object to be maintained) is assumed to be a transformer facility.

[0010] The work support device displays the maintenance procedure when the maintenance worker arrives at the transformer equipment. One of the maintenance tasks for transformer equipment is to check for oil leaks in the conservator (oil tank). The work support device guides the maintenance worker to the location of the conservator or points out the conservator (see Figure 3 described below) and instructs the maintenance worker to check for oil leaks or signs of oil leaks in the conservator or its surroundings. At this time, the work support device detects oil leaks in camera images, especially images in the direction of the maintenance worker's line of sight, and asks the maintenance worker to check. When the maintenance worker confirms an oil leak, the work support device records the image and location of the oil leak. The work support device also displays a marker at the location of the oil leak.

[0011] Such a work support device reduces the time and effort required for maintenance personnel to refer to and record maintenance procedure manuals. In addition, by detecting abnormalities (oil leaks) using the work support device, the number of abnormalities that are overlooked is reduced. This in turn improves the efficiency of maintenance work.

[0012] <Maintenance work overview> Before describing the configuration of the work support device 100 (see FIG. 7 described later), an overview of the maintenance work will be given. FIG. 1 is a flowchart showing the procedure of the maintenance work according to this embodiment. With reference to FIG. 1, an overview of the procedure of the maintenance work performed by a maintenance worker wearing the work support device 100, which is a head-mounted display, on a maintenance target will be described.

[0013] In step S11, the maintenance person moves to the installation location of the facility or device to be maintained, using the pointer 511 (see FIG. 2 described later). In the following description, it is assumed that the maintenance target is a transformer facility.

[0014] 2 is a diagram showing a pointer 511 displayed on a transmissive display 181 (see FIG. 7 described later) of the work support device 100 according to this embodiment, and a voltage transformer facility 400 as a scene seen through the transmissive display 181. The work support device 100 is equipped with the transmissive display 181. A maintenance worker sees a forward scene seen through the transmissive display 181 and an image displayed on the transmissive display 181 superimposed on each other. Note that the image is seen three-dimensionally by binocular stereoscopic vision.

[0015] The work support device 100 is equipped with an accelerometer and a gyroscope, and constantly (periodically) detects its own position and orientation. Even if the maintenance worker moves or turns his / her head, the work support device 100 displays a pointer 511 on the transmissive display 181 so as to point to the transformer facility 400.

[0016] Returning to Fig. 1, the explanation of the maintenance work procedure will be continued. When the maintenance worker moves to the front of the maintenance target (transformation equipment 400), the maintenance worker performs step S12 as the next work. The maintenance worker may perform step S12 after uttering "arrived" or "started work". In step S12, the maintenance person starts the process of repeating the work in step S13. In step S13, the maintenance person performs work in accordance with the work instructions (see message 521 in FIG. 3 described later) displayed on the transmissive display 181. In the following, the work will be described using an example of checking for oil leakage as an abnormality in the conservator.

[0017] 3 is a diagram showing a message 521 and a pointer 522 displayed on the transmissive display 181 according to this embodiment, and the transformer facility 400 as a scene seen through the transmissive display 181. The message 521 instructs the maintenance worker to check for oil leakage from the conservator 410, and to record the location of the leakage if any. The pointer 522 points to the conservator 410 as the target of the work. The maintenance worker looks at the conservator 410 and the area below it to check for oil leakage. The following explanation will be continued assuming that there is oil leakage 523 (traces of oil leakage).

[0018] 4 is a diagram showing a message 531 displayed on the transmissive display 181 according to this embodiment, and the transformer facility 400 as a scene seen through the transmissive display 181. The work support device 100 detects and photographs an oil leak 523. The message 531 asks the maintenance worker whether to record the oil leak 523. If the maintenance worker utters "No", the work support device 100 does not record anything. If the maintenance worker utters "Yes", the work support device 100 displays a message 541 (see FIG. 5 described later).

[0019] 5 is a diagram showing a message 541, an area 542, a three-dimensional axis 543, and the transformer facility 400 as a scene seen through the transmissive display 181 according to the present embodiment. The message 541 asks the maintenance worker whether it is appropriate to record the captured image of the area 542 as an image of an abnormality and to record the position indicated by the three-dimensional axis 543 as the position of the abnormality. When the maintenance worker utters "Yes", the work support device 100 stores the image of the oil leak 523 shown in the area 542, the position indicated by the three-dimensional axis 543, the confirmation date and time, etc., as records related to the conservator 410 of the transformer facility 400. When the maintenance worker utters "No", the work support device 100 does not record.

[0020] FIG. 6 is a diagram showing a message 551 and a marker 552 displayed on the transmissive display 181 according to this embodiment, and the transformer facility 400 as a scene seen through the transmissive display 181. The message 551 indicates that an image of the oil leak 523 as an abnormality and the position indicated by the marker 552 have been stored as a record of the maintenance work related to the conservator 410. The position indicated by the marker 552 is recorded as a relative position seen from the transformer facility 400, for example. Even if the maintenance worker moves or changes the direction of his / her head, the work support device 100 displays the marker 552 on the transmissive display 181 so that the marker 552 is at the recorded position (the position indicated by the three-dimensional axis 543 in FIG. 5). In this way, the work support device 100 detects the oil leak 523 as an abnormality and records the position of the oil leak 523.

[0021] Returning to Fig. 3, a case will be described where there is no oil leak 523. When no oil leak 523 is found and the maintenance worker utters "No abnormality", the work support device 100 stores the fact that there was no abnormality together with the confirmation date and time as a record related to the conservator 410 of the transformer facility 400. When the maintenance worker finds an oil leak 523 that is not detected by the work support device 100, he or she utters, "I found an abnormality. I'm going to take a picture," and indicates the area to be photographed. The subsequent operations are as described in Figs. 5 and 6.

[0022] The maintenance worker may also record any comments he or she notices, such as the cause of an oil leak. The maintenance worker may say, for example, "comment," to instruct the system to record the audio that follows, or to convert the audio into text and record it. The maintenance worker may also say, "take a picture," to instruct the system to record an image captured by the camera.

[0023] The above describes the maintenance work related to an abnormality of oil leakage from the conservator 410. Maintenance work related to other abnormalities includes, for example, damage, cracks, and discoloration of the bushing 420 (see FIG. 2). The work support device 100 supports the maintenance work by indicating the abnormality of the bushing 420 (see pointer 522 in FIG. 3), detecting the abnormality (see message 531 in FIG. 4), and recording it, just like the oil leakage 523.

[0024] The abnormality may be other abnormalities, for example, rust on the transformer equipment 400. In addition to the abnormality, the work support device 100 reads and records the displayed values ​​and values ​​indicated by the instruments. The work support device 100 may also photograph the instruments and record the images.

[0025] <Configuration of the work support device> 7 is a functional block diagram of a work support device 100 according to this embodiment. The work support device 100 is a computer also called a head-mounted display. The work support device 100 includes a control unit 110, a storage unit 120, a transmissive display 181, a speaker 182, a sensor 183, an operation device 184, and a communication device 185.

[0026] The see-through display 181 is capable of displaying three-dimensional CG images. To a maintenance worker wearing the work support device 100, which is a head-mounted display, an object represented by a CG image appears to exist three-dimensionally in the physical world seen through the see-through display 181. The sensor 183 includes a microphone, a camera, a depth sensor, an accelerometer, and a gyroscope. The camera includes a camera for capturing an image in front of the work support device 100, which is a head mounted display, as well as a camera for detecting the line of sight of a maintenance worker wearing the work support device 100. The camera may be an infrared camera for detecting the pupil.

[0027] The depth sensor measures the shape of an object in front of the work support device 100 and outputs it as three-dimensional spatial information (three-dimensional point cloud information). Based on the measurement results of the accelerometer and gyroscope, the position (relative position from a reference point) and orientation of the work support device 100 can be calculated. The sensor 183 may further include a Global Navigation Satellite System (GNSS) antenna and a geomagnetic sensor. The position and orientation of the work support apparatus 100 can be calculated based on information from these sensors. The operation device 184 is, for example, a button or a dial for operating the work support apparatus 100. The communication device 185 is, for example, a device for wireless communication.

[0028] ≪Work support device: Storage section≫ The storage unit 120 includes storage devices such as a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, etc. The storage unit 120 stores a maintenance information database 130, a recognition model 121, and a program 128. The program 128 includes a description of a task support process (see FIG. 8, which will be described later).

[0029] <Memory section: Maintenance information database> The maintenance information database 130 stores information relating to maintenance targets such as facilities and equipment, such as names, positions of maintenance targets, images, three-dimensional models, maintenance procedures, maintenance records, and the like. The maintenance information database 130 may store not only information on the facilities and devices themselves that are the targets of maintenance, but also information on parts / portions that constitute the targets of maintenance, such as names, relative positions from the targets of maintenance, images, three-dimensional models, maintenance procedures, maintenance records, etc. The maintenance information database 130 may also store map information of the areas surrounding the targets of maintenance.

[0030] <Memory section: Recognition model> The recognition model 121 is a machine learning model used to detect (recognize, detect) a maintenance target, a part / portion of the maintenance target, and an abnormality, and is, for example, a model of deep learning technology. Note that the recognition model 121 may be a plurality of recognition models 121 according to the detection targets. For example, there may be a recognition model 121 for each detection target (recognition target, detection target) of the maintenance target, the part / portion of the maintenance target, and an abnormality. By preparing a recognition model 121 for each detection target, the amount of hardware resources required for detection processing can be reduced, and it is expected that the processing speed and detection accuracy will be improved.

[0031] <Work support device: control unit> The control unit 110 includes a CPU (Central Processing Unit) and is provided with a detection unit 111, a support unit 112, and a gaze detection unit 113. The control unit 110 may include a processor such as a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), or an ASIC (Application Specific Integrated Circuit). The control unit 110 may also include a processor that performs processing (holographic processing) to make the image displayed on the transmissive display 181 appear three-dimensional.

[0032] <Work support device: detection unit> The detection unit 111 detects a maintenance target based on information acquired by the sensor 183, and acquires a relative position and orientation with respect to the work support apparatus 100. For example, the detection unit 111 compares three-dimensional space information (three-dimensional point cloud information) acquired by a depth sensor provided in the sensor 183 with a three-dimensional model of the maintenance target stored in the maintenance information database 130, thereby acquiring the position and orientation of the maintenance target.

[0033] Furthermore, the detection unit 111 may detect the maintenance target using the recognition model 121 based on an image acquired by a camera provided in the sensor 183, and may detect the position and orientation of the maintenance target based on the image by matching it with a three-dimensional model. Furthermore, the detection unit 111 may combine the three-dimensional model with three-dimensional space information and the image to detect the maintenance target and acquire the relative position and orientation.

[0034] In this way, the detection unit 111 detects the position and orientation of the maintenance target by comparing the three-dimensional spatial information and images of the maintenance target (transformation facility 400) and its surrounding objects (e.g., the ground) with the three-dimensional model. In addition, the detection unit 111 detects parts / portions that constitute the maintenance target in the same manner as the maintenance target, and obtains the relative positions and orientations.

[0035] Furthermore, the detection unit 111 detects an abnormality related to the maintenance target by using the recognition model 121 based on the image acquired by the camera. An example of an abnormality is an oil leak 523 (see Figs. 3 to 6) in a conservator (oil tank) constituting the transformer equipment to be maintained. The detection unit 111 detects the oil leak 523 (oil stain) on the main body of the conservator 410 or on the ground below the conservator 410, and calculates its position. Note that this position is a position in a three-dimensional space, and can be calculated based on the position and orientation of the work support device 100 and the position of the abnormality (oil leak 523) in the image. This position may be calculated by the support unit 112 described later instead of the detection unit 111.

[0036] When detecting oil leaks 523, the detection unit 111 may narrow down the images to be detected to images of the main body of the conservator 410 or the ground below the conservator 410. This is because oil leaks 523 are often found around the conservator 410. The range of images to be detected may be narrowed down depending on the type of abnormality. In this way, the detection unit 111 can efficiently detect abnormalities. Additionally, the detection unit 111 detects components provided in the maintenance target and acquires their status. For example, the detection unit 111 detects meters provided in the maintenance target and reads the values ​​indicated by the needles or the displayed values, or detects valves and acquires the open / closed state.

[0037] As described above, the work support device 100 includes a detection unit 111 that detects the position and orientation of a work target object (see transformer equipment 400) by comparing at least one of the following four items acquired from the sensor 183 with a three-dimensional model of the object: three-dimensional spatial information of the work target object (see transformer equipment 400), three-dimensional spatial information of surrounding objects (e.g., the ground) around the work target object, an image of the work target object, and an image of the surrounding objects. Furthermore, the detection unit 111 detects an abnormality (see oil leakage 523) related to an object or the surrounding object based on an image of the object or the surrounding object.

[0038] As explained above, the anomaly is an anomaly that relates to a part of the object (see conservator 410). The detection unit 111 detects an abnormality based on an image of the part and the surrounding area of ​​the part (the ground below the conservator 410).

[0039] ≪Work support equipment: Support department≫ The support unit 112 supports the maintenance work by displaying a pointer indicating the facility, device, or part / part to be maintained to the maintenance staff, showing the maintenance procedure, and acquiring records. For example, the support unit 112 displays a pointer (see pointer 511 in FIG. 2 and pointer 522 in FIG. 3) indicating the facility, device, or part / part on the transmissive display 181 based on the position of the facility, device, or part / part acquired by the detection unit 111. In addition, for example, the support unit 112 displays the maintenance procedure and a confirmation message (see messages 521 and 531 in FIG. 3 and FIG. 4). In addition, the support unit 112 calculates the position of the abnormality and displays a marker 552 (see FIG. 6) indicating the position.

[0040] The support unit 112 changes the positions and orientations of the displayed pointers 511, 522 and marker 552 according to the position and orientation of the work support apparatus 100 acquired by the accelerometer and gyroscope. Additionally, the support unit 112 records in the maintenance information database 130 images and positions indicating abnormalities acquired by the detection unit 111, numerical values ​​of instruments read by the detection unit 111, and the like.

[0041] <Work support device: gaze detection unit> The gaze detection unit 113 calculates the gaze (direction of gaze) of the maintenance worker based on the positions of the iris and pupil captured by the camera. More specifically, the gaze detection unit 113 calculates the gaze at the timing when the maintenance worker checks an abnormality or checks instruments, and outputs the calculation result to the detection unit 111. The detection unit 111 detects abnormalities and instruments by narrowing down the image subject to detection processing to the image around the gaze point, rather than the entire image captured by the camera. In this way, the detection unit 111 can efficiently detect abnormalities.

[0042] As described above, the work support device 100 includes a support unit 112 that calculates the position of an object (target object, see transformer equipment 400) having an abnormality (see oil leak 523) or a surrounding object (e.g., the ground) and displays a marker 552 of that position (marker 552 indicating that position). The support unit 112 displays a pointer 522 (see FIG. 3) that points to a part of the object where an abnormality may occur (see the conservator 410).

[0043] The work support device 100 includes a transmissive display 181 . The sensor 183 acquires the position and orientation of the work support apparatus 100. The support unit 112 changes the position and orientation of the marker 552 to be displayed according to the position and orientation acquired by the sensor 183, and displays the marker 552 on the transmissive display 181. The support unit 112 records the location and images of the abnormality. The work support device 100 includes a gaze detection unit 113 that detects the gaze of a worker wearing the work support device 100.

[0044] <Work support processing> Fig. 8 is a flowchart of the work support process according to this embodiment. The work support process of the work support device 100 corresponding to the maintenance work procedure shown in Fig. 1 will be described with reference to Fig. 8. In the following, it is assumed that the maintenance target is a transformer facility 400 (see Fig. 2).

[0045] In step S21, the support unit 112 guides the maintenance worker to the facility / equipment that is the maintenance target. For example, the support unit 112 may calculate the relative position of the maintenance target as seen by the maintenance worker based on the positional relationship between the maintenance worker and the maintenance target, and may display a figure (e.g., an arrow) indicating the direction of the maintenance target on the transmissive display 181 to guide the maintenance worker. Furthermore, when the distance between the maintenance worker and the maintenance target becomes equal to or smaller than a predetermined value, or when it is determined that the maintenance worker can visually recognize the maintenance target, the support unit 112 may display a pointer 511 (see FIG. 2) to guide the maintenance worker.

[0046] The position of the maintenance target and map information about the surrounding area are stored in the maintenance information database 130. The position and orientation of the maintenance worker before starting guidance may be set by the maintenance worker, or may be calculated based on the output results of the GNSS antenna and geomagnetic sensor included in the work support device 100.

[0047] In step S22, the detection unit 111 detects the maintenance target and acquires the relative position and orientation. Steps S21 and S22 may be executed in parallel at the same time, or step S22, which is a process of detecting the maintenance target, may be started from the start of guidance. Step S22 may also be started after it is determined that the distance to the maintenance target is equal to or less than a predetermined value, or that the maintenance target is in a visually recognizable position.

[0048] In step S23, when the support unit 112 acquires the maintenance worker's utterance of "arrived" or "start work" (see step S11 in FIG. 1) via the microphone provided in the sensor 183, it acquires the maintenance procedure for the maintenance target stored in the maintenance information database 130. In step S24, the support unit 112 repeats the process of step S25 for each maintenance procedure acquired in step S23. In step S25, the support unit 112 executes a support process corresponding to the maintenance procedure. The support process corresponding to the task of checking for oil leakage as an abnormality of the conservator 410 described with reference to Figs. 3 to 6 will be described below.

[0049] FIG. 9 is a flowchart of a support process corresponding to an operation of checking for oil leakage as an abnormality of the conservator 410 according to this embodiment. In step S31, the support unit 112 displays a message 521 (see FIG. 3) and a pointer 522. At the time of step S31, the position and orientation of the transformer equipment 400, which is the maintenance target as viewed from the work support device 100, have already been calculated, and the position and orientation of the conservator 410 can also be calculated. The detection unit 111 can also detect the conservator 410, which is a part / portion that constitutes the maintenance target, and can calculate its position and orientation. The support unit 112 displays the pointer 522 based on the calculated position and orientation of the conservator 410.

[0050] In step S32, the line of sight detection unit 113 detects the line of sight (direction of line of sight) of the maintenance worker. In step S33, the detection unit 111 detects an oil leak 523 (see FIG. 3). More specifically, the detection unit 111 detects the oil leak 523 by using the recognition model 121 based on an image acquired by a camera that captures an image in front of the work support device 100 and is included in the sensor 183. Note that the recognition model 121 may be a recognition model 121 used to detect an abnormality related to the conservator 410, in particular an oil leak 523, instead of the recognition model 121 used to detect the maintenance target (transformation equipment 400) in step S22 (see FIG. 8). When the detection unit 111 detects the oil leak 523, the support unit 112 displays a message 531 (see FIG. 4).

[0051] The detection unit 111 may narrow down the area to be detected in the image acquired by the camera. FIG. 10 is a diagram for explaining the image area to be detected according to this embodiment. A dashed line 573 indicates the line of sight (direction of line of sight) detected by the line of sight detection unit 113. The detection unit 111 detects the oil leak 523 by narrowing down the area 571 (field of view) that is within a predetermined angle 572 in the direction of the dashed line 573 (line of sight). The detection unit 111 may refer to the three-dimensional space information acquired by the depth sensor included in the sensor 183, and detect the oil leak 523 by narrowing down the area to an area within the area 571 and in the vicinity of the transformer facility 400 or the conservator 410.

[0052] As described above, detection unit 111 detects an abnormality (see oil leakage 523) based on an image within a field of view (see area 571) indicated by a predetermined angle 572 centered on the line of sight direction (see dashed line 573).

[0053] Returning to FIG. 9, the explanation of the support process will be continued. In step S34, if the maintenance person utters "yes" and confirms oil leak 523 (step S34→YES), the support unit 112 advances the process to step S35. If the maintenance person utters "no" (step S34→NO), the support unit 112 returns the process to step S33. These "yes" and "no" are the "yes" and "no" uttered by the maintenance person described in FIG. 4.

[0054] In step S35, the support unit 112 identifies an area 542 (see FIG. 5) of the oil leak 523 detected by the detection unit 111, and displays a message 541, a frame indicating the area 542, and a three-dimensional axis 543. The support unit 112 calculates a position in three-dimensional space indicated by the three-dimensional axis 543 based on the relationship between the positions and directions of the work support device 100 and the transformer equipment 400 including the ground.

[0055] In step S36, if the maintenance person utters "yes" and confirms the location of oil leak 523 (step S36→YES), the support unit 112 advances the process to step S37. If the maintenance person utters "no" (step S36→NO), the support unit 112 returns the process to step S35. These "yes" and "no" are the "yes" and "no" uttered by the maintenance person described in FIG. 5.

[0056] In step S37, the support unit 112 stores an image of the area 542, the position of its center, and the like in the maintenance information database 130 as a record of the maintenance work related to the conservator 410. The support unit 112 displays a message 551 and a marker 552 (see FIG. 6) at the center position of the area 542.

[0057] The above describes the support process of the work support device 100 corresponding to the task of checking for oil leakage 523 as an abnormality related to the conservator 410. As another support process, a support process corresponding to the task of recording values ​​indicated by instruments such as a voltage meter, an oil level gauge, and an oil level temperature gauge will be described. Fig. 11 is a flowchart of the support process corresponding to the task of recording values ​​indicated by instruments according to this embodiment.

[0058] In step S41, the support unit 112 displays a message. This message instructs the user to record the numerical values ​​of each of the meters provided in the transformer equipment 400 and to confirm that the values ​​are within the normal range. In step S42, the line of sight detection unit 113 detects the line of sight (direction of line of sight) of the maintenance worker. In step S43, the detection unit 111 detects the gauge and reads the value. The process of detecting the gauge is similar to the process of detecting the oil leak 523. When the detection unit 111 reads the value, the support unit 112 displays a message to confirm whether the value is correct or not.

[0059] In step S44, if the maintenance person utters "Yes" to confirm the numerical value (step S44→YES), the support unit 112 advances the process to step S46. If the maintenance person utters "No" (step S44→NO), the support unit 112 advances the process to step S45. The maintenance person utters "No" followed by the correct numerical value.

[0060] In step S45, the support unit 112 acquires the numerical value spoken by the maintenance worker and corrects the numerical value read in step S43. In step S46, the support unit 112 stores and records the numerical value in the maintenance information database 130. Next, the support unit 112 compares the numerical value with a normal range stored in the maintenance information database 130, and displays a message including the numerical value and the comparison result.

[0061] If the numerical value is outside the normal range, the maintenance person investigates the cause and instructs the work support device 100 to record the investigation results. For example, the maintenance person may say "comment" to instruct the work support device 100 to record the subsequent voice, or to convert it into text and record it. The maintenance person may also say "take a picture" to instruct the work support device 100 to record an image (video) captured by the camera. The support unit 112 stores and records the voice, the text converted from the voice, and the captured image in the maintenance information database 130 according to the instructions of the maintenance person. The process of recording such comments and images (video) is not limited to the task of recording numerical values ​​indicated by instruments, but is similar to other tasks including the task of checking for oil leaks as an abnormality.

[0062] When a maintenance worker specifies a position by gazing or pointing, the support unit 112 calculates and records the position of the gaze or pointed destination. The gaze is calculated by the gaze detection unit 113. The pointed direction can be calculated based on an image or three-dimensional space information. The support unit 112 finds the intersection of an object (such as the transformer equipment 400 or the ground) with the gaze or pointed direction, and calculates the position.

[0063] <Features of the work support device> The work support device 100 displays a pointer 511 that guides the maintenance worker to the maintenance target (see the transformer facility 400 in FIG. 2) and a pointer 522 that points to the part to be maintained (see the conservator 410 in FIG. 3). This allows the maintenance worker to easily understand the maintenance target and part.

[0064] The work support device 100 detects an abnormality (see oil leak 523) related to the maintenance target, calculates the position of the abnormality, and displays a marker 552 (see FIG. 6). This position is a position in three-dimensional space, and even if the position or orientation of the work support device 100 changes, the work support device 100 displays the marker 552 as if it were in the same position. Furthermore, the work support device 100 records an image and the position of the abnormality. By using such a work support device 100, maintenance personnel can reduce the amount of time spent overlooking an abnormality and recording the abnormality. Furthermore, the relevant parties can hold a discussion while referring to the records, which allows the discussion to proceed efficiently.

[0065] When detecting an abnormality, the work support device 100 narrows down the range of the image to be detected to the area 571 (see FIG. 10) around the line of sight, or the part related to the abnormality and its surroundings (for example, the conservator 410 and the ground below it). By narrowing down the image range of the detection target in this way, it is possible to reduce the calculation resources required for the detection process, and high-speed and high-precision detection process becomes possible even in the work support device 100, which is a head-mounted display.

[0066] <<Modification: Display Device>> In the above embodiment, the display device provided in the work support device 100 is the transmissive display 181, but is not limited to this. The work support device 100 (the support unit 112) may display a view in front of the work support device 100 captured by a camera on the display, and may further display a pointer 522 and messages 521, 531.

[0067] Although several embodiments of the present invention have been described above, these embodiments are merely illustrative and do not limit the technical scope of the present invention. The present invention can take various other embodiments, and various modifications such as omissions and substitutions can be made without departing from the gist of the present invention. These embodiments and their modifications are included in the scope and gist of the invention described in this specification, etc., and are included in the scope of the invention described in the claims and their equivalents. [Explanation of symbols]

[0068] 100 Work support device 111 Detection unit 112 Support Department 113 Line of Sight Detection Unit 121 Recognition Model 128 Programs 130 Maintenance Information Database 181 Transparent Display 183 Sensors 400 Transformer Equipment (Object) 410 Conservator (part) 523 Oil leak (abnormal) 511,522 Pointers 543 3D axis (position) 552 Marker 571 area (field of view) 572 angle 573 Dashed Line (Gaze)

Claims

1. The position and orientation of the object are detected by comparing at least one of the following four items obtained from the sensor: 3D spatial information of the object being the work target, 3D spatial information of surrounding objects around the object, an image of the object, and an image of the surrounding objects with a 3D model of the object; A detection unit that detects an abnormality related to the object or the surrounding objects based on an image of the object or the surrounding objects; a support unit that calculates the position of the object or the surrounding object having the abnormality and displays a marker of the position. Work support equipment.

2. a gaze detection unit that detects the gaze of a worker wearing the work support device, The detection unit is The abnormality is detected based on an image within a field of view indicated at a predetermined angle centered on the line of sight. The work support device according to claim 1 .

3. The abnormality is an anomaly involving a portion of the object; The detection unit is The abnormality is detected based on an image of the part and its surroundings. The work support device according to claim 1 .

4. The support unit includes: Displaying a pointer pointing to a portion of the object where the anomaly may occur The work support device according to claim 1 .

5. Equipped with a transparent display, The sensor includes: Acquire a position and an orientation of the work support device; The support unit includes: The position and the orientation of the marker to be displayed are changed according to the position and the orientation acquired by the sensor, and the marker is displayed on the transmissive display. The work support device according to claim 1 .

6. The support unit includes: Record the location and image of the anomaly. The work support device according to claim 1 .

7. The work support device is A step of detecting a position and orientation of an object by comparing at least one of four items obtained from the sensor, namely, three-dimensional spatial information of the object being the work target, three-dimensional spatial information of surrounding objects around the object, an image of the object, and an image of the surrounding objects, with a three-dimensional model of the object; detecting an abnormality related to the object or the surrounding object based on an image of the object or the surrounding object; calculating a position of the object or the surrounding object having the abnormality and displaying a marker of the position; Work support method.

Images