Supporting data generation method and work support device

Abstract

To provide a supporting data generation method which can reduce man-hours of work for labeling each of interfaces shown by a base map.SOLUTION: A base map 54a and a component template 54b are stored in a storage device 54 included in a generation device 50. The base map 54a is three-dimensional data showing a workplace. General-purpose components are provided in the workplace. The general-purpose components are provided with a plurality of interfaces. The component template 54b includes image data of each general-purpose component, a label variable specifying an interface provided in the general-purpose component, and position data of the interface. A PU 52 labels the respective interfaces in the general-purpose components shown by the base map in accordance with pattern matching and alignment between the general-purpose components shown by the base map 54a and the general-purpose components shown by the component template 54b.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a method for producing assistance data and a work assistance device. [Background technology]

[0002] A task support device that uses augmented reality (AR) is described in the following Patent Document 1. This device displays a contour image superimposed on an operation target in an image of the task target captured by a camera. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6736204 specification Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, when adding virtual reality information such as contours to the interface of an operation target, it is necessary to create 3D data showing the work area including the operation target in advance. In particular, the 3D data needs to be given a label specifying the interface of the operation target. However, the labor required to identify and label the interface in the operation target for each operation target is excessive. [Means for solving the problem]

[0005] Means for solving the above problems and their effects will be described below. 1. A method for generating support data for supporting work in a workplace using virtual information, the support data being data in which a base map, which is three-dimensional data representing the workplace, is linked to a label that identifies an interface indicated by the base map, the workplace being equipped with general-purpose parts that can be reused among a plurality of workplaces, the general-purpose parts being parts that have a plurality of the interfaces, a plurality of part templates being stored in a storage device, the part templates including the label and position data for each of a plurality of the interfaces equipped in the corresponding general-purpose part, the position data including data that identifies the position of the interface in the general-purpose part, and a method for manufacturing support data comprising a linking step of linking the label to the interface indicated by the base map by associating the corresponding part template with the general-purpose part indicated by the base map.

[0006] In the above method, a part template is prepared in advance. Then, a part template corresponding to a generic part indicated by the base map is associated with the base map. This identifies the positions of each of a plurality of interfaces provided in the generic part in the base map. Therefore, it is possible to assign a label to each of the plurality of interfaces indicated by the base map. Therefore, it is possible to reduce the man-hours required for the task of labeling each of the interfaces indicated by the base map.

[0007] 2. A method for producing support data as described in claim 1, wherein the linking process includes an assignment process for assigning information indicating the use of the general-purpose parts indicated by the base map in the workshop. The use of the generic parts installed in the workshop may not be clearly defined. Therefore, the above method includes an assignment step of assigning information indicating the use of the generic parts in the workshop to the generic parts. This allows the information on the use of the generic parts to be included in the support data.

[0008] 3. A method for producing support data as described in 1 or 2 above, wherein the linking step is a step of causing a computer to execute a matching process between the generic parts indicated by the base map and the part template.

[0009] In the above method, the process of matching generic parts with component templates is automated, thereby reducing the amount of manual work required. 4. The method for producing support data according to any one of 1 to 3 above, wherein the part template includes two-dimensional or three-dimensional CAD data representing the generic part.

[0010] At the stage when a generic part is not yet in practical use, it is not possible to obtain 3D data of the generic part, but it is possible to obtain 2D or 3D CAD data used to design the generic part. Therefore, in the above configuration, by generating a part template using 2D or 3D CAD data, the part template can be used to generate support data before the generic part is placed in the workplace.

[0011] 5. A work support device that supports a task using support data manufactured by the support data manufacturing method described in 1 above, comprising a storage device, an execution device, and a display unit, wherein the storage device stores work procedure data indicating a procedure of the task, and the execution device is configured to execute a task identification process and a work instruction process, wherein the task identification process is a process of identifying the next task to be performed based on the work procedure data, and the work instruction process is a process of displaying a work instruction for the interface corresponding to the task identified by the task identification process by operating the display unit based on the support data.

[0012] The task identification process can identify the interface that is the target of the task identified by the identification process based on the label attached to the support data. Therefore, task instructions for the interface that is the target of the task can be displayed. In particular, the task support device uses support data that reduces the number of manufacturing steps, so that costs can be reduced. [Brief description of the drawings]

[0013] [Figure 1] 1 is a diagram illustrating a configuration of a work support device according to an embodiment. [Diagram 2] 2 is a flowchart showing a procedure of a process executed by the work support device shown in FIG. 1. [Diagram 3] 11A and 11B are diagrams illustrating an example of work support according to the embodiment. [Figure 4] FIG. 2 is a diagram illustrating a generating device for support data according to the embodiment. [Diagram 5] 11 is a flowchart showing a process for generating support data according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment will be described with reference to the drawings. "Configuration of work support device" 1 shows a work support device 10. The work support device 10 is a device that provides, by means of virtual reality information, an interface that a person should confirm or operate when performing work according to a work procedure, and the work content.

[0015] The work support device 10 includes a PU 12 and a storage device 14. The PU 12 is a software processing device such as a CPU, a GPU, or a TPU. The storage device 14 may be an electrically non-rewritable non-volatile memory. The storage device 14 may also be an electrically rewritable non-volatile memory or a storage medium such as a disk medium. The work support device 10 executes a process for supporting work by the PU 12 executing a program stored in the storage device 14.

[0016] The work assistance device 10 may be mounted on a portable terminal 30 equipped with a display, such as a tablet terminal or a multi-function mobile phone. The work assistance device 10 may also be mounted on a glasses-type wearable terminal 32.

[0017] The PU 12 of the work support device 10 imports image data captured by the camera 20 to support the work. The PU 12 also supports the work by operating the display unit 22 to display augmented reality information. That is, when the work support device 10 is mounted on the portable terminal 30, the display unit 22 is a display. The PU 12 displays an image captured by the camera on the display unit 22 and adds virtual reality information to a predetermined interface of the image. When the work support device 10 is mounted on the wearable terminal 32, the display unit 22 includes a holographic display or the like. That is, the display unit 22 includes a transmissive screen or the like that can superimpose virtual reality visual information on real visual information. The PU 12 adds virtual reality information to a predetermined interface in the real world by operating the display unit 22.

[0018] The storage device 14 stores a work procedure list 14a and support data 14b. The work procedure list 14a is a list that specifies the work that the worker should perform. The work procedure list 14a includes data that specifies the order of the work that the worker should perform. The work that the worker should perform includes, for example, operating an operation unit such as a switch. The work that the worker should perform also includes the task of checking the contents displayed on the display. The work procedure list includes data that specifies the interface, including the operation unit and display, that is the subject of the work.

[0019] The support data 14b is data in which a label is added to a base map. The base map is three-dimensional data showing a work site where a worker should work according to the work procedure list 14a. The label is added to an interface shown by the base map. The label is data that enables the PU 12 to identify which interface shown by the base map is the interface specified by the work procedure list 14a.

[0020] "Work support" Fig. 2 shows the procedure of the process executed by the work support device 10. The process shown in Fig. 2 is realized by the PU 12 repeatedly executing a program stored in the storage device 14, for example, at a predetermined interval. In the following, the step number of each process is represented by a number preceded by "S."

[0021] 2, the PU 12 first acquires image data captured by the camera 20 (S10). Next, the PU 12 aligns the camera coordinate system with the coordinate system of the base map indicated by the support data 14b (S12).

[0022] Next, the PU 12 reads a work instruction defined in the work procedure list 14a (S14). This process is a process of selectively reading an instruction for a work to be done now from among a plurality of work instructions defined in the work procedure list 14a. The PU 12 identifies a label in the support data 14b that corresponds to the work instruction read in the process of S14 (S16). The PU 12 then operates the display unit 22 to mark an interface identified by the label, and displays augmented reality information indicating the work content (S18).

[0023] FIG. 3 shows an example of instructing a task when the task assistance device 10 is mounted on the mobile terminal 30. In FIG. As shown in Fig. 3, the PU 12 displays an image captured by the camera 20 of the portable terminal 30 on the display unit 22. Then, a marking is added to an interface in the image displayed on the display unit 22 where a task instruction is given, and the task is instructed in text. Fig. 3 shows an example in which turning off a switch 40 is the task. That is, the PU 12 puts a marking 42 on the switch 40 on the display unit 22, and displays "OFF" near the switch 40. That is, Fig. 3 shows an example in which the marking 42 and the text "OFF" are the augmented reality information to be displayed.

[0024] In addition, when the work assistance device 10 is mounted on the wearable terminal 32, the PU 12 may display the marking 42 at a position where the worker views the switch 40 via the wearable terminal 32, and may also display "OFF" near the marking.

[0025] Returning to Fig. 2, the PU 12 judges whether or not an input indicating that the work has been completed has been made by the worker operating the input unit 24 shown in Fig. 1 (S20). When the PU 12 judges that an input indicating that the work has been completed has been made (S20: YES), the PU 12 judges whether or not a next work is specified in the work procedure list 14a (S22). When the PU 12 judges that a next work is specified (S22: YES), the PU 12 returns to the process of S14. On the other hand, when the PU 12 judges that a next work is not specified (S22: NO), the PU 12 temporarily ends the series of processes shown in Fig. 2.

[0026] "Support data generation device" FIG. 4 shows a device 50 for generating the support data 14b. The generating device 50 includes a PU 52 and a storage device 54. The PU 52 is a software processing device such as a CPU, a GPU, or a TPU. The storage device 54 may be an electrically non-rewritable non-volatile memory. The storage device 14 may be an electrically rewritable non-volatile memory or a storage medium such as a disk medium. The generating device 50 executes a process of generating the assistance data 14b by the PU 52 executing a program stored in the storage device 54.

[0027] The storage device 54 stores a base map 54a and a part template 54b. The base map 54a is three-dimensional data of a workplace. This three-dimensional data may be data generated by point cloud data of the workplace sensed by a three-dimensional camera. Also, for example, this three-dimensional data may be data generated by 3D CAD.

[0028] The part template 54b is data for identifying each of the general-purpose parts commonly used in a plurality of work sites. The part template 54b includes a part ID for identifying each of the general-purpose parts. For example, when the work site is a substation, the general-purpose parts are a power line protection panel type A, a power line protection panel type B, ..., a control panel type A, a control panel type B, ..., a busbar protection panel type A, ..., etc. The part ID may be a value for identifying the specifications of the general-purpose part, such as the model of the general-purpose part. The part template 54b also includes a label variable for identifying a plurality of interfaces provided in each of the general-purpose parts, and position data for identifying the position of the interface in the general-purpose part. The label variable is linked to the interface to be worked on, which is defined in the work procedure list 14a.

[0029] The part template 54b includes three-dimensional data or two-dimensional data of the generic part. The three-dimensional data may be data generated from point cloud data of the generic part sensed by a three-dimensional camera. For example, the three-dimensional data may be data generated by 3D CAD. The two-dimensional data may be data generated by 2D CAD. The 3D CAD data or 2D CAD data may be data generated in the design stage of the generic product before the generic part is actually placed in the workplace.

[0030] "Support data generation process" A process for generating the support data 14b is shown in Fig. 5. The series of processes shown in Fig. 5 is realized by the PU 52 executing a program stored in the storage device .

[0031] In the series of processes shown in Fig. 5, the PU 52 first acquires a base map 54a (S30). Here, if the base map 54a has not yet been created for an existing work site, the base map 54a is generated as three-dimensional data by newly generating three-dimensional data for the work site using a three-dimensional camera. For a new work site being constructed, the three-dimensional data used in the design of the work site may be used as the base map 54a.

[0032] Next, the PU 52 executes pattern matching between each of the generic parts represented by the base map 54a and generic parts defined by the three-dimensional data or two-dimensional data included in the part template 54b (S32).

[0033] In this pattern matching, for example, the PU 52 first aligns the positions of the generic parts represented by the base map 54a and the generic parts represented by the corresponding part templates 54b. This may be a process in which the PU 52 determines whether or not there is a portion that matches the image represented by the base map 54a by translating, rotating, or the like an image represented by the three-dimensional data or two-dimensional data included in the part template 54b. The PU 52 searches for a portion of the image represented by the base map 54a that matches the image represented by the part template 54b through this process. Then, when the PU 52 finds a matching portion, the PU 52 determines that the matching portion of the image represented by the base map 54a is a generic part identified by the part ID indicated by the part template 54b. The PU 52 executes this series of processes for the images represented by all the part templates 54b. As a result, it is determined that all the generic parts present in the image represented by the base map 54a are generic parts of which part IDs are defined in the part templates 54b. Furthermore, the PU 52 executes the process of S34 at the stage where the generic part represented by the base map 54a and the generic part represented by the corresponding part template 54b are aligned. That is, the PU 52 assigns a label defined in the part template 54b to each of the interfaces of the generic part represented by the base map 54a (S34). Specifically, this process may be a process in which the PU 52 specifies an area of ​​the interface in the three-dimensional image shown by the base map 54a and assigns a label to the area. As a result, for example, in the process of assigning the marking 42 illustrated in FIG. 3, the PU 42 can specify the position of the switch 40 by the assistance data 14b.

[0034] When the process of S34 is completed, the PU 52 accepts input of the use names of the generic parts indicated by the base map 54a via the input unit 60 (S36). As a result, the use names designated by the user are assigned to the generic parts indicated by the base map 54a.

[0035] For example, even if the work site is a substation and the general-purpose part is a "power line protection panel type A," the use name differs depending on the connection destination of the "power line protection panel type A," such as for XX station, XX factory, etc. The user inputs information specifying the connection destination of the "power line protection panel type A" in the work site via the input unit 60.

[0036] When the PU 52 completes the process of S36, the PU 52 temporarily ends the series of processes shown in FIG. "Actions and Effects of the Present Embodiment" When there is a base map 54a of the workshop that is not associated with a label, the PU 52 performs pattern matching between the base map 54a and the three-dimensional data or two-dimensional data of the part template 54b, thereby making it possible to assign labels to each interface of each generic part represented by the base map 54a.

[0037] Here, if the part template 54b does not exist, the work of adding a label to the base map 54a is as follows. That is, a segmentation process is performed to identify generic parts in the image shown by the base map 54a and identify the interfaces that each generic part has. Then, the interfaces are given correct labels. There are multiple generic parts in the workshop. Each generic part has dozens of interfaces.

[0038] Here, it is possible to automatically perform the segmentation work using a trained model such as an existing semantic segmentation model. However, in that case, in the case of switches or the like that have different functions but similar shapes, different things may be mistaken for the same thing. In addition, some things, such as small displays, may not be detected in some cases. Furthermore, there may be an error in the position of the interface determined by a trained model such as a semantic segmentation model. Therefore, regardless of whether a trained model such as a semantic segmentation model is used, manual work is required to link the label to the interface indicated by the base map 54a.

[0039] In contrast, in the present embodiment, the generation of the part template 54b requires the same number of steps as described above for the segmentation work and the work of linking labels to each interface indicated by the 3D or 2D data representing the generic parts. However, after the part template 54b is generated, the labels can be linked by pattern matching between the generic parts in the base map 54a and the 3D or 2D data included in the part template 54b. Therefore, when the same generic parts are used in different workshops or when the layout of generic parts in a workshop is changed, the number of steps is significantly reduced.

[0040] In addition, the generic parts in the workshop may be modified on-site. In that case, the part template 54b may be generated using CAD data or the like generated in the design stage of the generic part to be modified. However, if accurate CAD data is not generated in the design stage of the modified generic part and if point cloud data cannot be acquired by a 3D camera for the modified generic part, the part template 54b for the modified generic part cannot be generated. In that case, after the process shown in FIG. 5 is performed using the base map 54a for the generic part before modification, the difference between the generic part before modification and the generic part before modification may be manually corrected by a person. That is, for example, when an interface is added by modification, the newly provided interface may be manually labeled by specifying the added position. Also, for example, when the position of the interface is changed, the label may be manually specified by specifying the changed position.

[0041] According to the present embodiment described above, the following actions and effects can be obtained. (1) The PU 12 searches for an image of a generic part that matches the image of the generic part indicated by the base map 54a among the images indicated by the multiple part templates 54b. This makes it possible to automatically extract the part template 54b that corresponds to the generic part indicated by the base map 54a.

[0042] (2) The PU 12 converts the relative coordinates between the component template 54b and the base map 54a so that the image indicated by the component template 54b matches the image of the generic component indicated by the base map 54a. This makes it possible to automatically align the interface indicated by the component template 54b with the interface of the generic component indicated by the base map 54a.

[0043] <Correspondence> The correspondence between the matters in the above embodiment and the matters described in the "Means for solving the problem" column is as follows. Below, the correspondence is shown for each number of the solving means described in the "Means for solving the problem" column. "1,4" The linking process corresponds to the processes of S32 and S34. [2] The assignment process corresponds to the process of S36. [3] It corresponds to PU52 executing the process of S32. The computer corresponds to PU52. [5] The work support device corresponds to the work support device 10. The storage device corresponds to the storage device 14. The execution device corresponds to PU12. The display unit corresponds to the display unit 22. The work identification process corresponds to the process of S14. The work instruction process corresponds to the processes of S16 and S18.

[0044] <Other embodiments> This embodiment can be modified as follows: This embodiment and the following modifications can be combined with each other to the extent that no technical contradiction occurs.

[0045] About Part Templates It is not essential that the part template is a mixture of 3D scan data, 3D CAD data, and 2D CAD data. The part template may be composed of only 3D CAD data and 2D CAD data, for example. The part template may be composed of only 3D CAD data, for example.

[0046] Data included in the part template for identifying each interface is not limited to label and position data. The part template may include data related to the function of the interface, for example. In that case, functional information of the interface to be worked on can be presented to the worker in the work instructions. Also, for example, the part template may include data indicating points to note regarding the handling of the interface. In that case, points to note regarding the interface to be worked on can be presented to the worker in the work instructions.

[0047] "About the pegging process" The pattern matching process is not limited to the process exemplified in the above embodiment. For example, the PU 52 may execute a process using a trained model in which the base map 54a and the shape data included in the part template 54b are input, and a variable specifying a portion of the base map 54a corresponding to the shape data is output. If there is no portion corresponding to the shape data, the trained model outputs a value indicating that there is no portion. Specifically, the PU 52 sequentially inputs each of the part templates 54b into the trained model, thereby determining whether or not the generic part indicated by the part template 54b is included in the image indicated by the base map 54a, and if so, specifying the position of the generic part. If the generic part indicated by the part template 54b is included in the image indicated by the base map 54a, the PU 52 may execute the following process. That is, the PU 52 may align the image of the generic part indicated by the part template 54b and the image of the generic part indicated by the base map 54a by rotating and translating them relative to each other.

[0048] The task of identifying a part that matches a generic part indicated by the base map 54a among the generic parts indicated by the part templates 54b is not necessarily performed by the PU 52. For example, a corresponding part template 54b may be extracted by a person from among a plurality of part templates 54b.

[0049] It is not essential that the process of aligning the positions of the generic parts indicated by the part template 54b and the generic parts indicated by the base map 54a be executed by the PU 52. For example, a person may align the positions of the generic parts indicated by the part template 54b and the generic parts indicated by the base map 54a by manually changing the relative positions and angles between the images indicated by the part template 54b and the images indicated by the base map 54a.

[0050] About Work Order Processing The work instruction process is not limited to the process illustrated in Fig. 3. In other words, it is not limited to the process of covering the interface to be worked on with a predetermined color. The work instruction process may be, for example, a process of displaying an arrow pointing to the interface to be worked on.

[0051] The work instruction process does not necessarily include a process of providing the work instruction contents as visual information. For example, the work instruction process may include a process of providing audio information by the PU 52 operating a speaker.

[0052] "About Completing Work" The process of S20 does not necessarily have to be a process for determining whether or not an input indicating that the task has been completed has been made. For example, in the case of an indicator light that changes color in response to a switch operation, PU 12 may determine that the task has been completed by detecting a change in color of the indicator light from an image captured by camera 20. Also, for example, a camera that generates an image of the worker may be provided, and PU 12 may determine whether or not the task has been completed based on the actions of the worker captured by the camera.

[0053] "About the Workshop" The workplace that is the target of support using the support data is not limited to a substation. For example, it may be a facility that constitutes a communication infrastructure. Furthermore, the workplace is not limited to a workplace that is the target of maintenance and inspection. For example, the workplace may be a manufacturing site for aircraft, vehicles, precision equipment, etc.

[0054] "About Computers and Execution Devices" The computer and the execution device are not limited to those that execute software processing. For example, a dedicated hardware circuit such as an ASIC that executes at least a part of the processing executed in the above embodiment may be included. That is, the computer and the execution device may include a processing circuit having any of the following configurations (a) to (c). (a) A processing circuit including a processing device that executes all of the above processing according to a program, and a program storage device such as a storage device that stores the program. (b) A processing circuit including a processing device and a program storage device that executes part of the above processing according to a program, and a dedicated hardware circuit that executes the remaining processing. (c) A processing circuit including a dedicated hardware circuit that executes all of the above processing. Here, there may be multiple software execution devices including a processing device and a program storage device. Also, there may be multiple dedicated hardware circuits. [Explanation of symbols]

[0055] 10...Work support device 30…Mobile terminal 32…Wearable devices 40…Switch 42…Marking 50…Generation device

Claims

1. A method for generating support data for supporting a task in a workplace using virtual information, comprising the steps of: The support data is data in which a base map, which is three-dimensional data representing a workplace, is associated with a label that identifies an interface indicated by the base map, The workshop is equipped with general-purpose parts that are shared among a plurality of workshops, the general-purpose component is a component having a plurality of the interfaces, A plurality of part templates are stored in a storage device; the part template includes the label and position data of each of a plurality of interfaces included in the corresponding generic part; the location data includes data identifying a location of the interface in the generic component; A method for producing support data, comprising a linking step of linking the label to the interface indicated by the base map by associating the corresponding part template with the generic part indicated by the base map.

2. The method for producing support data according to claim 1 , wherein the linking step includes an assignment step of assigning information indicating a use of the general-purpose parts indicated by the base map in the workshop.

3. The method for producing support data according to claim 1 , wherein the linking step is a step of causing a computer to execute a matching process between the generic parts indicated by the base map and the part templates.

4. 2. The method for producing support data according to claim 1, wherein the part template includes two-dimensional or three-dimensional CAD data representing the generic part.

5. A work support device that supports a work by using the support data produced by the method for producing support data according to claim 1, A storage device, an execution device, and a display unit, The storage device stores work procedure data indicating a procedure of the work and the support data, The execution device is configured to execute a work identification process and a work instruction process; The task identification process is a process of identifying a next task to be performed based on the task procedure data, The work instruction process is a process of displaying work instructions for the interface corresponding to the work identified by the work identification process by operating the display unit based on the assistance data.

Images