Work management device, work management system, and work management method

Abstract

To provide a work management device for setting coordinates serving as a reference into a work space without using a marker, wherein the coordinates are for expressing a position and attitude of a camera which records work.SOLUTION: The present invention relates to a work management device comprising: an input unit for inputting prior information including reference coordinates; a reception unit for receiving a plurality of frames indicating a work status of a worker and line-of-sight information about the worker in the frames; a position determination unit for determining positions of the reference coordinates in the frames using the prior information, the frames, and the line-of-sight information; an extraction unit for extracting a work frame indicating a work scene of the worker using the frames and the line-of-sight information; and an application unit for applying positions relative to the reference coordinates to the work frame by using the work frame extracted by the extraction unit, the frames received by the reception unit, and the positions of the reference coordinates determined by the position determination unit.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a work management device, a work management system, and a work management method for maintenance work of utilization facilities and infrastructure, etc.

Background Art

[0002] In various industrial fields, maintenance work of utilization facilities and infrastructure is continuously carried out as one of the important works. For example, in the railway field, there is maintenance work of railway vehicles. Specific works include daily inspection work on whether there is damage to the fastening parts and welded parts of the bogies that make up the lower part of the vehicle, and inspection work on the presence or absence of scratches, cracks, and foreign matter adhesion after an abnormality such as a collision with an animal occurs. In order to ensure the safety of transported goods and passengers, railway operating companies spend a lot of human and financial costs on these maintenance works. While the number of inspection points increases with the aging of railway vehicles, the number of skilled workers who can perform maintenance is decreasing, and as a result, these costs are expected to increase. Such aging of infrastructure facilities and decrease in the number of skilled workers occur not only in the railway field but also in various fields.

[0003] In order to record the work content including such maintenance work and utilize it for future maintenance inspections, the maintenance inspection results and work content may be saved as a moving image. As a method of recording the work content, the use of a head-mounted display is being promoted. The head-mounted display is equipped with a see-through type display, a camera, etc., and by being worn by the worker, in addition to collecting the first-person perspective moving image of the worker, it becomes possible to track and record the line of sight of the worker and provide information to the worker using a hologram.

[0004] As an example of the use of the saved moving image, for example, at the time of future maintenance inspection, it is used to compare the current state of the inspection point with the moving image to confirm the presence or absence of differences. By comparing with the moving image, higher accuracy in comparison and shorter inspection time are expected compared to comparing with information recorded orally or in documents. In addition, the moving image is also used to instruct workers with little experience in maintenance inspection work. By transmitting the work of skilled workers to workers with little experience, an increase in the number of workers capable of performing maintenance inspection can be expected.

[0005] When recording the moving image, it is also possible to record the position and orientation of the camera that captured the moving image. By recording the position and orientation of the camera and correctly grasping the spatial positional relationship of the work, it becomes possible to confirm whether the work was carried out at an appropriate location. In addition, when the position and orientation of the camera are obtained, they can be used as the target position and orientation of the camera mounted on a drone or a mobile robot. These robots can be used as a substitute for inspection workers, leading to a reduction in human and financial costs.

[0006] In order to make the correspondence relationship between the position and orientation of the camera and the work space known, it is necessary to set a reference coordinate, which is a common coordinate system in the work space, when recording the position and orientation of the camera. As a technique related to the method of setting the reference coordinate, for example, there is a technique such as Patent Document 1. Patent Document 1 discloses "a method in which an image of a marker locked to a first target unit in a real space is captured by an imaging unit, and the positional relationship between the marker and the imaging unit is detected based on the captured image of the marker" (see paragraph 0006). In addition, a prescribed marker is attached to the target unit, an image of the marker is captured by the imaging unit, and the positional relationship between the marker and the imaging unit is detected based on the captured image of the marker (see paragraph 0006 and paragraph 0052).

Prior Art Documents

Patent Documents

[0007]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0008] However, as a problem of Patent Document 1 mentioned above, there is a point that markers are prepared for the environment and objects for setting reference coordinates. It is difficult to install the markers at the assumed positions with high precision. In particular, in the case of a railway vehicle, it is difficult to use markers in an environment where there are few planes where markers can be attached, and the markers are frequently hidden by the vehicle components. Also, in the case of an inspection operation of a part where there is a driving part like a railway vehicle and there are concerns about device failures and safe operation due to entanglement of markers, etc., there may be a case where an object other than the part, that is, the act of attaching a marker cannot be performed.

[0009] An object of the present invention is to provide a work management device, a work management system, and a work management method for setting coordinates serving as a reference in a work space without using markers in order to represent the position and orientation of a camera recording work.

Means for Solving the Problems

[0010] In order to solve the above problems, the present invention includes a reception unit that receives a plurality of frames indicating the work situation of a worker and the line-of-sight information of the worker in the frames, and uses the pre-information, the frames, and the line-of-sight information to determine the position of the reference coordinates in the frames. A positioning unit, An extraction unit that extracts a work frame indicating the work scene of the worker using the frames and the line-of-sight information, and uses the work frame extracted by the extraction unit, the frames received by the reception unit, and the position of the reference coordinates determined by the positioning unit. A work management device comprising an assigning unit that assigns a position with respect to the reference coordinates to the work frame.

[0011] In addition, the present invention provides a work management system including a reception unit that receives a plurality of frames indicating the work situation of a worker and the line-of-sight information of the worker in the frames, a positioning unit that determines the position of a reference coordinate in the frame using the pre-information, the frame, and the line-of-sight information, an extraction unit that extracts a work frame indicating the work scene of the worker using the frame and the line-of-sight information, an assignment unit that assigns the position relative to the reference coordinate to the work frame using the work frame extracted by the extraction unit, the frame received by the reception unit, and the position of the reference coordinate determined by the positioning unit, and an acquisition unit that acquires the line-of-sight information.

[0012] Further, pre-information including reference coordinates is received by an input unit, a plurality of frames indicating the work situation of a worker and the line-of-sight information of the worker in the frames are received by a reception unit, and the position of the reference coordinate in the frame is determined by a positioning unit using the pre-information, the frame, and the line-of-sight information. A work management method, wherein a work frame indicating the work scene of the worker is extracted by an extraction unit using the frame and the line-of-sight information, and the position relative to the reference coordinate is assigned to the work frame by an assignment unit using the work frame extracted by the extraction unit, the frame received by the reception unit, and the position of the reference coordinate determined by the positioning unit.

Advantages of the Invention

[0013] According to the present invention, it is possible to set reference coordinates in a work space without markers, and it is also possible to set reference coordinates for a work space where it is difficult or prohibited to arrange markers.

Brief Description of the Drawings

[0014]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Embodiments for Carrying Out the Invention

[0015] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In each drawing, the same components are denoted by the same reference numerals, and detailed descriptions of overlapping explanations are omitted.

Examples

[0016] With reference to FIGS. 1 and 2, the work management apparatus of this embodiment will be described. FIG. 1 is a diagram showing the schematic configuration of the work management apparatus of this embodiment. FIG. 2 is a flowchart showing the work result management method of this embodiment. In this embodiment, a system including a configuration for giving various information to the work management apparatus and the work management apparatus is referred to as a work management system.

[0017] As shown in FIG. 1, the work management apparatus 100 of this embodiment is configured to include, as main components, an input unit 101, a reception unit 102, a positioning unit 103, an extraction unit 104, and an assignment unit 105. Also, FIGS. 1 and 2 are used to represent the position and orientation 711 of the ToF camera 709 used by the work management apparatus 100 for photographing the inspection point 707 in the work space as seen from the reference coordinate 705. A procedure for setting the necessary reference coordinate 705 will be described.

[0018] The input unit 101 receives prior information 106 including the reference coordinates 705 (step S100). The prior information 106 is information on the reference object 710 in the three-dimensional space, and for example, it may be a 3D model 107 of the reference object 710. This information is transmitted to the input unit 101 and received by the input unit. Also, as the 3D model 107, for example, a CAD (Computer Aided Design) model created when designing the railway vehicle 700 may be used. The 3D model 107 is often composed of mesh information which is points corresponding to vertices and surfaces formed by connecting between the vertices, but it may include other information. In this embodiment, the case where the CAD model of the reference object 710 is used as the 3D model 107 will be described.

[0019] When using the 3D model 107 of the reference object 710 as the prior information 106, the reference coordinates 705 can be set to arbitrary coordinates in the position and orientation seen from the model coordinates of the 3D model 107 by using general CAD software or the like. Here, it is desirable to set the location for setting the reference coordinates 705 at a location that can be easily specified when the user views the reference object 710. For example, when the wheel 701 is regarded as the reference object 710, the center of the wheel 701 or the joint portion between the wheel 701 and the axle 702 can be cited as a location that can be easily specified when the user views the wheel 701. Also, the reference coordinates 705 may be set as the same coordinate system as the model coordinates of the 3D model 107. By using the 3D model, the position and orientation of the camera that has taken a useful image for inspection can be calculated based on the point desired by the user.

[0020] Here, the flow of the work result management method will be described with reference to FIG. 2. In step S100, prior information 106 (e.g., 3D model 107) including the reference coordinates is input to the input unit. In step S101, the reception unit 102 receives a plurality of frames 108 indicating the work status of the worker 703 and the line-of-sight information 109 of the worker 703 in the frames. That is, this line-of-sight information 109 is transmitted to the reception unit 102 and the information is received and accepted.

[0021] Frame 108 indicates information from which 3D information of an object existing in the working space can be obtained. For example, it may be the point cloud 802 recorded by the operator 703 using the head-mounted display 704. Also, like the color image 801, information indirectly obtained by a 3D reconstruction method using SfM (Structure from Motion) such as OpenMVG by Moulon et al.: Open multiple view geometry (International WorkShop on Reproducible Research in Pattern Recognition, 2016, hereinafter referred to as OpenMVG) and OpenMVS: Multi-View Stereo Reconstruction Library by Cernea et al. (https: / / cdcseacave.github.io / openMVS) may be used, which is information from which 3D information of an object existing in the working space can be obtained.

[0022] The line-of-sight information 109 may be information regarding any location in the space that the operator 703 was looking at during work. For example, a 2D fixation point 804 is conceivable.

[0023] In step S102, the positioning unit 103 determines the position 110 of the reference coordinates of the frame 108 with the pre-information 106, the frame 108, and the line-of-sight information 109 as inputs.

[0024] With reference to FIG. 3, the process performed in the positioning unit 103 will be described. The positioning unit 103 includes a 3D fixation point calculator 300 that calculates a 3D fixation point 301 using the 2D fixation point 804 and the point cloud 802. In the 3D fixation point calculator 300, a ray connecting the 2D fixation point 804 and the origin of the coordinates of the ToF camera 709 is calculated. When the ray intersects a point included in the point cloud 802, the intersecting point, or a point where the ray intersects the surface created using a surface reconstruction method with the point cloud 802 as an input, is output as the 3D fixation point 301 corresponding to the 2D fixation point 804. For the surface reconstruction method, for example, the Poisson Surface Reconstruction method is used.

[0025] Further, the positioning unit 103 has a gaze point selector 305 that takes the 2D gaze point 804 as an input and outputs the selection time 306 when the reference object 710 for setting the reference coordinates 705 is gazed at.

[0026] In the gaze point selector 305, in S305-1 of Step 1, first, the time i when the 2D gaze point 804 is obtained for the first time is acquired. Next, in S305-2, a set A having the 2D gaze points 804 recorded within t seconds before and after the time i as elements is acquired with the time i as the center. Here, the t seconds may be changed depending on the work. For example, it may be the number of seconds specified when an instruction is given to the operator to gaze at the reference object for a certain number of seconds before or after the work. When the gaze information exists in a specific part for a certain period of time, by extracting the work frame, the inspection work efficiency can be improved and the necessary information can be accurately extracted.

[0027]

Number

[0028] Next, in S305-3, it is determined whether all the elements of the set A exist within a specific range centered on the 2D gaze point at time i. The specific range may be, for example, a rectangle centered on the 2D gaze point at time i as shown in FIG. 2. Also, as a method other than the method of reflecting whether all the elements exist, for example, an average value or a median value for all the elements of the set A may be calculated, and a method of determining whether these values exist within the specific range may be used.

[0029]

Number

[0030] Then, in S305-4, the selection time 306 is output as the time when the reference object 710 for setting the reference coordinates 705 is gazed at. In this process, there may be a case where a plurality of 2D gaze points satisfying FIG. 2 are obtained. When a plurality are obtained, for example, the selection time 306 of one of them may be output. Also, the average of the plurality of 2D gaze points may be calculated, and the selection time 306 of the point closest to the average may be output.

[0031] In addition, there may be a case where the act of gazing at the reference object 710 before and after the operation and the actual operation are continuously performed. At this time, there may be a case where a 2D gaze point satisfying Equation 2 is obtained by the act of gazing during the actual operation. In order to distinguish the 2D gaze point satisfying Equation 2 by the act of gazing during the actual operation from the 2D gaze point satisfying Equation 2 by the act of gazing at the reference object 710, for example, the selection time 306 of one point at the time closest to the start time of recording or the end time of recording may be output.

[0032] Next, the positioning unit 103 has a reference coordinate setter 307 that determines the position of the reference coordinate 705 with the selection time 306, the prior information 106, and the point cloud 802 as inputs. In the reference coordinate setter 307, in S307-1 of Step 1, first, the 3D gaze point 301 (Ps) recorded at the selection time 306 and the point cloud 802 (PCs) are selected.

[0033] Then, in S307-2, a matrix T representing the coordinates indicating the initial position of the prior information 106 as seen from the camera coordinates Cs of the selected point cloud 802 (PCs) is obtained as in Equation 3 by using the selected 3D gaze point 301 (Ps). Here, I is a 3×3 identity matrix.

[0034]

Equation

[0035] In S307-3, Equation 3 is used to move the prior information 106 to the initial position as in Equation 4.

[0036]

Equation

[0037] In S307-4, alignment is performed using the prior information 106 moved to the initial position and the selected point cloud 802 (PCs), and a matrix T representing the coordinates indicating the position of the reference object 710 after alignment as seen from the coordinates indicating the initial position of the prior information 106 is obtained.

[0038] For alignment, for example, the ICP (Iterative Closest Point) method, which is an algorithm commonly used for aligning two point clouds, may be used. When the initial position of the pre-information 106 at the time of execution is far from the position of the point cloud of the reference object 710 within the selected point cloud 802 (PCs), there is a possibility of performing incorrect alignment. Therefore, as described above, by using Equation 3 to move the pre-information 106 to the spatial position where the reference object 710 was being gazed at, there is an advantage that correct alignment can be performed when using the ICP method. By further moving the pre-information 106 that has been moved to the initial position using the matrix T, the pre-information 106 can be moved to the position of the point cloud of the reference object 710 within the selected point cloud 802 (PCs).

[0039] By using Equation 3 and the matrix T, a matrix representing the coordinates indicating the final position of the pre-information 106 as seen from the camera coordinates Cs of the selected point cloud 802 (PCs) is obtained as in Equation 5.

[0040]

Equation

[0041] Here, the coordinates indicating the final position of the pre-information 106 can be regarded as the reference coordinates 705 set for the pre-information 106.

[0042] That is, in S307-4, obtaining the coordinates indicating the final position of the pre-information 106 corresponds to setting the reference coordinates 705 necessary for expressing the position and orientation 711 of the ToF camera 709 used for imaging at the inspection point 707 as seen from the reference coordinates 705. Then, a matrix representing the position and orientation of the camera coordinates where the selected point cloud 802 (PCs) was imaged as seen from the reference coordinates 705 is obtained. The matrix T indicates the position 110 of the reference coordinates. The reference coordinates can be set by such a matching process for position adjustment.

[0043] The extraction unit 104 extracts a work frame 111 showing the work scene of the operator 703 by using the frame 108 and the gaze information 109 (S103). As the work frame 111 showing the work scene of the operator 703, for example, the frame 108 recorded at the selection time 306 when selecting the 2D gaze point 804 extracted by the same process as the process performed by the gaze point selector 305 of the positioning unit 103 may be extracted. Note that the work scene is the scene that the worker gazes at after setting the reference coordinates, and is the time during work when obtaining the gaze information.

[0044] Also, instead of the 2D gaze point 804 in the gaze point selector 305, using the 3D gaze point 301 calculated by the 3D gaze point calculator 300, among the processes performed by the gaze point selector 305, the frame 108 recorded at the selection time 306 when selecting the 3D gaze point 301 extracted by the process of expanding the specific range in step S305-2 three-dimensionally may be extracted.

[0045] The assignment unit 105 assigns the position 112 of the work frame with respect to the reference coordinates 705 by using the work frame 111 showing the work scene of the operator, the position 110 of the reference coordinates, and the frame 108. Here, an example using the point cloud 802 in the frame 108 is described. First, the assignment unit 105 takes the point cloud 802 as input and calculates the camera coordinates and the relative camera position and orientation. For the output of the camera coordinates and the relative camera position and orientation, for example, Open3d-SLAM (https: / / open3d-slam.readthedocs.io / en / latest / ) is used. The camera coordinates are expressed based on the ToF sensor coordinates that captured the first acquired point cloud PCi among the point cloud 802, for example. Also, the relative camera position and orientation expresses the position and orientation of the other camera coordinates when viewed from each of the camera coordinates. For example, the relative camera position and orientation between the i-th camera coordinate Ci and the j-th camera coordinate Cj among the camera coordinates is, for example, a 4x4 homogeneous transformation matrix.

[0046] An example in which the point cloud 802 is used as an input has been described. However, in addition to this example, for instance, in addition to the point cloud 802, or independently, the output angular velocity and acceleration of the color image 801 or the IMU 806 may be used as inputs, and the camera coordinates and the relative camera position and orientation may be calculated using, for example, a combined method of SLAM (Simultaneous Localization and Mapping), SfM (Structure from motion), and MVS (Multi View Stereo). By inputting the point cloud 802, the positioning unit 103 can perform positioning. Also, by inputting the angular velocity and acceleration, the positioning unit 103 can perform positioning.

[0047] Next, the assigning unit 105 obtains a matrix T indicating the position and orientation of the camera coordinates as seen from the reference coordinates 705 using the position 110 of the reference coordinates output by the positioning unit 103 and the relative camera orientation, as shown in Equation 6.

[0048]

Equation

[0049] Finally, the assigning unit 105 selects a matrix T indicating the position and orientation of the camera coordinates corresponding to the work frame 111 extracted by the extraction unit 104 and outputs it as the position 112 of the work frame with respect to the reference coordinates.

[0050] As in this embodiment, by setting the reference coordinates 705 based on the reference object 710 in the work space using the line-of-sight information 109, the prior information 106, and the frame 108, it becomes possible to set the reference coordinates 705 without using a marker.

[0051] Next, referring to FIGS. 4, 5, and 6, the operations, problems, and variables to be used for Application Example 1 will be described. FIGS. 4 and 5 are diagrams for explaining the operations to which Application Example 1 is applicable, and in this description, the inspection operation of a railway vehicle will be described as an example. FIG. 4 shows an inspection operation with a head-mounted display 704 attached, and shows an operation in which an operator 703 visually inspects a railway vehicle 700. FIG. 5 shows the position and orientation as seen from the reference coordinates of the ToF camera used for photography at the inspection location.

[0052] Examples of the inspection operations include daily confirmation operations to check for damage to the fastening parts and welded parts of the bogies that make up the lower part of the vehicle, and confirmation operations for the presence or absence of scratches, cracks, and foreign object attachments after an abnormality such as a collision with an animal has occurred. The operator inspects the presence or absence of abnormalities in the wheels 701, axles 702 of the railway vehicle 700, and parts 706 under the bogie of the railway vehicle 700 that have been transported into the vehicle maintenance yard, which is the work space. The operator also checks the working conditions, such as what state the inspection operation is in and what the progress is like.

[0053] As the imaging camera, which is an acquisition unit for acquiring the gaze information 109, various commercially available products can be used. In this embodiment, the head-mounted display 704 will be described as an example. The operator 703 wears the head-mounted display 704 and performs the inspection operation. While repeatedly moving to the inspection location, the operator 703 carries out the inspection operation.

[0054] The head-mounted display 704 is equipped with a color camera 708 and a ToF camera 709 (Time of Flight), and records the color image 801 and the point cloud 802 taken at the inspection location 707. The head-mounted display 704 uses a 2D gaze point calculator 805 to calculate the 2D gaze point 804 on the color image 801 from the eye camera 803 that captures the movement of the operator's eyeballs and the captured image of the eye camera 803. The calculation of the 2D gaze point 804 by the 2D gaze point calculator 805 can be realized by a function built into, for example, the commercially available head-mounted display Hololens2 (https: / / www.microsoft.com / ja-jp / hololens).

[0055] In addition, as an operation performed with the head-mounted display 704 attached, for example, an operation useful for instructing an operator 703 to gaze at a work location carefully for confirmation, such as an inspection operation, is suitable.

[0056] In addition, the head-mounted display 704 includes an IMU 806 which is an Inertial Measurement Unit shown in FIG. 6, and the angular velocity and acceleration of the head-mounted display 704 can be obtained. The color image 801, point cloud 802, and 2D gaze point 804 in FIG. 6 are acquired in a state where the recording times match, and are collected as N pieces of data respectively.

[0057] However, in the vehicle maintenance yard which is the work space, the position and orientation of the ToF camera 709 photographed at these inspection locations 707 cannot be measured. Therefore, as shown in FIG. 5, by defining the position and orientation 711 of the ToF camera 709 used for photographing at the inspection location 707 as viewed from the reference coordinate 705, which exists in the work space and is a reference object (hereinafter referred to as the reference object 710), it is recorded where in the work space the inspection location 707 exists.

[0058] In this embodiment, the reference coordinate 705 necessary for expressing the position and orientation 711 of the ToF camera 709 used for photographing at the inspection location 707 as viewed from the reference coordinate 705 in the work space is set in this way. When the position and orientation 711 of the ToF camera 709 used for photographing at the inspection location 707 as viewed from the reference coordinate is known, since the relative position and orientation between the color camera 708, which is another camera, and the ToF camera 709 can be made known by using the external parameters at the time of creating the head-mounted display or a general estimation method for external parameters, it becomes possible to calculate the position and orientation 711 as viewed from the reference coordinate 705 also for the color camera 708.

[0059] In this embodiment, the inspection work of the railway vehicle 700 will be described. However, even for work in other fields, the present invention can be applied when it is necessary to record at which position in the work space the inspection point 707 exists. Examples of work in other fields include daily inspection work in plants and power plants.

Embodiment

[0060] In the first embodiment, a method of setting the reference coordinates 705 without a marker was described by setting the reference coordinates 705 for the reference object 710 in the work space using the line-of-sight information 109, the prior information 106, and the frame 108.

[0061] Since the main purpose of the work management device 100 is recording, there may be a configuration for recording the position of the work frame with respect to the reference coordinates 705 obtained by setting the reference coordinates 705, and an output method including visualization for the recorded information.

[0062] Therefore, next, with reference to FIG. 7, a configuration of the work management device 100 of the first embodiment will be described, which includes a storage unit 400 that stores the position 112 of the work frame with respect to the reference coordinates 705 and the work frame 111, and an output unit 403 that outputs a specified work frame 402 among the work frames 401 stored in the storage unit 400.

[0063] The storage unit 400 stores the position 112 of the work frame with respect to the reference coordinates 705 and the work frame 111. Since the position 112 of the work frame with respect to the reference coordinates 705 has a correspondence with the work frame 111, when storing, it receives the correct correspondence relationship and stores the work frame 111 with the position 112 of the work frame with respect to the reference coordinates 705 as the work frame 401.

[0064] The output unit 403 outputs the specified work frame 402 from among the work frames 401 stored in the storage unit 400. The output may be, for example, outputting the stored work frame 401 in numerical notation or presenting it to the user using a GUI (Graphic User Interface) or the like.

[0065] Further, an example of visualizing the position of the work frame with respect to the reference coordinates 705 and the reference coordinates attached to the work frame 401 stored in the storage unit 400 is shown in FIG. 8. As shown in FIG. 8, the position of the work frame with respect to the reference coordinates attached to the stored work frame 401 can be visualized by a GUI capable of three-dimensional representation. Further, by visualizing the coordinate system corresponding to the origin at the time of visualization, it becomes possible to visually recognize the spatial relative positional relationship between the reference coordinates 705 and the position of the work frame with respect to the reference coordinates attached to the stored work frame 401.

Example

[0066] In the first and second embodiments, the method of setting the reference coordinates 705 without a marker and the method of recording and outputting the information after the setting have been described by setting the reference coordinates 705 for the reference object 710 in the work space using the line-of-sight information 109, the prior information 106, and the frame 108.

[0067] By using the reference coordinates 705 set by the work management device, the position 112 of the work frame with respect to the reference coordinates 705 can be obtained. Further, when another worker or an image-capturing robot, etc. knows the relative positional relationship between the reference coordinates 705 and itself, on the reference coordinates 705, the relative positional deviation amount between the current position of another worker or the image-capturing robot and the position 112 of the work frame can be grasped. When the positional deviation amount can be grasped, since the movement amount to the position where the work frame 111 was acquired can be calculated, it is possible to instruct another worker or the image-capturing robot to move to the position where the past work was performed. Then, for example, the same work by another worker or image capturing from the same position by the image-capturing robot can be realized.

[0068] Next, with reference to FIG. 9, a configuration of the work management device 100 according to the first embodiment will be described, which includes a usage information input unit 602 that receives usage information 601 of an operator or a robot 600, and a movement amount calculation unit 604 that calculates a movement amount 603 of the operator or the robot 600 using the usage information 601 and the position 112 of the work frame with respect to the reference coordinates as inputs.

[0069] The usage information input unit 602 receives the usage information 601 of the operator or the robot 600. The usage information 601 may be information indicating the current position of the operator or the robot 600 with respect to the reference coordinates 705. For example, when the operator is equipped with a head-mounted display 704, it is possible to indicate the position of the operator himself / herself by using SLAM described in the first embodiment. Also, by performing the processes shown by the positioning unit 103 and the assigning unit 105, the position of the operator himself / herself with respect to the reference coordinates 705 can be obtained. In the case of the robot as well, when it is equipped with a sensor corresponding to the head-mounted display 704, the current position of the robot itself can be obtained by the same process.

[0070] Also, the information indicating the current position of the operator or the robot 600 with respect to the reference coordinates 705 may be expressed, for example, using a 4x4 matrix similar to the numbers 3, 5, 6.

[0071] The movement amount calculation unit 604 calculates the movement amount 603 of the operator or the robot 600 as shown in Equation 7, taking the usage information 601 and the position 112 of the work frame with respect to the reference coordinates as inputs.

[0072]

Equation

[0073] This matrix T is a matrix indicating the current position of the operator or the robot 600 with respect to the reference coordinate 705 in the usage-time information 601. And this matrix T is the position of the work frame 111 as seen from the current position of the operator or the robot 600 with respect to the reference coordinate 705. For example, when it is desired to move the operator or the robot 600 to the position 112 of the work frame with respect to the reference coordinate, this can be achieved by giving the operator or the robot 600 a movement amount 603 such that this matrix becomes the identity matrix.

[0074] FIG. 10 is an example of the hardware of an information processing apparatus (computer) used to realize the work management apparatus 100 having the configurations shown in the respective embodiments. As shown in FIG. 10, the illustrated information processing apparatus 1000 includes a processor 1001, a main storage device 1002, an auxiliary storage device 1003, an input device 1004, an output device 1005, and a communication device 1006. These are communicably connected to each other via communication means such as a bus (not shown).

[0075] Note that the information processing apparatus 1000 may be a part of another system, such as a microcomputer mounted on a robot. Also, the information processing apparatus 1000 may be realized using virtual information processing resources such as a cloud server provided by a cloud system.

[0076] The processor 1001 is configured using, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an FPGA (Field-Programmable Gate Array), a GPU (Graphics Processing Unit), an AI chip, or the like.

[0077] The main storage device 1002 is a device that stores programs and data, and is, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a non-volatile memory (NVRAM (Non Volatile RAM)), or the like.

[0078] The auxiliary storage device 1003 is, for example, a hard disk drive, an SSD (Solid State Drive), an optical storage device (CD (Compact Disc), DVD (Digital Versatile Disc), etc.), a storage system, an IC card, a recording medium reading / writing device such as an SD card or an optical recording medium, a storage area of a cloud server, etc. Programs and data can be read into the auxiliary storage device 1003 via a recording medium reading device or the communication device 1006. Programs and data stored in the auxiliary storage device 1003 are read into the main storage device 1002 at any time.

[0079] The input device 1004 is an interface that receives external input, and is, for example, a keyboard, a mouse, a touch panel, a card reader, a voice input device, etc.

[0080] The output device 1005 is an interface that outputs various types of information such as the progress of processing and the processing results. The output device 1005 is, for example, a display device (liquid crystal monitor, LCD (Liquid Crystal Display), graphics card, etc.) that visualizes the above various types of information, a device (voice output device (speaker, etc.)) that vocalizes the above various types of information, a device (printing device, etc.) that characterizes the above various types of information. Incidentally, for example, the information processing device 1000 may be configured to input and output information to and from other devices via the communication device 1006.

[0081] The communication device 1006 is a device that realizes communication with other devices. The communication device 1006 is a wired or wireless communication interface that realizes communication with other devices via communication means such as a LAN (Local Area Network), a WAN (Wide Area Network), or the Internet, and is, for example, a NIC (Network Interface Card), a wireless communication module, a USB (Universal Serial Interface) module, a serial communication module, etc.

[0082] The above-described input unit 101, reception unit 102, positioning unit 103, extraction unit 104, attachment unit 105, memory unit 400, output unit 403, and usage-time information input unit 602 are realized, for example, when the processor 1001 of the information processing apparatus 1000 reads and executes a program stored in the main storage device 1002.

[0083] Further, the main storage device 1002 and the auxiliary storage device 1003 store advance information 106, 3D model 107, frame 108, line-of-sight information 109, position of the reference coordinates 110, work frame 111, position of the work frame with respect to the reference coordinates 112, 3D fixation point 301, camera coordinates 303, relative camera position and orientation 304, selection time 306, work frame 401 to be stored, designated work frame 402, usage-time information 601, movement amount 603, color image 801, point cloud 802, 2D fixation point 804.

[0084] Each of the above-described configurations, functions, processing units, processing means, etc. may be realized in hardware by designing a part or all of them, for example, with an integrated circuit, or may be realized in software by a processor interpreting and executing a program that realizes each function.

[0085] Information such as programs, tables, files, etc. that realize each function can be stored in a storage device such as a memory, hard disk drive, SSD (Solid State Drive), or a recording medium such as an IC (Integrated Circuit) card, SD card, or DVD (Digital Versatile Disc).

[0086] Note that the present invention is not limited to the above-described embodiments, and various modifications are included. For example, in the above embodiments, the gondola car has been described, but other objects may also be used. Further, the above embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. Also, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Further, for a part of the configuration of each embodiment, addition, deletion, or replacement with other configurations is possible.

Explanation of Signs

[0087] 100 ··· Work management device 101 ··· Input unit 102 ··· Reception unit 103 ··· Position determination unit 104 ··· Extraction unit 105 ··· Assignment unit 106 ··· Pre-information 107 ··· 3D model 108 ··· Frame 109 ··· Line-of-sight information 110 ··· Position of reference coordinates 111 ··· Work frame 112 ··· Position of work frame 300 ··· 3D gaze point calculator 301 ··· 3D gaze point 302 ··· Camera position and orientation calculator 303 ··· Camera coordinates 304 ··· Relative camera position and orientation 305 ··· Gaze point selector 306 ··· Selection time 307 ··· Reference coordinate setter 400 ··· Storage unit 401 ··· Work frame to be stored 402 ··· Specified work frame 403 ··· Output unit 600 ··· Robot 601 ··· Information at the time of use 602 ··· Utilization time information input unit 603 ··· Movement amount 700 ··· Railway vehicle 701 ··· Wheel 702 ··· Axle 703 ··· Operator 704 ··· Head-mounted display 705 ··· Reference coordinates 706 ··· Parts under the bogie 707 ··· Inspection point 708 ··· Color camera 709 ··· ToF camera 710 ··· Reference object 711 ··· Position and orientation 801 ··· Color image 802 ··· Point cloud 803 ··· Eye camera 804 ··· 2D fixation point 805 ··· 2D fixation point calculator 806 ··· IMU 1000 ··· Information processing device 1001 ··· Processor 1002 ··· Main memory device 1003 ··· Auxiliary storage device 1004 ··· Input device 1005 ··· Output device 1006 ··· Communication device

Claims

1. An input unit that inputs preliminary information including reference coordinates, A reception unit that receives a plurality of frames indicating the working status of the operator and the line-of-sight information of the operator in the frames, A positioning unit that determines the position of the reference coordinates in the frame using the preliminary information, the frame, and the line-of-sight information, An extraction unit that extracts a work frame indicating the work scene of the operator using the frame and the line-of-sight information, A work management device comprising: an assignment unit that assigns a position relative to the reference coordinates to the work frame using the work frame extracted by the extraction unit, the frame received by the reception unit, and the position of the reference coordinates determined by the positioning unit.

2. The work management device according to claim 1, wherein the preliminary information is a 3D model of an object.

3. The work management device according to claim 1, wherein the frame is at least one point cloud.

4. The work management device according to claim 1, wherein in the extraction unit, when the line-of-sight information exists in a specific part for a certain period of time, the work frame is extracted.

5. The work management device according to claim 1, wherein in the positioning unit, when the line-of-sight information exists in a specific part for a certain period of time, based on the line-of-sight information, the preliminary information is moved to perform a matching process between the preliminary information and the frame, and the position of the reference coordinates is determined.

6. The work management device according to claim 5, wherein in the positioning unit, alignment is performed using the preliminary information moved to the initial position and the selected point cloud, and the preliminary information is moved to the spatial position where the reference object was being gazed at.

7. The work management device according to claim 1, comprising a usage information input unit that receives usage information of the operator or the robot, and a movement amount calculation unit that inputs the usage information and the position relative to the reference coordinates and calculates the movement amount of the operator or the robot.

8. The work management device according to claim 1, comprising a storage unit that stores the position relative to the reference coordinates and the work frame, and an output unit that outputs the work frame with the position attached to the specified reference coordinates among the work frames with the position attached to the reference coordinates stored in the storage unit.

9. ​ An input unit that inputs prior information including reference coordinates; A reception unit that receives a plurality of frames indicating the working status of an operator and the line-of-sight information of the operator in the frames; A positioning unit that determines the position of the reference coordinates in the frame using the prior information, the frame, and the line-of-sight information; An extraction unit that extracts a work frame indicating the work scene of the operator using the frame and the line-of-sight information; An assignment unit that assigns a position relative to the reference coordinates to the work frame using the work frame extracted by the extraction unit, the frame received by the reception unit, and the position of the reference coordinates determined by the positioning unit; A work management system comprising an acquisition unit that acquires the line-of-sight information.

10. Receiving prior information including reference coordinates by an input unit; Receiving a plurality of frames indicating the working status of an operator and the line-of-sight information of the operator in the frames by a reception unit; Determining the position of the reference coordinates in the frame by a positioning unit using the prior information, the frame, and the line-of-sight information; Extracting a work frame indicating the work scene of the operator by an extraction unit using the frame and the line-of-sight information; A work management method of assigning a position relative to the reference coordinates to the work frame by an assignment unit using the work frame extracted by the extraction unit, the frame received by the reception unit, and the position of the reference coordinates determined by the positioning unit.

Images