Work management device, work management system, and work management method

Abstract

An objective of the present invention is to provide a work management device for setting coordinates serving as a reference into a work space without using a marker, wherein the coordinates are for expressing the position and attitude of a camera recording work.　The present invention relates to a work management device comprising: an input unit 101 for inputting prior information 106 including reference coordinates; a reception unit 102 that receives a plurality of frames 108 indicating the work status of a worker and line-of-sight information 109 about the worker in the frames; a position determination unit 103 that determines the position 110 of the reference coordinates in the frame using the prior information, the frames, and the line-of-sight information; an extraction unit 104 that extracts a work frame 111 showing a work scene of the worker using the frames and the line-of-sight information; and an application unit 105 that applies a position 112 relative to the reference coordinates to the work frame using the work frame extracted by the extraction unit, the frames received by the reception unit, and the position of the reference coordinates determined by the position determination unit.

Description

Work management device, work management system, and work management method

[0001] The present invention relates to a work management device, a work management system, and a work management method for maintenance work of utilization facilities and infrastructure.

[0002] In various industrial sectors, maintenance work on facilities and infrastructure is an important ongoing task. For example, in the railway industry, there is the maintenance of railway vehicles. Specific tasks include daily checks for damage to the fastenings and welds of the bogies that make up the undercarriage of the vehicles, as well as checking for scratches, cracks, and foreign objects after abnormal events such as collisions with animals. To ensure the safety of transported cargo and passengers, railway operators expend a great deal of human and financial costs on this maintenance work. As railway vehicles age, the number of inspection points increases, while the number of skilled workers available to perform maintenance decreases, and as a result, these costs are expected to increase. This deterioration of infrastructure and the decrease in the number of skilled workers is occurring in various fields, not just the railway industry.

[0003] In order to record the details of work, including such maintenance work, and to utilize them for future maintenance inspections, the results of the maintenance inspection and the details of the work may be saved as video images. Head-mounted displays are increasingly being used as a method of recording work details. Head-mounted displays are equipped with a see-through display, a camera, and other components. Wearing a head-mounted display allows for the collection of first-person perspective video images of the worker, as well as tracking and recording the worker's line of sight and providing information to the worker using holograms.

[0004] One example of how saved video images can be used is to compare the current state of the inspection area with the video images during future maintenance inspections to check for any differences. Comparing with video images is expected to improve the accuracy of the comparison and reduce the time required for inspections compared to comparing with oral or written information. Video images can also be used to teach less experienced workers how to perform maintenance and inspection work. By teaching less experienced workers how to perform the work of experienced workers, it is expected that the number of workers available to perform maintenance and inspections will increase.

[0005] When recording video, it is also possible to record the position and orientation of the camera that captured the video. By recording the camera's position and orientation and correctly understanding the spatial relationship of the work, it becomes possible to confirm whether the work was performed in the appropriate location. Furthermore, once the camera's position and orientation are obtained, they can be used as the target position and orientation for cameras mounted on drones or mobile robots. These robots can be used in place of inspection workers, leading to reduced human and financial costs.

[0006] In order to make the correspondence between the camera's position and orientation and the workspace known, it is necessary to set a reference coordinate, which is a common coordinate system within the workspace, when recording the camera's position and orientation. Patent Document 1, for example, discloses a technique for setting the reference coordinate. Patent Document 1 discloses a "method in which an image of a marker attached to a first target unit in real space is captured by an imaging unit, and the positional relationship between the marker and the imaging unit is detected based on the captured image of the marker" (see paragraph 0006). Furthermore, a specified marker is attached to the target unit, an image of the marker is captured by an imaging unit, and the positional relationship between the marker and the imaging unit is detected based on the captured image of the marker (see paragraphs 0006 and 0052).

[0007] Japanese Patent Application Laid-Open No. 2020-149140

[0008] However, one of the problems with the above-mentioned Patent Document 1 is that markers are provided in the environment or on objects to set reference coordinates. It is difficult to install markers at intended positions with high accuracy. It is particularly difficult to use markers in environments such as railway vehicles, where there are few flat surfaces on which markers can be attached and where markers are frequently hidden by vehicle components. Furthermore, when inspecting parts of railway vehicles that have driving parts and where entanglement of the marker could cause equipment failure or concerns about safe operation, it may be impossible to attach markers to objects other than the components.

[0009] An object of the present invention is to provide a work management device, a work management system, and a work management method that set reference coordinates within a work space to represent the position and orientation of a camera recording work without using markers.

[0010] In order to solve the above problems, the present invention provides a work management device that includes a reception unit that receives multiple frames showing a worker's work status and the worker's gaze information in the frames; a position determination unit that determines the position of the reference coordinate in the frames using the prior information, the frames, and the gaze information; an extraction unit that extracts a work frame showing the worker's work scene using the frames and the gaze information; and an assignment unit that assigns a position relative to the reference coordinate to the work frame using the work frame extracted by the extraction unit, the frame accepted by the reception unit, and the position of the reference coordinate determined by the position determination unit.

[0011] The present invention also provides a work management system that includes a reception unit that receives a plurality of frames showing the work status of a worker and gaze information of the worker in the frames; a position determination unit that determines the position of the reference coordinate in the frames using the prior information, the frames, and the gaze information; an extraction unit that extracts a work frame showing the work scene of the worker using the frames and the gaze information; an assignment unit that assigns a position relative to the reference coordinate to the work frame using the work frame extracted by the extraction unit, the frame accepted by the reception unit, and the position of the reference coordinate determined by the position determination unit; and an acquisition unit that acquires the gaze information.

[0012] The work management method further includes receiving advance information including reference coordinates at an input unit, receiving a plurality of frames showing the work status of a worker and line-of-sight information of the worker in the frames at a reception unit, determining the position of the reference coordinates in the frames at a position determination unit using the advance information, the frames, and the line-of-sight information, extracting a work frame showing the work scene of the worker at an extraction unit using the frame and the line-of-sight information, and assigning a position relative to the reference coordinates to the work frame at an assignment unit using the work frame extracted by the extraction unit, the frame accepted by the reception unit, and the position of the reference coordinates determined by the position determination unit.

[0013] According to the present invention, reference coordinates can be set in a work space without markers, and reference coordinates can be set even in a work space where it is difficult or prohibited to place markers.

[0014] FIG. 1 is a diagram showing a schematic configuration of a work management device according to a first embodiment of the present invention. FIG. 2 is a flowchart showing the processing of the work management device according to the first embodiment of the present invention. FIG. 3 is a diagram showing the processing of a position determination unit. FIG. 4 is a diagram explaining work to which the first embodiment is applied. FIG. 5 is a diagram explaining work to which the first embodiment is applied. FIG. 6 is a diagram explaining variables used in explaining the first embodiment. FIG. 7 is a diagram showing a schematic configuration of a work management device according to a second embodiment of the present invention. FIG. 8 is a diagram explaining an example of visualizing the position of a work frame relative to reference coordinates and reference coordinates assigned to the work frame stored in a memory unit. FIG. 9 is a diagram showing a schematic configuration of a work management device according to a third embodiment of the present invention. FIG. 10 is an example of an information processing device used in the configuration of the embodiments of the present invention.

[0015] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the drawings, the same components are designated by the same reference numerals, and detailed description of overlapping components will be omitted.

[0016] The work management device of this embodiment will be described with reference to Figures 1 and 2. Figure 1 is a diagram showing the schematic configuration of the work management device of this embodiment. Figure 2 is a flowchart showing the work result management method of this embodiment. In this embodiment, a system equipped with the work management device and a configuration for providing various information to the work management device is called a work management system.

[0017] As shown in Fig. 1, the work management device 100 of this embodiment is configured to include, as its main components, an input unit 101, a reception unit 102, a position determination unit 103, an extraction unit 104, and an assignment unit 105. Also, using Figs. 1 and 2, the position and orientation 711 of the ToF camera 709 used by the work management device 100 to photograph an inspection point 707 in the work space is expressed as viewed from a reference coordinate system 705. The procedure for setting the reference coordinate system 705 required for this purpose will be described.

[0018] The input unit 101 receives prior information 106 including reference coordinates 705 (step S100). The prior information 106 is information on the reference object 710 in three-dimensional space, and may be, for example, a 3D model 107 of the reference object 710. This information is transmitted to the input unit 101 and received by the input unit. Furthermore, for example, a CAD (Computer Aided Design) model created when designing the railcar 700 may be used as the 3D model 107. The 3D model 107 is often configured with mesh information, which is points corresponding to vertices and surfaces formed by connecting vertices, but may also include other information. In this embodiment, a case where a CAD model of the reference object 710 is used as the 3D model 107 will be described.

[0019] When the 3D model 107 of the reference object 710 is used as the prior information 106, the user can set any coordinates for the reference coordinates 705 as the position and orientation as viewed from the model coordinates of the 3D model 107 using general CAD software or the like. Here, it is desirable to set the reference coordinates 705 at a location that the user can easily identify when viewing the reference object 710. For example, if the wheel 701 is considered to be the reference object 710, the center of the wheel 701 or the joint between the wheel 701 and the axle 702 may be cited as a location that the user can easily identify when viewing the wheel 701. Furthermore, the reference coordinates 705 may be set in the same coordinate system as the model coordinates of the 3D model 107. By using the 3D model, the position and orientation of the camera that captured an image useful for inspection can be calculated based on a point desired by the user.

[0020] The flow of the work result management method will now be described with reference to Figure 2. In step S100, prior information 106 (e.g., 3D model 107) including reference coordinates is input to the input unit. In step S101, the reception unit 102 receives a plurality of frames 108 showing the work status of a worker 703 and line-of-sight information 109 of the worker 703 in the frames. In other words, this line-of-sight information 109 is transmitted to the reception unit 102, which then receives and accepts the information.

[0021] The frame 108 shows information from which 3D information of an object existing in the workspace can be obtained, and may be, for example, a point cloud 802 recorded by a worker 703 using a head-mounted display 704. Also, as in the color image 801, SfM (Structure from Motion) such as OpenMVG: Open multiple view geometry by Moulon et al. (International Workshop on Reproducible Research in Pattern Recognition, 2016, hereinafter referred to as OpenMVG) and OpenMVS: Multi-View Stereo Reconstruction by Cernea et al. Alternatively, information that can indirectly obtain 3D information of an object present in the workspace may be used, using a three-dimensional reconstruction method using the Library (https: / / cdcseacave.github.io / openMVS) or the like.

[0022] The line of sight information 109 may be information about a location in space that the worker 703 is looking at while working, and may be, for example, a 2D gaze point 804 .

[0023] In step S102, the position determination unit 103 receives the prior information 106, the frame 108, and the line-of-sight information 109 as input, and determines the reference coordinate position 110 of the frame 108.

[0024] 3, the processing performed by the position determination unit 103 will be described. The position determination unit 103 has a 3D gaze point calculator 300 that calculates a 3D gaze point 301 using a 2D gaze point 804 and a point cloud 802. The 3D gaze point calculator 300 calculates a ray connecting the 2D gaze point 804 and the origin of the coordinate system of the ToF camera 709, and if the ray intersects with a point included in the point cloud 802, outputs the point of intersection, or the point of intersection of the ray with a surface created using a surface reconstruction method with the point cloud 802 as input, as the 3D gaze point 301 corresponding to the 2D gaze point 804. For example, the Poisson Surface Reconstruction method is used as the surface reconstruction method.

[0025] The position determination unit 103 also has a gaze point selector 305 that receives the 2D gaze point 804 as an input and outputs a selected time 306 at which the reference object 710 for setting the reference coordinates 705 is gazed upon.

[0026] In Step 1, the gaze point selector 305 first obtains the time i at which the 2D gaze point 804 was obtained in S305-1. Next, in S305-2, a set A is obtained whose elements are the 2D gaze points 804 recorded t seconds before and after time i. Here, t seconds may be changed depending on the task, and may be, for example, the number of seconds specified when an operator is instructed to gaze at a reference object for a certain number of seconds before or after the task. By extracting a task frame when gaze information is present at a specific part for a certain period of time, the efficiency of the inspection task can be improved and the necessary information can be accurately extracted.

[0027]

[0028] Next, in S305-3, it is determined whether all elements of set A exist within a specific range centered on the 2D gaze point at time i. The specific range may be, for example, a rectangle centered on the 2D gaze point at time i, as shown in Equation 2. As a method other than the method of reflecting whether all elements exist, for example, a method of calculating the average value or median value for all elements of set A and determining whether these values ​​exist within the specific range may be used.

[0029]

[0030] Then, in S305-4, the selection time 306 is output as the time when the reference object 710 for setting the reference coordinates 705 was gazed upon. In this process, multiple 2D gaze points that satisfy Equation 2 may be obtained. If multiple 2D gaze points are obtained, the selection time 306 of one of them may be output, for example. Alternatively, the average of the multiple 2D gaze points may be calculated, and the selection time 306 of the one point that is closest to the average may be output.

[0031] Furthermore, there are cases where the act of gazing at the reference object 710 and the actual work are performed consecutively before and after the work. In this case, a 2D gaze point that satisfies Equation 2 may be obtained due to the gaze action that occurs during the actual work. In order to distinguish between a 2D gaze point that satisfies Equation 2 due to the gaze action during the actual work and a 2D gaze point that satisfies Equation 2 due to the gaze action at the reference object 710, for example, the selection time 306 of one point that is closest to the time when recording started or the time when recording ended may be output.

[0032] Next, the position determination unit 103 has a reference coordinate setter 307 that receives the selection time 306, the prior information 106, and the point cloud 802 as input and determines the position of the reference coordinates 705. In step S307-1 of step 1, the reference coordinate setter 307 first selects the 3D gaze point 301 (Ps) and the point cloud 802 (PCs) recorded at the selection time 306.

[0033] Then, in S307-2, the selected 3D gaze point 301 (Ps) is used to obtain a matrix T expressing coordinates indicating the initial position of the prior information 106 as seen from the camera coordinate Cs of the selected point cloud 802 (PCs), as shown in Equation 3, where I is a 3-row, 3-column unit matrix.

[0034]

[0035] In S307-3, the prior information 106 is moved to the initial position using Equation 3 as shown in Equation 4.

[0036]

[0037] In S307-4, alignment is performed using the prior information 106 moved to its initial position and the selected point cloud 802 (PCs), and a matrix T is obtained that represents the coordinates indicating the position of the reference object 710 after alignment as viewed from the coordinates indicating the initial position of the prior information 106.

[0038] For example, the ICP (Iterative Closet Point) method, which is a commonly used algorithm for aligning two point clouds, may be used for the alignment. The ICP method may result in incorrect alignment if the initial position of the prior information 106 at the time of execution is far from the position of the point cloud of the reference object 710 in the selected point cloud 802 (PCs). Therefore, by using Equation 3 as described above to move the prior information 106 to the spatial position where the reference object 710 was being gazed, correct alignment can be achieved when using the ICP method. By further moving the prior information 106, which has been moved to its initial position, using matrix T, the prior information 106 can be moved to the position of the point cloud of the reference object 710 in the selected point cloud 802 (PCs).

[0039] By using Equation 3 and the matrix T, a matrix expressing coordinates indicating the final position of the prior information 106 as viewed from the camera coordinates Cs of the selected point cloud 802 (PCs) is obtained as shown in Equation 5.

[0040]

[0041] Here, the coordinates indicating the final position of the prior information 106 can be regarded as the reference coordinates 705 set in the prior information 106 .

[0042] That is, obtaining coordinates indicating the final position of the preliminary information 106 in S307-4 corresponds to setting the reference coordinates 705 required to express the position and orientation 711 of the ToF camera 709 used to take images at the inspection point 707 as viewed from the reference coordinates 705. Then, a matrix is ​​obtained that expresses the position and orientation of the camera coordinates that captured the selected point cloud 802 (PCs) as viewed from the reference coordinates 705. The matrix T indicates the position 110 of the reference coordinates. The reference coordinates can be set by such a position adjustment matching process.

[0043] The extraction unit 104 uses the frame 108 and the line-of-sight information 109 to extract an operation frame 111 showing the operation scene of the worker 703 (S103). As the operation frame 111 showing the operation scene of the worker 703, for example, the frame 108 recorded at the selection time 306 at which the 2D gaze point 804 extracted by processing similar to the processing performed by the gaze point selector 305 of the position determination unit 103 was selected may be extracted. Note that the operation scene is the scene that the worker gazes at after the reference coordinates are set, and is the time of operation when the line-of-sight information is obtained.

[0044] In addition, the gaze point selector 305 may use the 3D gaze point 301 calculated by the 3D gaze point calculator 300 instead of the 2D gaze point 804, and extract the frame 108 recorded at the selection time 306 at which the 3D gaze point 301 extracted by the process of expanding the specific range into three dimensions in step S305-2 was selected in the process performed by the gaze point selector 305.

[0045] The assignment unit 105 assigns a position 112 of the work frame relative to the reference coordinates 705 using a work frame 111 showing the work scene of the worker, a reference coordinate position 110, and a frame 108. Here, an example is described in which a point cloud 802 is used in the frame 108. The assignment unit 105 first inputs the point cloud 802 and calculates the camera coordinates and the relative camera position and orientation. To output the camera coordinates and the relative camera position and orientation, for example, Open3d-SLAM (https: / / open3d-slam.readthedocs.io / en / latest / ) is used. The camera coordinates are expressed, for example, based on the ToF sensor coordinates that captured the first acquired point cloud PCi among the point cloud 802. Furthermore, the relative camera position and orientation expresses the position and orientation of each camera coordinate when viewed from the other camera coordinate. For example, the relative camera position and orientation between the ith camera coordinate Ci and the jth camera coordinate Cj among the camera coordinates is, for example, a homogeneous transformation matrix of 4 rows and 4 columns.

[0046] Although an example in which the point cloud 802 is input has been described, other than this example, for example, the angular velocity and acceleration output from the color image 801 or the IMU 806 may be input in addition to or independently of the point cloud 802, and the camera position and orientation relative to the camera coordinates may be calculated using a combined method of SLAM (Simultaneous Localization and Mapping) or SfM (Structure from Motion) and MVS (Multi View Stereo). Inputting the point cloud 802 enables the position determination unit 103 to determine the position. Furthermore, inputting the angular velocity and acceleration enables the position determination unit 103 to determine the position.

[0047] Next, the assigning unit 105 uses the camera orientation relative to the reference coordinate position 110 output by the position determining unit 103 to obtain a matrix T indicating the position and orientation of the camera coordinates viewed from the reference coordinate 705 as shown in Equation 6.

[0048]

[0049] Finally, the assigning unit 105 selects a matrix T indicating the position and orientation of the camera coordinates corresponding to the working frame 111 extracted by the extracting unit 104, and outputs it as the position 112 of the working frame relative to the reference coordinates.

[0050] As in this embodiment, by setting the reference coordinates 705 on the reference object 710 in the working space using the line of sight information 109, the prior information 106, and the frame 108, it becomes possible to set the reference coordinates 705 without using a marker.

[0051] Next, the work to which the first embodiment is applied, the tasks, and the variables to be used will be described with reference to Figures 4, 5, and 6. Figures 4 and 5 are diagrams for explaining the work to which the first embodiment is applied, and in this explanation, the inspection work of a railway vehicle will be described as an example. Figure 4 shows the inspection work performed while wearing a head-mounted display 704, and shows the work in which a worker 703 visually inspects a railway vehicle 700. Figure 5 shows the position and orientation of the ToF camera used for photographing at the inspection location as viewed from the reference coordinates.

[0052] Examples of inspection work include daily checks for damage to fastenings and welds on the bogie that make up the underside of the vehicle, and checks for scratches, cracks, and foreign matter after an abnormality such as a collision with an animal. The worker checks for abnormalities in the wheels 701, axles 702, and parts 706 under the bogie of the railcar 700 that have been brought into the vehicle maintenance yard, which is the work space. The worker checks the status of such inspection work, such as the state and progress of the work.

[0053] Various commercially available cameras can be used as the image capturing camera, which is an acquisition unit that acquires the line-of-sight information 109. In this embodiment, a head-mounted display 704 will be used as an example. A worker 703 works while wearing the head-mounted display 704. The worker 703 performs inspection work by repeatedly moving to inspection locations.

[0054] Head-mounted display 704 is equipped with a color camera 708 and a ToF camera 709 (Time of Flight), and records a color image 801 and a point cloud 802 captured at inspection location 707. Head-mounted display 704 uses a 2D gaze point calculator 805 to calculate an eye camera 803 that captures the movement of the worker's eyeballs and a 2D gaze point 804 on color image 801 from the image captured by eye camera 803. When using, for example, Hololens2 (https: / / www.microsoft.com / ja-jp / hololens), a commercially available head-mounted display, the calculation of 2D gaze point 804 by 2D gaze point calculator 805 can be realized by a function built into Hololens2.

[0055] In addition, work performed while wearing the head-mounted display 704 is suitable, for example, for inspection work, which is useful in instructing the worker 703 to look closely to carefully check the work area.

[0056] The head-mounted display 704 also includes an IMU 806, which is an inertial measurement unit shown in Fig. 6, and is capable of obtaining the angular velocity and acceleration of the head-mounted display 704. The color image 801, point cloud 802, and 2D gaze point 804 in Fig. 6 are acquired with the same recording time, and are each collected as N pieces of data.

[0057] However, in the vehicle maintenance workshop, which is the work space, it is not possible to measure the position and orientation of the ToF camera 709 that photographed these inspection points 707. Therefore, as shown in FIG. 5 , an object that exists within the work space and serves as a reference (hereinafter referred to as a reference object 710) is defined as a position and orientation 711 when viewed from a reference coordinate system 705 of the ToF camera 709 that was used to photograph the inspection point 707, and the position of the inspection point 707 within the work space is recorded.

[0058] In this embodiment, reference coordinates 705 are set in this way in order to express the position and orientation 711 of the ToF camera 709 used to capture images at the inspection point 707 within the work space as viewed from the reference coordinates 705. Once the position and orientation 711 of the ToF camera 709 used to capture images at the inspection point 707 as viewed from the reference coordinates is known, the relative position and orientation of the ToF camera 709 and the color camera 708, which is another camera, can be known using external parameters when the head-mounted display is created or a general external parameter estimation method, and therefore it becomes possible to calculate the position and orientation 711 of the color camera 708 as viewed from the reference coordinates 705.

[0059] In this embodiment, an explanation will be given of the inspection work of a railway vehicle 700, but the present invention can also be applied to work in other fields when it is necessary to record the position within the work space where the inspection point 707 is located. An example of work in other fields is daily inspection work at a plant or power plant.

[0060] In the first embodiment, a method for setting the reference coordinates 705 without using a marker has been described, in which the reference coordinates 705 are set on the reference object 710 in the working space using the line of sight information 109, the prior information 106, and the frame 108.

[0061] Since the main purpose of the work management device 100 is recording, the work frame position relative to the reference coordinates 705 obtained by setting the reference coordinates 705, the recording configuration, and an output method including visualization of the recorded information may exist.

[0062] Next, referring to Figure 7, we will explain the configuration of the work management device 100 of Example 1, which is equipped with a memory unit 400 that stores the position 112 of the work frame relative to the reference coordinates 705 and the work frame 111, and an output unit 403 that outputs a specified work frame 402 from the work frame 401 stored in the memory unit 400.

[0063] The storage unit 400 stores the work frame position 112 relative to the reference coordinates 705 and the work frame 111. Since the work frame position 112 relative to the reference coordinates 705 corresponds to the work frame 111, the work frame 111 is received in the correct correspondence when stored, and the work frame 111 with the work frame position 112 relative to the reference coordinates 705 is stored as the work frame 401.

[0064] The output unit 403 outputs a working frame 402 designated by a user from among the working frames 401 stored in the storage unit 400. The output may be, for example, a case in which the stored working frame 401 is output in numerical notation, or may be presented to the user using a GUI (Graphical User Interface) or the like.

[0065] 8 shows an example of visualizing the position of the work frame relative to the reference coordinates 705 and the reference coordinates assigned to the work frame 401 stored in the memory unit 400. As shown in Fig. 8, the position of the work frame relative to the reference coordinates assigned to the stored work frame 401 can be visualized using a GUI capable of three-dimensional representation. Furthermore, by visualizing a coordinate system corresponding to the origin during visualization, it becomes possible to visually confirm the spatial relative positional relationship between the reference coordinates 705 and the position of the work frame relative to the reference coordinates assigned to the stored work frame 401.

[0066] In the first and second embodiments, a method for setting the reference coordinates 705 without a marker by using the line of sight information 109, the prior information 106, and the frame 108 to set the reference coordinates 705 on the reference object 710 in the working space, and a method for recording and outputting the information after setting have been described.

[0067] By using the reference coordinates 705 set by the work management device, the position 112 of the work frame relative to the reference coordinates 705 can be obtained. Furthermore, if another worker, an image capture robot, or the like knows their own positional relationship relative to the reference coordinates 705, the amount of relative positional deviation between the current position of the other worker or the image capture robot and the position 112 of the work frame can be determined on the reference coordinates 705. By knowing the amount of positional deviation, it is possible to calculate the amount of movement to the position where the work frame 111 was acquired, and therefore it is possible to instruct the other worker or the image capture robot to move to the position where the previous work was performed. This then makes it possible, for example, for the same work to be performed by another worker or for the image capture robot to take images from the same position.

[0068] Next, referring to Figure 9, we will explain the configuration of the work management device 100 of Example 1, which is equipped with a usage information input unit 602 that accepts usage information 601 of a worker or robot 600, and a movement amount calculation unit 604 that inputs the usage information 601 and the position 112 of the work frame relative to the reference coordinates and calculates the movement amount 603 of the worker or robot 600.

[0069] The use information input unit 602 receives use information 601 of the worker or robot 600. The use information 601 may be information indicating the current position of the worker or robot 600 relative to the reference coordinates 705. For example, if the worker is equipped with a head-mounted display 704, the SLAM described in the first embodiment can be used to indicate the worker's own position. Furthermore, by performing the processing shown by the position determination unit 103 and the assignment unit 105, the position of the worker relative to the reference coordinates 705 can be obtained. In the case of a robot, if equipped with a sensor equivalent to the head-mounted display 704, the current position of the robot itself can be obtained by similar processing.

[0070] Furthermore, the information indicating the current position of the worker or robot 600 relative to the reference coordinates 705 may be expressed using a 4-row, 4-column matrix, similar to equations 3, 5, and 6, for example.

[0071] A movement amount calculation unit 604 receives the use time information 601 and the position 112 of the work frame relative to the reference coordinates as input, and calculates the movement amount 603 of the worker or robot 600 as shown in Equation 7.

[0072]

[0073] This matrix T is a matrix that indicates the current position of the worker or robot 600 relative to the reference coordinates 705 in the use time information 601. This matrix T is the position of the work frame 111 as seen from the current position of the worker or robot 600 relative to the reference coordinates 705. For example, when it is desired to move the worker or robot 600 to the position 112 of the work frame relative to the reference coordinates, this can be achieved by giving the worker or robot 600 a movement amount 603 that makes this matrix a unit matrix.

[0074] 10 shows an example of hardware of an information processing device (computer) used to realize the work management device 100 having the configuration shown in each embodiment. As shown in Fig. 10, the illustrated information processing device 1000 includes a processor 1001, a main memory device 1002, an auxiliary memory device 1003, an input device 1004, an output device 1005, and a communication device 1006. These are connected to each other so as to be able to communicate with each other via communication means such as a bus (not shown).

[0075] The information processing device 1000 may be a part of another system, such as a microcomputer mounted on a robot, or may be realized using virtual information processing resources such as a cloud server provided by a cloud system.

[0076] The processor 1001 is configured using, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an FPGA (Field-Programmable Gate Array), a GPU (Graphics Processing Unit), an AI chip, etc.

[0077] The main memory device 1002 is a device that stores programs and data, and is, for example, a read-only memory (ROM), a random access memory (RAM), or a non-volatile memory (NVRAM (Non Volatile RAM)).

[0078] The auxiliary storage device 1003 is, for example, a hard disk drive, an SSD (Solid State Drive), an optical storage device (such as a CD (Compact Disc) or a DVD (Digital Versatile Disc)), a storage system, a reading / writing device for a recording medium such as an IC card, an SD card or an optical recording medium, or a storage area of ​​a cloud server. Programs and data can be read into the auxiliary storage device 1003 via a recording medium reading device or a communication device 1006. The programs and data stored in the auxiliary storage device 1003 are read into the main storage device 1002 as needed.

[0079] The input device 1004 is an interface that accepts input from the outside, and is, for example, a keyboard, a mouse, a touch panel, a card reader, a voice input device, or the like.

[0080] The output device 1005 is an interface that outputs various information such as the progress of processing and the results of processing. The output device 1005 is, for example, a display device (liquid crystal monitor, LCD (Liquid Crystal Display), graphic card, etc.) that visualizes the various information described above, a device that converts the various information described above into audio (audio output device (speaker, etc.)), or a device that converts the various information described above into text (printer, etc.). Note that, for example, the information processing device 1000 may be configured to input and output information to and from other devices via the communication device 1006.

[0081] The communication device 1006 is a device that realizes communication with other devices. The communication device 1006 is a wired or wireless communication interface that realizes communication with other devices via communication means such as a LAN (Local Area Network), a WAN (Wide Area Network), or the Internet, and is, for example, a NIC (Network Interface Card), a wireless communication module, a USB (Universal Serial Interface) module, or a serial communication module.

[0082] The aforementioned input unit 101, reception unit 102, position determination unit 103, extraction unit 104, assignment unit 105, memory unit 400, output unit 403, and usage information input unit 602 are realized, for example, by the processor 1001 of the information processing device 1000 reading and executing a program stored in the main memory device 1002.

[0083] In addition, the main memory device 1002 and the auxiliary memory device 1003 store the prior information 106, the 3D model 107, the frame 108, the gaze information 109, the position of the reference coordinates 110, the working frame 111, the position of the working frame relative to the reference coordinates 112, the 3D gaze point 301, the camera coordinates 303, the relative camera position and attitude 304, the selection time 306, the working frame to be stored 401, the specified working frame 402, the usage information 601, the movement amount 603, the color image 801, the point cloud 802, and the 2D gaze point 804.

[0084] The aforementioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits, or may be realized in software by having a processor interpret and execute a program that realizes each function.

[0085] Information such as programs, tables, and files that realize each function can be stored in a storage device such as a memory, hard disk, or SSD (Solid State Drive), or in a recording medium such as an IC (Integrated Circuit) card, SD card, or DVD (Digital Versatile Disc).

[0086] The present invention is not limited to the above-described embodiments and includes various modifications. For example, although the above embodiments have been described with reference to a cart, other objects may also be used. Furthermore, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0087] DESCRIPTION OF SYMBOLS 100: Work management device 101: Input unit 102: Reception unit 103: Position determination unit 104: Extraction unit 105: Assignment unit 106: Prior information 107: 3D model 108: Frame 109: Gaze information 110: Reference coordinate position 111: Work frame 112: Work frame position 300: 3D gaze point calculator 301: 3D gaze point 302: Camera position and orientation calculator 303: Camera coordinates 304: Relative camera position and orientation 305: Gaze point selector 306: Selection time 307: Reference coordinate setter 400: Memory unit 401: Work frame to be stored 402: Specified work frame 403: Output unit 600: Robot 601: Usage time information 602: Usage time information input unit 603: Movement amount 700: Railway vehicle 701: Wheel 702: Axle 703: Worker 704: Head-mounted display 705: Reference coordinates 706: Parts under the cart 707: Inspection point 708: Color camera 709: ToF camera 710: Reference object 711: Position and orientation 801: Color image 802: Point cloud 803: Eye camera 804: 2D gaze point 805: 2D gaze point calculator 806: IMU 1000: Information processing device 1001: Processor 1002: Main memory device 1003: Auxiliary memory device 1004: Input device 1005: Output device 1006: Communication device

Claims

1. A work management device comprising: an input unit for inputting prior information including reference coordinates; a reception unit for receiving a plurality of frames showing a worker's work status and gaze information of the worker in the frames; a position determination unit for determining a position of the reference coordinate in the frames using the prior information, the frames, and the gaze information; an extraction unit for extracting a work frame showing the work scene of the worker using the frame and the gaze information; and an assignment unit for assigning a position relative to the reference coordinate to the work frame using the work frame extracted by the extraction unit, the frame received by the reception unit, and the position of the reference coordinate determined by the position determination unit.

2. A work management device according to claim 1, wherein the advance information is a 3D model of an object.

3. A work management device according to claim 1, wherein the frame is at least one point cloud.

4. A work management device as claimed in claim 1, wherein the extraction unit extracts the work frame when the line of sight information is present in a specific part for a certain period of time.

5. A work management device as described in claim 1, wherein, in the position determination unit, when the gaze information is present in a specific portion for a certain period of time, the advance information is moved based on the gaze information, a matching process is performed between the advance information and the frame, and the position of the reference coordinate is determined.

6. A work management device as described in claim 5, characterized in that in the position determination unit, alignment is performed using the prior information moved to an initial position and a selected point cloud, and the prior information is moved to the spatial position where the reference object was being gazed upon.

7. A work management device as described in claim 1, comprising: a usage information input unit that accepts usage information of the worker or robot; and a movement amount calculation unit that inputs the usage information and a position relative to the reference coordinates and calculates the movement amount of the worker or robot.

8. A work management device as described in claim 1, comprising: a memory unit that stores a position relative to the reference coordinates and the work frame; and an output unit that outputs a work frame that is assigned a position relative to a specified reference coordinate from among the work frames that are assigned a position relative to the reference coordinates stored in the memory unit.

9. A work management system comprising: an input unit for inputting prior information including reference coordinates; a reception unit for receiving a plurality of frames showing a worker's work status and gaze information of the worker in the frames; a position determination unit for determining a position of the reference coordinate in the frames using the prior information, the frames, and the gaze information; an extraction unit for extracting a work frame showing the work scene of the worker using the frame and the gaze information; an assignment unit for assigning a position relative to the reference coordinate to the work frame using the work frame extracted by the extraction unit, the frame accepted by the reception unit, and the position of the reference coordinate determined by the position determination unit; and an acquisition unit for acquiring the gaze information.

10. A work management method, comprising: receiving advance information including reference coordinates at an input unit; receiving a plurality of frames showing the work status of a worker and gaze information of the worker in the frames at a reception unit; determining the position of the reference coordinates in the frames using the advance information, the frames, and the gaze information at a position determination unit; extracting a work frame showing the work scene of the worker using the frame and the gaze information at an extraction unit; and assigning a position relative to the reference coordinates to the work frame at an assignment unit using the work frame extracted by the extraction unit, the frame accepted by the reception unit, and the position of the reference coordinates determined by the position determination unit.

Images