Work status management system, work status management method and program

Abstract

To provide a work state management system, a work state management method, and a program for supporting work using a working machinery tool.SOLUTION: A work state management system 1 includes a working machinery tool 10, and a worker terminal 20. The working machinery tool 10 has an action body 11 for giving predetermined action to an object, and a 3DAR marker 15. The worker terminal 20 has a working machinery tool recognition part 201 for photographing the 3DAR marker 15 by a camera, recognizing the position and the attitude of the 3DAR marker 15 on the basis of the photographed image, and estimating the position of the action body 11 on the basis of the position and the attitude of the 3DAR marker 15, a work instruction part 202 for displaying an object indicating a position where the action body 11 should be acted on a transmission type display so that the object can be visually recognized so as to overlap the object, and a work state display part 203 for determining whether or not the position of the action body 11 estimated by the working machinery tool recognition part 201 is settled within a range indicated by the object, and displaying the determination result on the transmission type display.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a work status management system, a work status management method, and a program, and more particularly to a technology for supporting work using work machines and implements. [Background technology]

[0002] At civil engineering and construction sites, many tasks involve using work machinery and tools in a specific location for a specific period of time to achieve a specific purpose. Examples include compacting soil using rolling machines (also known as tamping rammers or plate compactors), compacting crushed stone under railroad tracks using tie tampers, tapping inspections of exterior walls using tapping rods, and filling or coating walls with insulation or fireproofing materials using spray guns. Even outside of civil engineering and construction, tasks such as spraying pesticides on fields using sprayers or drones, cleaning indoor carpets with cleaning machines, and shoveling brewed food all involve using specific work machinery and tools in a specific location for a specific period of time.

[0003] In these operations, the position or range, time, angle, etc. of the work equipment to be applied may be instructed in advance, but in reality, these are determined based on the intuition and experience of each worker. Therefore, variations tend to occur between workers, and even the same worker may produce different results each time. From the perspective of work quality management, the challenge is to minimize such variations as much as possible.

[0004] Related technologies include Patent Documents 1 to 4. All of these relate to the compaction of fresh concrete using a vibrator, which is a type of work machine, and aim to objectively grasp the work situation.

[0005] Patent documents 1 to 3 describe a system that involves placing AR markers around the formwork, capturing images of the AR markers and a vibrator during compaction work with a camera, and using the images to estimate the position of the vibrator and identify the area where compaction has been performed.

[0006] Patent Document 4 describes a system that acquires the position of a vibrator using GPS and displays an augmented reality image of the vibrator at that position using a head-mounted display worn by a worker. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent Publication No. 2021-161845 [Patent Document 2] Patent Publication No. 2021-161846 [Patent Document 3] Patent Publication No. 2021-161847 [Patent Document 4] Japanese Patent Application Publication No. 2019-035265 Summary of the Invention [Problem to be solved by the invention]

[0008] However, the methods described in Patent Documents 1 to 3 require a process of installing a camera in advance in a position where it can simultaneously and continuously capture images of the AR marker and the work machine. This process requires a considerable amount of time, effort, and cost. Furthermore, with these methods, the operating position of the work machine is estimated based on the parts visible in the image (e.g., the state of the hose connected to the vibrator) and on the structure specific to that work machine. Such an estimation method specialized for a specific work machine cannot be generally applied to other work machines. Furthermore, these systems do not include a configuration for instructing the worker on the position, range, time, angle, etc. at which the work machine should be operated, or a configuration for reporting the work results to a manager.

[0009] The method described in Patent Document 4 acquires the position of a work machine or tool using a GPS unit. However, depending on the type of work machine or tool, the GPS unit may have difficulty receiving signals from satellites, making it difficult to measure its position. It is said that GPS-based position detection can have an error of several meters, limiting the ability of this method to accurately reproduce the position of the work machine or tool. Furthermore, this system does not include a configuration for instructing the worker on the position, range, time, angle, etc. at which the work machine or tool should be operated, nor does it include a configuration for reporting the results of the work to a manager.

[0010] The present invention has been made to solve these problems, and has as its object to provide a work status management system, a work status management method, and a program for supporting work using work machines and implements. [Means for solving the problem]

[0011] According to one embodiment, a work status management system includes a work machine and a worker terminal, wherein the work machine has an acting body that applies a predetermined action to an object and a 3DAR marker, and the worker terminal has a work machine recognition unit that photographs the 3DAR marker with a camera, recognizes the position and orientation of the 3DAR marker based on the photographed image, and estimates the position of the acting body based on the position and orientation of the 3DAR marker, a work instruction unit that displays an object indicating the position at which the acting body should act on a transmissive display so that it is superimposed on the object and visible, and a work status display unit that determines whether the position of the acting body estimated by the work machine recognition unit is within the range indicated by the object, and displays the determination result on the transmissive display. According to one embodiment, the work status display unit estimates the distance between the target and the position of the operating body, and displays the distance together with or instead of the determination result. According to one embodiment, the work status display unit estimates an angle between the target and the position of the operating body, and displays the angle together with or instead of the determination result. According to one embodiment, the work machine or tool further has a switch that activates or stops the acting body and notifies the worker terminal of whether the acting body is operating or not, the work machine or tool recognition unit determines whether the acting body is operating or not based on the notification from the switch, and the work status display unit estimates a work time based on the determination result of whether the acting body is operating or not and the distance estimation result, and displays the work time together with at least one of the determination result and the distance, or instead of the determination result and the distance. According to one embodiment, a work status management system includes a work machine or tool and a worker terminal, wherein the work machine or tool has an acting body that applies a predetermined action to an object, and a self-location estimation device, the self-location estimation device estimates its own position and attitude and transmits it to the worker terminal, and the worker terminal has a work machine or tool recognition unit that estimates the position of the acting body based on the position and attitude received from the self-location estimation device, a work instruction unit that displays an object indicating the position at which the acting body should act on a transmissive display so that it is superimposed on the object and visible, and a work status display unit that determines whether the position of the acting body estimated by the work machine or tool recognition unit is within the range indicated by the object, and displays the determination result on the transmissive display. According to one embodiment, a work status management method includes a work machine / tool ​​recognition step of capturing an image of a 3DAR marker equipped on a work machine / tool ​​with a camera, recognizing the position and orientation of the 3DAR marker based on the captured image, and estimating the position of an acting body equipped on the work machine / tool ​​for applying a predetermined action to an object based on the position and orientation of the 3DAR marker; a work instruction step of displaying an object indicating the position at which the acting body should act on a transmissive display so that it is superimposed on the object and visible; and a work status display step of determining whether the position of the acting body estimated by the work machine / tool ​​recognition unit is within a range indicated by the object, and displaying the determination result on the transmissive display. According to one embodiment, a program causes a computer to carry out the above method. [Effects of the Invention]

[0012] According to the present invention, it is possible to provide a work status management system, a work status management method, and a program for supporting work using work machines and tools. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a diagram illustrating an example of a hardware configuration of a work status management system 1. FIG. [Figure 2] 1 is a diagram showing an example of a working machine 10. FIG. [Figure 3] FIG. 2 is a diagram illustrating an example of a worker terminal 20. [Figure 4] FIG. 2 is a diagram illustrating an example of an administrator terminal 30. [Figure 5A] 2 is a block diagram showing the functional configuration of a worker terminal 20 and a manager terminal 30. FIG. [Figure 5B] 2 is a block diagram showing the functional configuration of a worker terminal 20 and a manager terminal 30. FIG. [Figure 6] 4 is a flowchart showing an example of the operation of the work status management system 1. [Figure 7] 1 is a diagram showing an example of a working machine 10. FIG. [Figure 8] 4 is a flowchart showing an example of the operation of the work status management system 1. DETAILED DESCRIPTION OF THE INVENTION

[0014] <First Embodiment> 1 is a diagram showing the hardware configuration of a work status management system 1 according to a first embodiment of the present invention. The work status management system 1 includes a work machine or tool 10, a worker terminal 20, and a manager terminal 30.

[0015] 2 is a diagram showing an example of a work machine 10. The work machine 10 typically includes a housing having an operating body 11 and a switch 14, and a 3DAR marker 15.

[0016] The acting body 11 is a part or member that applies a predetermined action to an object. For example, the striking plate of a tamping rammer, the vibration plate of a plate compactor, the beater of a tie tamper, the tip of a tapping rod, the nozzle of a spray gun, a sprayer or carpet cleaner, the tip of a paddle, etc. are examples of the acting body 11.

[0017] When the switch 14 is in the ON state, it drives the work machine 10 to continue acting on the target, and when it is in the OFF state, it stops the operation of the work machine 10 and discontinues acting on the target. In addition, the switch 14 has a so-called smart switch function, and can notify the worker terminal 20 via Bluetooth (registered trademark) communication or the like of whether it is in the ON or OFF state. Note that if the work machine 10 is not equipped with a power source (for example, if it is a diagnostic wand or paddle), it is not necessary to have the switch 14.

[0018] A typical use of the work implement 10 will now be described. The worker grasps the work machine / tool ​​10 and brings the acting body 11 into contact with, inserts, or holds it facing the target. When the worker turns on the switch 14 and drives a power source (not shown), the acting body 11 acts on the target. For example, it applies vibration or impact, or injects a filler or agent.

[0019] The 3DAR marker 15 is an AR marker having a three-dimensional shape such as a hexahedron. The 3DAR marker 15 is attached to a position that can be easily seen from the outside, such as the outside of the housing of the work machine or tool 10. The 3DAR marker 15 is used by the worker terminal 20 to grasp the position and orientation of the work machine or tool 10.

[0020] The 3DAR marker 15 may have the following shapes: The 3DAR marker 15 is not limited to a hexahedron, and may be any three-dimensional shape such as a sphere. Alternatively, a QR code (registered trademark) or the like may be attached to several surfaces (for example, four surfaces) around the work machine 10, and these may be used as the 3DAR marker 15. Alternatively, the external shape of the work machine 10 may be created and stored in advance as a 3D model, and when an object matching this 3D model is found in real space, the object (i.e., the work machine 10 itself) may be used as the 3DAR marker 15 (object marker).

[0021] 3 is a diagram showing an example of the worker terminal 20. The worker terminal 20 is typically a head-mounted housing equipped with a camera 21, a depth sensor 22, an inertial sensor 23, a processing unit 24, a communication unit 25, and a transmissive display 26. It is a so-called HMD (Head Mounted Display), and examples of such include Hololens (registered trademark).

[0022] The camera 21 captures images of the real world. Typically, the camera captures an image of a range corresponding to the field of view of the worker wearing the worker terminal 20 at a predetermined frame rate.

[0023] The depth sensor 22 measures the distance to an object by irradiating the object with infrared light or the like and measuring the time it takes for the light to be reflected. The depth sensor 22 can acquire three-dimensional point cloud data by measuring (scanning) the distances to multiple points on the object. The three-dimensional point cloud data is information that indicates the shape of an object that exists in the real world.

[0024] The inertial sensor 23 is a sensor that measures acceleration, angular velocity, etc. The acceleration, angular velocity, etc. are used to calculate, for example, the amount of movement, movement direction, movement speed, and attitude (inclination, i.e., line of sight) of the worker terminal 20.

[0025] The processing unit 24 is an information processing device including a CPU (Central Processing Unit) 241, a memory 242, etc., and the CPU 241 executes a program stored in the memory 242 to realize a predetermined function.

[0026] The communication unit 25 communicates with external devices such as the work machine or tool 10 or the manager terminal 30.

[0027] The transmissive display 26 displays the information output by the processing unit 24. A worker wearing the worker terminal 20 can view the real world through the transmissive display 26 and simultaneously view the information displayed on the transmissive display 26.

[0028] The basic functions of the worker terminal 20 will be described. The worker terminal 20 has a function for setting a stationary coordinate system in the real space around the worker. The stationary coordinate system here refers to a coordinate system with an origin and coordinate axes fixed in the real space (the origin and coordinate axes do not change even when the worker terminal 20 moves). The origin and coordinate axes of the stationary coordinate system can be defined, for example, by placing a reference object such as an AR marker in the real space. In this case, the camera 21 captures an image of the AR marker placed in the real space. The AR marker is typically a flat plate on which a predetermined pattern is printed, and the pattern encodes information indicating the position of the origin and the orientation of the coordinate axes. Alternatively, a three-dimensional object can be used as the AR marker (3D AR marker). The processing unit 24 recognizes and analyzes the AR marker included in the captured image and determines the origin and coordinate axes of the stationary coordinate system. Alternatively, the position of the worker terminal 20 at the time of startup can be used as the origin. The worker terminal 20 typically sets the stationary coordinate system at startup.

[0029] Furthermore, the worker terminal 20 can constantly recognize its own position in real space. For example, by using a known technology such as SLAM (Simultaneous Localization and Mapping), the worker terminal 20 can generate a map of the real world based on information acquired by the camera 21 or the depth sensor 22, and simultaneously recognize its own position. The recognized self-position can be output as coordinates in a stationary coordinate system.

[0030] The worker terminal 20 also has a function of superimposing and displaying any object in real space. The object here refers to an image generated by the processing unit 24, and includes, for example, text, a two-dimensional image, a three-dimensional model, and the like. The processing unit 24 displays the object on the transmissive display 26. As a result, a worker wearing the worker terminal 20 on his / her head can view the real world through the transmissive display 26 and simultaneously view the object displayed on the transmissive display 26. At this time, the processing unit 24 places the object at predetermined coordinates in a stationary coordinate system. In other words, even if the worker wearing the worker terminal 20 on his / her head changes his / her position or posture, the processing unit 24 detects the change using SLAM, the inertial sensor 23, or the like, and changes the display position of the object on the transmissive display 26 in response to the change. As a result, the worker sees the object as if it has always been placed at a predetermined position in real space.

[0031] The worker terminal 20 also has a function of grasping the shape of an object existing in real space. Specifically, the depth sensor 22 scans the real space and acquires three-dimensional point cloud data indicating the shape of the object. The processing unit 24 forms a polygon mesh from the three-dimensional point cloud data, thereby recognizing and reproducing the surface of the object.

[0032] 4 is a diagram showing an example of the administrator terminal 30. The administrator terminal 30 is typically a so-called tablet computer that incorporates a processing unit 31, a communication unit 32, a touch panel display 33, etc. in a plate-shaped housing.

[0033] The processing unit 31 is an information processing device including a CPU (Central Processing Unit) 311, a memory 312, etc., and the CPU 311 executes a program stored in the memory 312 to realize predetermined functions.

[0034] The communication unit 32 communicates with external devices such as the work machine or tool 10 or the worker terminal 20.

[0035] The display 33 displays the information output by the processing unit 31 .

[0036] 5A and 5B are block diagrams showing the functional configurations of the worker terminal 20 and the manager terminal 30. As shown in FIG. The worker terminal 20 has a work machine / tool ​​recognition unit 201, a work instruction unit 202, and a work status display unit 203. These components are logically realized by the CPU 241 of the processing unit 24 executing a program stored in the memory 242.

[0037] The work machine / tool ​​recognition unit 201 recognizes the position and orientation (tilt) of the work machine / tool ​​10 existing in real space. The work machine / tool ​​recognition unit 201 detects a 3DAR marker 15 fixed to the exterior of the work machine / tool ​​10 from an image captured by the camera 21, and recognizes the position and orientation of the 3DAR marker 15. The work machine / tool ​​recognition unit 201 also stores in advance an offset indicating the relative position between the 3DAR marker 15 and the operating body 11. Therefore, the work machine / tool ​​recognition unit 201 can calculate the coordinates of the operating body 11 based on the position and orientation of the 3DAR marker 15 and the offset indicating the relative position between the 3DAR marker 15 and the operating body 11.

[0038] Here, it is assumed that the work machine / tool ​​recognition unit 201 stores information regarding the shape and size of the 3DAR marker 15 (for example, that it is a hexahedron with sides of 10 cm). When the work machine / tool ​​recognition unit 201 detects a 3DAR marker 15 in an image captured by the camera 21, it recognizes it as a hexahedron with sides of 10 cm in the stationary coordinate system. In other words, it can recognize at what coordinates in the stationary coordinate system the hexahedron with sides of 10 cm exists and at what angle. Therefore, if the offset indicating the relative position between the 3DAR marker 15 and the operating body 11 is known, the work machine / tool ​​recognition unit 201 can also calculate the coordinates of the operating body 11 in the stationary coordinate system.

[0039] The work machine / tool ​​recognition unit 201 also recognizes the operating status of the work machine / tool ​​10. The switch 14 of the work machine / tool ​​10 notifies the worker terminal 20 via Bluetooth (registered trademark) communication or the like whether it is in an ON or OFF state. The work machine / tool ​​recognition unit 201 receives this notification via the communication unit 25 and determines whether the work machine / tool ​​10 is operating.

[0040] The work instruction unit 202 displays on the transmissive display 26 information indicating at which position on the work site the work machine or tool 10 should be applied. The work instruction unit 202 displays, for example, a circular object indicating the position where the work machine or tool 10 should be contacted, inserted, or directed, superimposed on the surface of the target. Note that the work instruction unit 202 is assumed to have stored in advance in the memory 242 information indicating the position where the work machine or tool 10 should be applied. The work instruction unit 202 may also superimpose drawing data stored in advance in the memory 242 on the work site. Superimposing drawing information or other objects on the work site can be realized by publicly known technology, including Patent No. 6438995.

[0041] The work instruction unit 202 may also display information instructing the time or angle at which the work machine or tool 10 should be brought into contact with, inserted into, or directed toward the target on the transmissive display 26. It is assumed that the work instruction unit 202 has stored this information in advance in the memory 242.

[0042] The work status display unit 203 can generate information indicating the progress of the work (hereinafter simply referred to as work status) and display it on the transmissive display 26. The work status can include, but is not limited to, information indicating the position of the acting body 11 (for example, information indicating the determination result as to whether the acting body 11 is in contact with, inserted into, or directed at the indicated position, or information indicating the distance between the target and the acting body 11), information indicating the angle of the acting body 11 relative to the target, information indicating the time during which the acting body 11 is activated and in correct contact with, inserted into, or directed, etc. This information can be generated, for example, by the method shown below.

[0043] Position: The work instruction unit 202 displays, for example, a circular object superimposed on the surface of the target, indicating the position where the work machine / tool ​​10 should be contacted, inserted, or directed. The position where the work machine / tool ​​10 should be contacted, inserted, or directed may be displayed using a shape other than a circle, grid lines, or the like. The work machine / tool ​​recognition unit 201 calculates the coordinates of the operating body 11 based on the position and orientation (tilt) of the 3DAR marker 15 and a predetermined offset. The work status display unit 203 determines whether the position of the operating body 11 is within the position indicated by the work instruction unit 202. Typically, the work status display unit 203 determines whether the coordinates obtained by perpendicularly projecting the coordinates of the operating body 11 onto the target surface are contained within the circular object. If the operating body 11 is contained within the circular object, for example, "OK" is output as a result; otherwise, "NG" is output. The position determination result is typically displayed on the transmissive display 26 as text information.

[0044] Distance: The work machine / tool ​​recognition unit 201 calculates and outputs the distance between the operating body 11 and the target surface. Alternatively, it may calculate and output the distance from an object indicating the position where the work machine / tool ​​10 should be contacted, inserted, or directed, or the distance from the center point of the object. The work status display unit 203 typically displays the distance as text information on the transmissive display 26.

[0045] Angle: The work machine / tool ​​recognition unit 201 calculates and outputs the angle between the operating body 11 and the target surface. The work status display unit 203 typically displays the angle as text information on the transmissive display 26.

[0046] To perform this calculation, the work machine / tool ​​recognition unit 201 is assumed to previously store the relative tilt of the operating body 11 with respect to the 3DAR marker. For example, if the operating body 11 is generally rod-shaped (such as the beater of a tie tamper, the tip of a tapping rod, the nozzle of a spray gun, a sprayer or carpet cleaner, or the tip of a paddle), the unit is assumed to store information indicating the degree to which the line corresponding to the axis of the rod-shaped shape is tilted with respect to the 3DAR marker. Furthermore, if the operating body 11 acts on the target via a surface (such as the striking plate of a tamping rammer or the vibration plate of a plate compactor), the unit is assumed to store information indicating the degree to which the line corresponding to the normal to that surface is tilted with respect to the 3DAR marker. This allows the work machine / tool ​​recognition unit 201 to indirectly calculate the tilt of the line corresponding to the axis or normal of the operating body 11 in the stationary coordinate system by detecting the attitude (tilt) of the 3DAR marker 15 in the stationary coordinate system. The work machine / tool ​​recognition unit 201 calculates the angle between this line and a plane corresponding to the target surface.

[0047] Time: The work status display unit 203 accumulates the time that work has been performed by the work machine or tool 10 at a predetermined position. The work status display unit 203 typically displays the accumulated time as text information on the transmissive display 26.

[0048] For example, when work is performed by contacting or inserting the operating body 11 into an object (such as the striking plate of a tamping rammer, the vibrating plate of a plate compactor, the beater of a tie tamper, the tip of a sounding rod, the nozzle of a carpet cleaner, or the tip of a paddle), the work status display unit 203 determines that the operating body 11 is in contact with the object surface, i.e., that work is being performed, when the distance between the operating body 11 and the object surface is equal to or less than a predetermined threshold. When the operating body 11 does not need to contact the object (such as the nozzle of a spray gun or sprayer), the work status display unit 203 determines that work is being performed when the operation status of the work machine or tool 10 notified by the work machine or tool recognition unit 201 is ON (in operation). The work status display unit 203 repeatedly performs these determinations at regular intervals and accumulates and outputs the elapsed time during which it is determined that work is being performed. Note that in addition to the above-mentioned conditions for contact determination or operation determination, it may also be possible to add, for example, that the position or angle meets certain requirements, and accumulate the elapsed time when all conditions are met.

[0049] The work status display unit 203 can visualize the work status of a task using colors, graphs, etc. In this example, the time of the work status is visualized using colors. The work instruction unit 202 is assumed to have stored in advance a specified value for the work time at the location where the task is to be performed (e.g., 8 seconds or more). If the cumulative work time at each location exceeds the specified value, the work status display unit 203 colors the horizontal circle object indicating that location in a color indicating "pass" (e.g., green). In other cases, the object indicating the insertion position is colored in a different color depending on the ratio of the cumulative work time to the specified value. For example, if the cumulative work time is 50% or more less than the specified value, it is colored yellow, and if it is less than 50%, it is colored red. At this time, text indicating the cumulative work time may be displayed near the object.

[0050] The work status display unit 203 may also visualize the work status related to distance, work time, angle, etc. using colors, graphs, etc. in the same way.

[0051] The work status display unit 203 stores the acquired work status in a storage area within the memory 242. If necessary, statistical values ​​of the work status may be calculated and stored.

[0052] Furthermore, the work status display unit 203 may store the visualized information displayed on the transmissive display 26 during work and the image of the real world captured by the camera 21 at that time. For example, the information expressing the work execution time using colors as described above can be saved as a set with the image of the real world at that time.

[0053] The manager terminal 30 has a work performance display unit 301. This component is logically realized by the CPU 311 of the processing unit 31 executing a program stored in the memory 312.

[0054] The work performance display unit 301 communicates with the worker terminal 20 via the communication unit 32 and acquires the work status stored in a storage area in the memory 242. In response to a request from the manager, the work performance display unit 301 displays the acquired work status on the display 33. For example, as described above, information that uses colors to represent the time the work has been performed can be displayed superimposed on an image of the real world at that time. In addition, various work statuses (distance, angle, time, etc.) at each work position may be displayed as text information on the display 33.

[0055] The operation of the work status management system 1 will be explained in chronological order using the flowchart of FIG.

[0056] S101: Start The worker wears the worker terminal 20 on his / her head and starts the program. The worker terminal 20 sets a stationary coordinate system in real space and starts the operation of the SLAM function.

[0057] S102: Reading and placing drawing data The work instruction unit 202 reads out drawing data stored in advance in the memory 242 and displays it superimposed on the work site. That is, the drawing data is placed in a stationary coordinate system. The work instruction unit 202 also places an object indicating the insertion position of the work machine or tool 10 in the stationary coordinate system. The placed drawing and object are displayed on the transmissive display 26. This allows the worker to visually recognize the drawing and object superimposed on the real space.

[0058] S103: 3DAR marker recognition The work machine / tool ​​recognition unit 201 acquires an image captured by the camera 21, detects the 3DAR marker 15 fixed to the exterior of the work machine / tool ​​10 from the image, and recognizes the position and orientation of the 3DAR marker 15.

[0059] S104: Estimation of the position of the acting body 11 The work machine / tool ​​recognition unit 201 calculates the coordinates of the acting body 11 based on the position and orientation of the 3DAR marker 15 and an offset indicating the relative position between the 3DAR marker 15 and the acting body 11.

[0060] S105: Detecting the start of operation of the acting body 11 The work machine / tool ​​recognition unit 201 receives, for example, a signal indicating the state of the switch 14, and recognizes the operating status of the work machine / tool ​​10. If the operating body 11 is in operation, the process proceeds to S106.

[0061] Alternatively, the work machine / tool ​​recognition unit 201 determines whether or not the operating body 11 is in contact with the target. If it is in contact, it is assumed that work is being performed by the operating body 11, and the process proceeds to S106.

[0062] S106: Work status display The work status display unit 203 calculates and displays the angle of the working body 11 during work, the work time, and the like.

[0063] S107: Detecting the end of operation of the operating body 11 and saving the work result The work machine / tool ​​recognition unit 201 receives, for example, a signal indicating the state of the switch 14 and recognizes that the work machine / tool ​​10 has stopped. Alternatively, the work machine / tool ​​recognition unit 201 determines whether the acting body 11 is in contact with an object, and if not, considers that the work by the acting body 11 has finished. The work status display unit 203 saves the work status and work status statistics acquired in S106 in a storage area within the memory 242. Furthermore, for example, in response to a request from the manager terminal 30, the work status display unit 203 transmits the saved work status, etc. to the manager terminal 30.

[0064] S108: View work results The work performance display unit 301 of the manager terminal 30 acquires the work status and the like from the worker terminal 20 and displays it on the display 33 .

[0065] <Effects> According to the first embodiment, a 3DAR marker 15 is attached to the work machine 10, and the position of the acting body 11 is expressed as an offset from the 3DAR marker 15. The 3DAR marker 15 is installed in a position that is always visible to the worker terminal 20, so the position of the acting body 11 can be continuously tracked no matter where it is located. Furthermore, the worker terminal 20 can recognize the position and orientation of the 3DAR marker 15 using its own unique function, so it can always accurately estimate the position of the acting body 11 that is fixedly offset from it.

[0066] According to the embodiment, the work instruction unit 202 of the worker terminal 20 displays instructions regarding the position, angle, time, etc. where the work should be performed, as well as the work results, superimposed on the real space on the transmissive display 26. This allows the worker to perform the work accurately, uniformly, and efficiently while checking the instructions and the work status in real time.

[0067] According to the embodiment, the work performance display unit 301 of the manager terminal 30 acquires the work status accumulated by the worker terminal 20 and displays it as work performance on the display 33. This makes it possible to efficiently and automatically keep quality control records that have conventionally been created manually by the manager.

[0068] <Embodiment 2> In the above-described first embodiment, the 3DAR marker 15 is attached to the work machine 10, and the worker terminal 20 identifies the position of the operating body 11 by tracking the 3DAR marker 15.

[0069] In contrast to this, in the second embodiment, a self-position estimation device 17 is attached to the work machine or tool 10 instead of the 3DAR marker 15. The worker terminal 20 identifies the position of the operating body 11 by recognizing the position of the self-position estimation device 17.

[0070] The hardware configuration of the work status management system 1 according to the second embodiment is the same as that of the first embodiment (see FIG. 1). The work status management system 1 includes a work machine or tool 10, a worker terminal 20, and a manager terminal 30.

[0071] 7 is a diagram showing an example of a work machine 10 according to Embodiment 2. The work machine 10 typically includes a housing having an operating body 11 and a switch 14, and a self-position estimation device 17.

[0072] The operating body 11 and the switch 14 are the same as those in the first embodiment, and therefore the description thereof will be omitted.

[0073] The self-location estimation device 17 has a processor and a memory, and realizes various functions by the processor reading and executing programs stored in the memory. The self-location estimation device 17 also has a communication device for communicating with the worker terminal 20, various sensors (e.g., a camera, a depth sensor, an acoustic sensor, etc.) used to detect the amount of movement of the self-location estimation device 17, and a tilt sensor (e.g., a gyro, etc.) for detecting the attitude of the self-location estimation device 17. Typically, a smartphone or the like can be used as the self-location estimation device 17.

[0074] The self-position estimation device 17 sequentially estimates its own movement amount and outputs three-dimensional coordinates indicating its current position. Typical methods for calculating the movement amount include the SLAM (Simultaneous Localization and Mapping) method and the acceleration sensor method. In the SLAM method, the self-position estimation unit 17 senses the surrounding environment while moving using a sensor (e.g., a camera, a depth sensor, an acoustic sensor, etc.) and creates a three-dimensional environmental map. At the same time, the self-position estimation device 17 sequentially estimates the movement amount and estimates its own position on the environmental map. In the acceleration sensor method, the self-position estimation device 17 estimates its own position by calculating the movement distance based on the acceleration obtained by the acceleration sensor. In either case, it is possible to improve the estimation accuracy by using information obtained from a tilt sensor (e.g., a gyro, etc.).

[0075] Furthermore, the self-position estimation device 17 measures and outputs its own attitude, that is, its tilt with respect to the coordinate axes, using a tilt sensor (for example, a gyro or the like).

[0076] The self-localization device 17 also has a function of setting a stationary coordinate system in the real space around itself. The origin and coordinate axes of the stationary coordinate system can be defined, for example, by placing a reference object such as an AR marker in the real space. The AR marker is typically a flat plate on which a predetermined pattern is printed, and information indicating the position of the origin and the orientation of the coordinate axes is encoded in the pattern. Alternatively, a three-dimensional object can be used as the AR marker (3D AR marker). For example, a camera included in the self-localization device 17 captures an image of an AR marker placed in real space. Then, a processor recognizes and analyzes the AR marker included in the captured image to determine the origin and coordinate axes of the stationary coordinate system.

[0077] When the self-position estimation device 17 sets a stationary coordinate system, it can use the same AR marker (including a 3D AR marker) as the AR marker used when the worker terminal 20 sets the stationary coordinate system. This allows the self-position estimation device 17 and the worker terminal 20 to use the same stationary coordinate system. In other words, they can share a space.

[0078] The self-position estimation device 17 successively notifies the worker terminal 20 of the estimated self-position (coordinates in the stationary coordinate system) and attitude (tilt in the stationary coordinate system).

[0079] The hardware configuration and functional configuration of the worker terminal 20 and the manager terminal 30 in the second embodiment are basically the same as those in the first embodiment (see FIGS. 3, 4, 5A, and 5B). Below, only the differences from the first embodiment will be mainly explained, and the rest will not be explained.

[0080] The work machine / tool ​​recognition unit 201 of the worker terminal 20 receives the position and orientation from the self-position estimation device 17. The work machine / tool ​​recognition unit 201 also stores in advance an offset indicating the relative position between the self-position estimation device 17 and the acting body 11. Therefore, the work machine / tool ​​recognition unit 201 can calculate the coordinates of the acting body 11 based on the position and orientation of the self-position estimation device 17 and the offset indicating the relative position between the self-position estimation device 17 and the acting body 11.

[0081] The operation of the work status management system 1 according to the second embodiment will be described in chronological order using the flowchart of FIG.

[0082] S201: Start The worker wears the worker terminal 20 on his / her head and starts the program. He / she also starts the self-location estimation device 17. The worker terminal 20 and the self-location estimation device 17 use the same AR marker to set a stationary coordinate system in real space and start the operation of the SLAM function.

[0083] S202: Reading and placing drawing data The work instruction unit 202 reads out drawing data stored in advance in the memory 242 and displays it superimposed on the work site. That is, the drawing data is placed in a stationary coordinate system. The work instruction unit 202 also places an object indicating the insertion position of the work machine or tool 10 in the stationary coordinate system. The placed drawing and object are displayed on the transmissive display 26. This allows the worker to visually recognize the drawing and object superimposed on the real space.

[0084] S203: Recognize the position of the self-location estimation device 17 The self-position estimation device 17 estimates its own position and orientation in a stationary coordinate system using the SLAM function. The work machine / tool ​​recognition unit 201 receives the position and orientation from the self-position estimation device 17.

[0085] S204: Estimation of the position of the acting body 11 The work machine / tool ​​recognition unit 201 calculates the coordinates of the acting body 11 based on the position and orientation of the self-position estimation device 17 and an offset indicating the relative position between the self-position estimation device 17 and the acting body 11 .

[0086] S205: Detecting the start of operation of the acting body 11 The work machine / tool ​​recognition unit 201 receives, for example, a signal indicating the state of the switch 14, and recognizes the operating status of the work machine / tool ​​10. If the operating body 11 is in operation, the process proceeds to S206.

[0087] Alternatively, the work machine / tool ​​recognition unit 201 determines whether or not the operating body 11 is in contact with the target. If it is in contact, it is assumed that work is being performed by the operating body 11, and the process proceeds to S206.

[0088] S206: Work status display The work status display unit 203 calculates and displays the angle of the working body 11 during work, the work time, and the like.

[0089] S207: Detecting the end of operation of the operating body 11 and saving the work result The work machine / tool ​​recognition unit 201 receives, for example, a signal indicating the state of the switch 14 and recognizes that the work machine / tool ​​10 has stopped. Alternatively, the work machine / tool ​​recognition unit 201 determines whether the acting body 11 is in contact with an object, and if not, considers that the work by the acting body 11 has finished. The work status display unit 203 saves the work status and work status statistics acquired in S206 in a storage area within the memory 242. Furthermore, for example, in response to a request from the manager terminal 30, the work status display unit 203 transmits the saved work status, etc. to the manager terminal 30.

[0090] S208: View work results The work performance display unit 301 of the manager terminal 30 acquires the work status and the like from the worker terminal 20 and displays it on the display 33 .

[0091] <Effects> According to the second embodiment, a self-position estimation device 17 is attached to the work machine or tool 10, and the self-position and orientation estimated by the self-position estimation device 17 are transmitted to the worker terminal 20 as needed. In the first embodiment described above, the camera 21 of the worker terminal 20 had to constantly capture the 3DAR marker 15. However, with the configuration of the second embodiment described above, the worker terminal 20 can grasp the position and orientation of the self-position estimation device 17 even when the camera 21 is not capturing the work machine or tool 10. Furthermore, since the processing load on the worker terminal 20 is smaller than in the first embodiment, the self-position estimation device 17 and the operating body 11 can be tracked more smoothly.

[0092] Although the embodiments of the present invention have been described above, the present invention is not limited to these, and components and various processes can be modified, deleted, or added as appropriate without departing from the spirit of the present invention. For example, each processing means constituting the present invention may be configured by hardware, or any process may be realized by having a CPU execute a computer program. Furthermore, the computer program can be stored and supplied to a computer using various types of temporary or non-temporary computer-readable media. Temporary computer-readable media include, for example, electromagnetic signals supplied to a computer via wire or wirelessly. [Explanation of symbols]

[0093] 1. Work status management system 10. Work machinery and equipment 11 Effector 14 Switch 15 3DAR markers 17 Self-position estimation device 20 Worker terminal 21 Camera 22 Depth sensor 23 Inertial Sensor 24 Processing section 241 CPU 242 memory 25 Communications Department 26 Transmissive Display 30 Administrator terminal 31 Processing section 311 CPU 312 memory 32 Communications Department 33 Display 201 Work equipment recognition unit 202 Work instruction department 203 Work status display section 301 Work performance display section

Claims

1. A work status management system including a work machine and a worker terminal, The working machine or tool has an action body that applies a predetermined action to a target and a 3DAR marker, The worker terminal includes: a work machine / tool ​​recognition unit that captures an image of the 3DAR marker with a camera, recognizes a position and orientation of the 3DAR marker based on the captured image, and estimates a position of the operating body based on the position and orientation of the 3DAR marker; a work instruction unit that displays an object indicating a position where the acting body should act on a transmissive display so that the object is superimposed on the target and visible; a work status display unit that determines whether the position of the operating body estimated by the work machine / tool ​​recognition unit is within the range indicated by the object, and displays the determination result on the transmissive display. Work status management system.

2. The work status display unit estimates the distance between the target and the position of the operating body, and displays the distance together with or instead of the determination result. The work status management system according to claim 1.

3. The work status display unit estimates an angle between the target and the position of the operating body, and displays the angle together with or instead of the determination result. The work status management system according to claim 1.

4. the work machine or tool further includes a switch that activates or stops the acting body and notifies the worker terminal of whether the acting body is operating, the work machine / tool ​​recognition unit determines whether the operating body is operating based on the notification from the switch; The work status display unit estimates a work time based on a determination result of whether the acting body is operating or not and the distance estimation result, and displays the work time together with at least one of the determination result and the distance, or instead of the determination result and the distance. The work status management system according to claim 2.

5. A work status management system including a work machine and a worker terminal, the work machine or tool includes an action body that applies a predetermined action to an object and a self-position estimation device; the self-position estimation device estimates its own position and orientation and transmits the estimates to the worker terminal; The worker terminal includes: a work machine / tool ​​recognition unit that estimates the position of the working body based on the position and orientation received from the self-position estimation device; a work instruction unit that displays an object indicating a position where the acting body should act on a transmissive display so that the object is superimposed on the target and visible; a work status display unit that determines whether the position of the operating body estimated by the work machine / tool ​​recognition unit is within the range indicated by the object, and displays the determination result on the transmissive display. Work status management system.

6. a work machine / tool ​​recognition step of photographing a 3DAR marker provided on the work machine / tool ​​with a camera, recognizing the position and orientation of the 3DAR marker based on the photographed image, and estimating the position of an acting body provided on the work machine / tool ​​for applying a predetermined action to an object based on the position and orientation of the 3DAR marker; a work instruction step of displaying an object indicating a position where the acting body should act on the target on a transmissive display so that the object is superimposed on the target and visible; a work status display step of determining whether the position of the operating body estimated by the work machine / tool ​​recognition unit is within a range indicated by the object, and displaying the determination result on the transmissive display. How to manage work status.

7. A program for causing a computer to execute the method according to claim 6.

Images