Work support system and work support program

Abstract

This prevents users from forgetting to re-register the reference posture information when moving the work machine. [Solution] A work support system 1 comprising: posture sensors 2A to 2C attached to the work section 103 of a hydraulic excavator 100 for detecting posture information of the work section 103; and a control unit 33 that registers posture information as reference posture information in response to a user registration operation with the work section 103 as the reference posture, and after registration calculates information to be notified to the user based on the reference posture information and the current posture information, further comprising an imaging unit 31 (or imaging device 4) attached to the hydraulic excavator 100 for capturing images of the surrounding scenery from the hydraulic excavator 100, and the control unit 33 comprising: movement determination means for determining the movement of the hydraulic excavator 100 based on the image captured by the imaging unit 31; and re-registration operation request means for requesting the user to perform a re-registration operation of the reference posture information when it is determined that the hydraulic excavator 100 has moved.

Description

Technical Field

[0001] The present disclosure relates to a work support system and a work support program.

Background Art

[0002] Conventionally, construction machinery incorporating a machine guidance function (so-called ICT construction machinery) has been provided. Here, machine guidance is a technology that supports the operation of construction machinery based on information obtained using measurement technology. Machine guidance includes 2D machine guidance and 3D machine guidance. 2D machine guidance is a relative work support system based on the cutting edge of a bucket or the like. 3D machine guidance is an absolute work support system that uses position information obtained from measurement technologies such as a total station (TS) and a global navigation satellite system (GNSS).

[0003] As conventional 2D machine guidance, for example, there is one disclosed in Patent Document 1. This 2D machine guidance is composed of a sensor unit, a control unit, a portable information terminal, and a notification unit. The sensor unit is held by a movable part of a hydraulic excavator and acquires the attitude information of the movable part. The control unit collects the attitude information acquired by the sensor unit through data communication with the sensor unit. The portable information terminal calculates the movable amount to the excavation target based on the attitude information collected by the control unit. The notification unit notifies the operator (user) of the hydraulic excavator of the movable amount to the excavation target calculated by the portable information terminal device.

[0004] In such 2D machine guidance, it is necessary to register a reference attitude (for example, a state where the bucket is placed on the ground surface) serving as a reference for calculating the movable amount for each work location. For example, after the work at one point is completed, when the construction machinery moves several meters, the angle of the construction machinery and the height with respect to the excavation target surface may change. Therefore, even if the movement is small, after the movement, it is necessary to re-register the reference attitude before resuming the work in order to ensure accuracy.

Prior Art Documents

[0005] [Patent Document 1] Patent No. 7207575 [Overview of the project] [Problems that the invention aims to solve]

[0006] However, conventional 2D machine guidance allows construction machinery to continue working without re-registering the reference posture when it moves, which means there is a possibility that the user may be notified of the amount of movement required to reach the construction target calculated based on an incorrect reference posture.

[0007] Therefore, this disclosure aims to provide a work support system and work support program that can prevent users from forgetting to re-register reference posture information when moving work machinery. [Means for solving the problem]

[0008] In one embodiment, the following solution is provided. A sensor unit attached to the movable part of a work machine to detect the posture information of the movable part, A work support system comprising: a control unit that, in response to a user's registration operation with the movable part as the reference posture, registers the posture information as reference posture information, and after registration calculates information to be notified to the user based on the reference posture information and the current posture information; The machine is attached to the aforementioned work machine and further comprises an imaging unit that captures images of the surrounding scenery from the work machine, The control unit, A movement determination means that determines the movement of the work machine based on the image captured by the imaging unit, The system includes a re-registration operation request means that, when it is determined that the work machine has moved, requests the user to perform a re-registration operation of the reference posture information. [Effects of the Invention]

[0009] According to this disclosure, it becomes possible to provide a work support system and work support program that can prevent users from forgetting to re-register reference posture information when they move a work machine. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows a hydraulic excavator to which the work support system of this disclosure is applied. [Figure 2] This is a view looking forward from the control panel of a hydraulic excavator. [Figure 3] This is a block diagram showing the configuration of the work support system. [Figure 4] This diagram shows the display screen of a portable information terminal device; (a) shows the screen for requesting re-registration of reference posture information, and (b) shows the screen for requesting movement of on-site installations for feature points. [Figure 5] This is a flowchart showing the processing procedure for a mobile information terminal device. [Modes for carrying out the invention]

[0011] The following describes each embodiment in detail with reference to the attached drawings. (Working machinery) Figure 1 shows a hydraulic excavator 100 to which the work support system 1 of this disclosure is applied, and Figure 2 is a view of the hydraulic excavator 100 from the control unit 105 looking forward. As shown in Figure 1, the work support system 1 of this disclosure is applied to a hydraulic excavator 100, which is a work machine (construction machine), and supports the work of the user operating the hydraulic excavator 100 through functions such as machine guidance.

[0012] As shown in Figures 1 and 2, the hydraulic excavator 100 comprises a lower traveling body 101, an upper rotating body 102 that is rotatably connected to the upper part of the lower traveling body 101, and a working section 103 connected to the front of the upper rotating body 102.

[0013] The lower traveling body 101 includes left and right crawler traveling devices 104L and 104R. When the left and right crawler traveling devices 104L and 104R are simultaneously driven and operated in the same direction, the lower traveling body 101 travels straight in the forward or reverse direction. When either one of the left and right crawler traveling devices 104L and 104R is driven and operated in the forward or reverse direction, the lower traveling body 101 performs a field rotation. When the left and right crawler traveling devices 104L and 104R are simultaneously driven and operated in opposite directions, the lower traveling body 101 performs a super field rotation.

[0014] The upper slewing body 102 includes an operation unit 105 on which a user rides. The operation unit 105 is provided with a driver's seat 106 on which the user sits, left and right traveling operation levers 107L and 107R arranged in front of the driver's seat 106, and left and right work operation levers 108L and 108R arranged on the left and right sides of the driver's seat 106.

[0015] The left and right traveling operation levers 107L and 107R are operation tools for operating the left and right crawler traveling devices 104L and 104R, and are operated in the front-rear direction from the neutral position. When the left traveling operation lever 107L is operated forward, the left crawler traveling device 104L is driven in the forward direction, and when the left traveling operation lever 107L is operated backward, the left crawler traveling device 104L is driven in the reverse direction. When the right traveling operation lever 107R is operated forward, the right crawler traveling device 104R is driven in the forward direction, and when the right traveling operation lever 107R is operated backward, the right crawler traveling device 104R is driven in the reverse direction.

[0016] The left and right work operation levers 108L and 108R are operation tools for operating a slewing device (not shown) of the upper slewing body 102 and a work unit 103, and are operated in the front-rear direction and the left-right direction from the neutral position. There are a plurality of types of operation patterns of the work operation levers 108L and 108R, and they are selected according to the user's preference and the like.

[0017] The working unit 103 includes a boom 111 that is connected to the front part of the upper slewing body 102 so as to be able to perform a pitching operation, an arm 112 that is connected to the tip of the boom 111 so as to be able to perform a rotation operation, and a bucket 114 that is connected to the tip of the arm 112 via a link 113 so as to be able to perform a rotation operation.

[0018] The boom 111 pitches according to the hydraulic expansion and contraction operation of a boom cylinder (not shown) interposed between the upper slewing body 102 and the boom 111. The arm 112 rotates according to the hydraulic expansion and contraction operation of an arm cylinder (not shown) interposed between the boom 111 and the arm 112. The bucket 114 rotates according to the hydraulic expansion and contraction operation of a bucket cylinder (not shown) interposed between the arm 112 and the link 113. Note that the work support system 1 is not limited to the hydraulic excavator 100, and can be widely applied to various work machines used for work such as civil engineering, construction, agriculture, and transportation.

[0019] (Work support system) FIG. 3 is a block diagram showing the configuration of the work support system 1. As shown in FIGS. 1 to 3, the work support system 1 includes a plurality of attitude sensors 2A to 2C, a portable information terminal device 3, an imaging device 4, and a feature point field installation object 5.

[0020] (Attitude sensor) The plurality of attitude sensors 2A to 2C include a boom attitude sensor 2A, an arm attitude sensor 2B, and a link attitude sensor 2C. The boom attitude sensor 2A is attached to the boom 111 and detects the attitude information (pitching angle) of the boom 111. The arm attitude sensor 2B is attached to the arm 112 and detects the attitude information (rotation angle) of the arm 112. The link attitude sensor 2C is attached to the link 113 and indirectly detects the attitude information (rotation angle) of the bucket 114.

[0021] As shown in Figure 3, each attitude sensor 2A to 2C comprises a detection unit 21 for detecting attitude information of the object to be detected, a calculation unit 22 for processing the attitude information detected by the detection unit 21, a data communication unit 23 for wirelessly transmitting the attitude information, and a battery (not shown) for supplying power to each unit. By operating on battery power and transmitting attitude information via wireless communication, each attitude sensor 2A to 2C can be easily installed in any location without the need to provide separate power supply and data communication cables. The detection unit 21 uses an IMU (INERTIAL MEASUREMENT UNIT) sensor, and Bluetooth® is used for wireless communication in the data communication unit 23.

[0022] (Mobile Information Terminal Device) The mobile information terminal device 3 is a so-called smartphone or tablet device, and it performs 2D machine guidance by executing application software (hereinafter sometimes referred to as "app") related to the work support system 1. When using 2D machine guidance, the user performs a predetermined reference posture registration operation (for example, a tap operation on the reference posture registration screen) with the work unit 103 in a reference posture (for example, a posture in which the bucket 114 is placed on the ground) before starting work. When the predetermined reference posture registration operation is performed, the mobile information terminal device 3 acquires posture information from posture sensors 2A to 2C and registers it as reference posture information. Subsequently, based on the registered reference posture information and the current posture information, the mobile information terminal device 3 calculates the current cutting edge position of the bucket 114 and the amount of movement (deviation) from the current cutting edge position to the construction target, and notifies the user of this calculated information.

[0023] In such 2D machine guidance systems, reference posture information must be re-registered for each work location. For example, if the hydraulic excavator 100 moves a few meters after completing work at one location, the angle of the hydraulic excavator 100 and its height relative to the excavation target surface may change. Therefore, even after moving a small distance, it is necessary to re-register the reference posture information before resuming work to ensure accuracy. The work support system 1 of this disclosure is characterized by a configuration that prevents the user from forgetting to re-register the reference posture information when the hydraulic excavator 100 is moved. This characteristic configuration will be described in detail below.

[0024] (Features of on-site installations) The feature point site installation 5 is installed at the construction site so that the work support system 1 (portable information terminal device 3) can identify feature points at the construction site with high accuracy and continuously. The feature point site installation 5 has a shape and color that have been pre-learned so that it can be identified as a feature point in the captured images of the site. For example, the feature point site installation 5 is composed of a sign with a distinctive color (for example, red) that can be clearly recognized even among the soil color that makes up the majority of construction sites, and an installation pin that is inserted into the ground, all integrated together.

[0025] (Functional configuration of the mobile information terminal device 3) The portable information terminal device 3 comprises a movement determination means, a re-registration operation request means, and an object movement request means, as a functional configuration realized through the cooperation of hardware and software.

[0026] The movement determination means determines the movement of the hydraulic excavator 100 based on the image of the construction site captured by the imaging unit 31 or imaging device 4 of the portable information terminal device 3 from the control unit 105. For example, in this embodiment, the movement determination means recognizes feature points (site installations for feature points 5) included in the image in response to the registration or re-registration operation of reference posture information, and determines the movement of the hydraulic excavator 100 based on the movement of the feature points in the image. However, even if the movement determination means determines that the hydraulic excavator 100 has moved based on the image captured by the imaging unit 31 or imaging device 4, if it determines that the hydraulic excavator 100 has returned to its original location within a predetermined time, it considers that the hydraulic excavator 100 has not moved.

[0027] The re-registration request means requests the user to re-register the reference posture information when it determines that the hydraulic excavator 100 has moved. For example, as shown in Figure 4(a), the re-registration request screen G1 for reference posture information is displayed on the display screen G of the portable information terminal device 3, and an alarm sound is emitted to request the user to re-register the reference posture information.

[0028] Such movement determination means and re-registration request means prevent the user from forgetting to re-register the reference posture information when the hydraulic excavator 100 is moved. Furthermore, even if it is determined that the hydraulic excavator 100 has moved, if it is determined that the hydraulic excavator 100 has returned to its original location within a predetermined time, the user is not requested to re-register the reference posture information, thus preventing unnecessary re-registration requests.

[0029] In other words, the movement determination means determines the movement of the hydraulic excavator 100 based on the images of the construction site captured from the control unit 105. Therefore, it may determine that the hydraulic excavator 100 has moved not only when the hydraulic excavator 100 is traveling, but also when the upper slewing body 102 is rotating. However, excavation work by the hydraulic excavator 100 usually involves repeating the steps of excavating and putting soil into the bucket 114, rotating the upper slewing body 102 to unload the soil to another location, and rotating the upper slewing body 102 to continue excavating at the original location. During such excavation work, it is not necessary to re-register the reference posture information. Therefore, even if the movement determination means determines that the hydraulic excavator 100 has moved, if it determines that the hydraulic excavator 100 (upper slewing body 102) has returned to its original location within a predetermined time, it will consider that the hydraulic excavator 100 has not moved, thereby preventing unnecessary re-registration requests. Furthermore, even if the hydraulic excavator 100 moves by driving, if the hydraulic excavator 100 returns to its original location within a predetermined time, the hydraulic excavator 100 has not actually moved, and therefore, re-registration of the reference posture information is unnecessary.

[0030] The object movement request means requests the user to move the feature point site installation object 5 when a state occurs in which the feature point site installation object 5 cannot be recognized in the image captured by the imaging unit 31 or imaging device 4. For example, as shown in Figure 4(b), the user is requested to move the feature point site installation object 5 by displaying the feature point site installation object 5 movement request screen G2 on the display screen G of the portable information terminal device 3 and outputting an alarm sound. With such an object movement request means, it is possible to prevent work from continuing in a state in which the feature point site installation object 5 cannot be recognized, that is, in a state in which the movement of the hydraulic excavator 100 cannot be properly determined.

[0031] (Specific configuration of a mobile information terminal device) As shown in Figure 3, the portable information terminal device 3 comprises an imaging unit 31, a storage unit 32, a control unit 33, a notification unit 34, and a data communication unit 35. The portable information terminal device 3 is installed on the indoor side of the windshield 120 that covers the front of the control unit 105, for example, as shown in Figure 2. In this case, the portable information terminal device 3 is installed so that the imaging unit 31 faces outwards and the display screen G constituting the notification unit 34 faces inwards.

[0032] The imaging unit 31 captures images of the external scenery in front of the control unit 105 (the working range of the hydraulic excavator 100). The site-mounted objects 5 for feature points are installed at the construction site so as to be within the imaging range of the imaging unit 31.

[0033] The memory unit 32 stores imaging data from the imaging unit 31, learning data related to the recognition of on-site installed objects 5 for feature points, and application software related to the work support system 1.

[0034] The control unit 33 recognizes feature points (site-installed objects 5 for feature points) included in the image captured by the imaging unit 31 based on the data stored in the memory unit 32, and determines the movement of the hydraulic excavator 100 based on the movement of the feature points in the image captured. When the control unit 33 determines that the hydraulic excavator 100 has moved, it instructs the notification unit 34 to execute a notification requesting the re-registration of the reference posture information. Furthermore, when the control unit 33 determines that the site-installed objects 5 for feature points cannot be recognized in the image captured by the imaging unit 31, it instructs the notification unit 34 to execute a notification requesting the movement of the site-installed objects 5 for feature points.

[0035] The notification unit 34 executes notifications using a display screen G or an alarm sound in response to a notification execution instruction from the control unit 33. For example, when the notification unit 34 receives a notification execution instruction from the control unit 33 regarding a request to re-register the reference attitude information, it displays the reference attitude information re-registration request screen G1 (see Figure 4(a)) and outputs an alarm sound to request the user to re-register the reference attitude information.

[0036] Furthermore, when the notification unit 34 receives a notification execution instruction from the control unit 33 regarding a request to move the feature point site installation 5, it displays the feature point site installation 5 movement request screen G2 (see Figure 4(b)) and outputs an alarm sound to request the user to move the feature point site installation 5. Note that the notification means of the notification unit 34 are not limited to screen display or sound, and various notification means (e.g., lamp, vibration, etc.) can be applied as long as the user can recognize them.

[0037] The data communication unit 35 communicates with the attitude sensors 2A-2C and the imaging device 4. The data communication unit 35 can also be used for downloading and upgrading application software related to the work support system 1. The data communication unit 35 can use both wireless and wired communication. Wireless communication can utilize, for example, Bluetooth®, while wired communication can utilize, for example, USB 3.0. Various communication methods, both wireless and wired, can be used for data communication.

[0038] (Imaging device) The imaging device 4 is used to capture the forward view from the control unit 105 in place of the imaging unit 31 of the portable information terminal device 3. As shown in Figure 3, the imaging device 4 comprises an imaging unit 41, a control unit 42, and a data communication unit 43. When the imaging device 4 is powered on, the control unit 42 issues an instruction to start imaging, and the image captured by the imaging unit 41 is transmitted from the data communication unit 43 to the data communication unit 35 of the portable information terminal device 3. When the control unit 33 of the portable information terminal device 3 receives the image captured by the imaging device 4, it does not take an image with its own imaging unit 31, but instead stores the image captured by the imaging device 4 in the storage unit 32 for use.

[0039] (Processing procedure for mobile information terminal devices) Next, the processing procedure of the portable information terminal device 3 (control unit 33) that implements the work support system 1 described above will be explained with reference to Figure 5 and other figures.

[0040] Figure 5 is a flowchart showing the processing procedure of the mobile information terminal device 3. As shown in Figure 5, when the control unit 33 of the portable information terminal device 3 determines that a reference pose has been registered in the 2D machine guidance application software (S0), it instructs the imaging unit 31 to start imaging at a predetermined frame rate (S1). The control unit 33 then instructs the imaging unit 31 to store a certain number of imaging data in the storage unit 32 (S2). When a certain number of imaging data is reached, the oldest data is deleted sequentially, and at least two new imaging data are always continuously stored. If imaging data is received from the imaging device 4 in step S1, the imaging unit 31 does not take images, and the imaging data from the imaging device 4 is stored in the storage unit 32 for use.

[0041] The control unit 33 extracts the on-site installation object 5 for feature points from the imaging data as feature points and recognizes its position (S3). The control unit 33 compares the position of the feature point in the image captured immediately after registration of the reference orientation information with the position of the feature point in the current image captured, and calculates the amount of movement of the feature point between the two frames (S4). Here, it is assumed that the amount of movement of the feature point is calculated from the difference in the pixel position of the feature point in the two images captured, but a wide range of methods capable of discriminating the movement of the feature point can be widely applied.

[0042] After calculating the amount of movement of the feature points, the control unit 33 determines whether the calculated amount of movement exceeds a predetermined threshold (S5). For example, the threshold used for determination could be 1 / 10 of the vertical pixel value of the captured image used. In step S5, the control unit 33 also determines whether the calculated amount of movement exceeded the predetermined threshold for a predetermined period of time or longer. In other words, the control unit 33 determines that the hydraulic excavator 100 has moved if the calculated amount of movement has exceeded the predetermined threshold for a predetermined period of time or longer, but if the time during which the calculated amount of movement exceeded the predetermined threshold is less than the predetermined time, it is considered that the hydraulic excavator 100 has not moved.

[0043] If the result of the judgment in step S5 is "not exceeded", the control unit 33 repeats steps S4 and S5. If it is "exceeded", it determines whether there are feature points in the captured image (S6). If the control unit 33 determines that there are feature points within the imaging range, it issues an alert to the user via the notification unit 34 (see (a) in Figure 4) (S7) and prompts the user to re-register the reference posture information. Once the user has completed re-registering the reference posture information (S8), the control unit 33 stops the alert on the notification unit 34 (S9) and sets the current feature point position as the reference position for calculating the amount of movement (S10). After that, the control unit 33 determines whether there is a request to terminate the application (S11). If there is no request, it continues to perform the processing from step S4 onwards. If there is a request to terminate the application, it terminates this flow along with the termination of the application (S12).

[0044] On the other hand, in step S6, if the control unit 33 determines that there is no reference point within the imaging range, it displays a feature point movement instruction (feedback screen G2 for requesting movement of the on-site installation object 5 for feature points (see Figure 4(b))) on the notification unit 34, prompting the user to move the on-site installation object 5 for feature points (S13). The control unit 33 continuously determines whether the feature point is within the imaging range (S14), and if it is confirmed that the feature point is within the imaging range, it proceeds to step S7, but continues to display the feature point movement instruction until it is confirmed.

[0045] Although each embodiment has been described in detail above, the invention is not limited to any particular embodiment, and various modifications and changes are possible within the scope described in the claims. Furthermore, it is possible to combine all or more of the components of the embodiments described above. [Explanation of Symbols]

[0046] 1. Work support system 2A Boom attitude sensor (sensor unit) 2B Attitude sensor for arm (sensor unit) 2C Link Attitude Sensor (Sensor Unit) 21 Detection unit 22 Arithmetic section 23 Data Communications Department 3. Portable Information Terminal Device (Control Unit) 31 Imaging Unit 32 Storage section 33 Control Unit 34 Notification Department 35 Data Communications Department 4. Imaging device 41 Imaging Unit 42 Control Unit 43 Data Communications Department 5. On-site installations (objects) for distinctive features 100 Hydraulic Excavator (Work Machine) 101 Lower running body 102 Upper rotating body 103 Working part (movable part) 104L, 104R Crawler Track System 105 Control Unit 106 Driver's seat 107L, 107R Driving control lever 108L, 108R Operation lever 111 Boom 112 Arm 113 links 114 buckets 120 Windshield G display screen G1 Re-registration request screen G2 movement request screen

Claims

1. A sensor unit attached to the movable part of a work machine to detect the posture information of the movable part, A work support system comprising: a control unit that, in response to a user's registration operation with the movable part as the reference posture, registers the posture information as reference posture information, and after registration calculates information to be notified to the user based on the reference posture information and the current posture information; The machine is attached to the aforementioned work machine and further comprises an imaging unit that captures images of the surrounding scenery from the work machine, The control unit, A movement determination means that determines the movement of the work machine based on the image captured by the imaging unit, A work support system comprising: a re-registration operation request means that, when it is determined that the work machine has moved, requests the user to perform a re-registration operation of the reference posture information.

2. The work support system according to claim 1, wherein the movement determination means determines, based on the image captured by the imaging unit, that the work machine has moved, and if it determines that the work machine has returned to its original location within a predetermined time, the movement of the work machine is deemed not to have occurred.

3. The work support system according to claim 1, wherein the movement determination means recognizes feature points included in the captured image in response to the registration operation or the re-registration operation, and determines the movement of the work machine based on the movement of the feature points in the captured image.

4. The aforementioned characteristic point is a predetermined object installed at the site, The work support system according to claim 3, further comprising an object movement request means for requesting the user to move the object when a state occurs in which the object cannot be recognized in the captured image.

5. A work support program that causes a computer to operate as a control unit according to any one of claims 1 to 4.

Images