Work support system, work support method, and work support program
The work support system addresses the challenge of representing the cutting edge's three-dimensional position by superimposing a guide image on the monitor, enhancing operator efficiency and safety in remote construction operations.
Patent Information
- Application Number
- JP2023085339
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-24
- Publication Date
- 2024-12-05
AI Technical Summary
Existing remote operation systems for construction machinery struggle to accurately represent the three-dimensional position of the cutting edge, making it difficult for operators to perform efficient work.
A work support system that includes a control unit connected to a construction machine, which acquires images and calculates the distance of the work tool to the work area, superimposing a guide image corresponding to the tool's position on the monitor display.
Enhances operator understanding of the work tool's position, improving productivity and safety by providing clear visual guidance during remote operations.
Smart Images

Figure 2024168555000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a work assistance system, a work assistance method, and a work assistance program for assisting work using a construction machine. [Background technology]
[0002] Work using construction machinery is sometimes performed by remote control. In this case, the operator of the construction machinery operates the machinery while looking at a screen. When remotely controlling the construction machinery, an image of the work site is acquired using a camera and is displayed on a monitor in front of the operator who is operating it at a remote location. In this case, a technology for presenting information to the operator to support the work has also been considered (see, for example, Patent Document 1). In the technology described in this document, the captured image acquisition unit acquires a captured image showing the work machine from a camera installed on the work machine. The blade tip shadow generation unit generates a blade tip shadow by projecting the blade tip of the work machine in the vertical direction onto the projection surface. The display image generation unit generates a display image by superimposing the captured image, the blade tip shadow, and a reference range figure by projecting the reachable range of the blade tip in the vertical direction onto the projection surface. The display control unit outputs a display signal for displaying the display image. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Patent Publication No. 2022-55808 Summary of the Invention [Problem to be solved by the invention]
[0004] In the technology described in Patent Document 1, the operator can check the reference range figure, but it is difficult to recognize the three-dimensional position of the cutting edge. In particular, when it is difficult to secure operators who are skilled in remote operation, efficient work cannot be achieved. [Means for solving the problem]
[0005] The work support system for solving the above problem includes a control unit connected to a construction machine equipped with a work tool, which acquires an image of a work area captured by the construction machine, calculates a distance from the work tool to the work area, and outputs a guide image of a size according to the distance superimposed on the image at a position where the work tool abuts the work area. Effect of the Invention
[0006] According to the present disclosure, work using construction machinery can be supported. [Brief description of the drawings]
[0007] [Figure 1] FIG. 1 is an explanatory diagram of a work support system according to an embodiment. [Diagram 2] FIG. 2 is an explanatory diagram of a hardware configuration of the embodiment. [Diagram 3] FIG. 2 is an explanatory diagram of functions of the work support system according to the embodiment. [Figure 4] FIG. 2 is an explanatory diagram of a processing procedure according to an embodiment. [Diagram 5] FIG. 2 is an explanatory diagram of a processing procedure according to an embodiment. [Figure 6] FIG. 2 is an explanatory diagram of a processing procedure according to an embodiment. [Figure 7] 5A and 5B are explanatory diagrams of a processing procedure according to the embodiment, in which FIG. 5A is an explanatory diagram of a case where the bucket is in a high position, and FIG. 5B is an explanatory diagram of a case where the bucket is in a low position. [Figure 8] 5A and 5B are explanatory diagrams of a display screen according to an embodiment, in which FIG. 5A is an explanatory diagram of a case where the bucket is in a high position, and FIG. 5B is an explanatory diagram of a case where the bucket is in a low position. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0008] Hereinafter, a specific embodiment of a work assistance system, a work assistance method, and a work assistance program will be described with reference to FIGS. As shown in Fig. 1, the work support system A1 uses a construction machine 10, a remote control device 15, and an operation support device 20. In this embodiment, the construction machine 10 is assumed to be a backhoe.
[0009] (Configuration of construction machinery) The construction machine 10 includes a control device 10a, a traveling unit 10b, a rotating unit 10c, a boom 10d, an arm 10e, and a bucket 10f. The base end of the boom 10d is attached to the rotating unit 10c by a pin joint. The base end of the arm 10e is attached to the tip of the boom 10d by a pin joint. The base end of the bucket 10f is attached to the tip of the arm 10e by a pin joint. In this way, the arm 10e connects the boom 10d and the bucket 10f. The bucket 10f functions as a work tool and includes a cutting edge for excavating soil and sand, and a container for storing the excavated soil.
[0010] The control device 10a acquires operation information input by an operator from the remote control device 15 and controls the traveling unit 10b, the rotating unit 10c, the boom 10d, the arm 10e, and the bucket 10f. The boom 10d, the arm 10e, and the bucket 10f can change their postures by hydraulic cylinders. Furthermore, the control device 10a acquires status information of the boom 10d, the arm 10e, and the bucket 10f and transmits it to the operation support device 20. The traveling unit 10b moves the construction machine 10 by means of caterpillar tracks or the like.
[0011] The construction machine 10 includes a camera 11, a distance measuring device 12, and a sensor 13. The camera 11 and the distance measuring device 12 are fixed to the rotating part 10c at positions facing the work tool (bucket 10f). The camera 11 captures a color image of an excavation area (work area) using the bucket 10f. A LIDAR (Light Detection And Ranging) device is used as the distance measuring device 12. The distance measuring device 12 scans the front of the construction machine 10 with laser light to identify the shape of the work area using the construction machine 10. In this embodiment, the ground surface shape, such as unevenness, is scanned for the area to be excavated by the bucket 10f. The sensor 13 detects the angle to identify the state (rotation angle) of the boom 10d, the arm 10e, and the bucket 10f.
[0012] The remote control device 15 is a computer device used by an operator who operates the construction machine 10. In this embodiment, it is assumed that the remote control device 15 is connected to the construction machine 10 via a network and operates the construction machine 10 from a remote location. In this embodiment, when the remote control device 15 is used, an external operation support device 20 is used. The details of the operation support device 20 will be described later.
[0013] (Hardware configuration description) 2, the hardware configuration of the information processing device H10 constituting the construction machine 10, the remote control device 15, and the operation support device 20 will be described. The information processing device H10 includes a communication device H11, an input device H12, a display device H13, a storage unit H14, and a processor H15. Note that this hardware configuration is an example, and it is also possible to realize it by other hardware.
[0014] The communication device H11 is an interface that establishes a communication path with another device and transmits and receives data, and is, for example, a network interface or a wireless interface.
[0015] The input device H12 is a device that accepts input of various information, such as a mouse, a keyboard, etc. The display device H13 is a display or the like that displays various information. The memory unit H14 is a storage device that stores data and various programs for executing various functions of the construction machine 10, the remote control device 15, and the operation support device 20. Examples of the memory unit H14 include a ROM, a RAM, and a hard disk.
[0016] The processor H15 uses the programs and data stored in the memory unit H14 to control each process in the construction machine 10, the remote control device 15, and the operation support device 20. Examples of the processor H15 include a CPU and an MPU. The processor H15 loads the programs stored in the ROM, etc., into the RAM and executes various processes for each process.
[0017] The processor H15 is not limited to a processor that performs software processing for all the processes it executes. For example, the processor H15 may include a dedicated hardware circuit (e.g., an application specific integrated circuit (ASIC)) that performs hardware processing for at least a part of the processes it executes. That is, the processor H15 may be configured as follows:
[0018] [1] One or more processors that operate according to a computer program (software). [2] One or more dedicated hardware circuits that perform at least some of the various processes; or [3] Circuits that include combinations of these The processor includes a CPU and memory, such as RAM and ROM, that stores program codes or instructions configured to cause the CPU to execute processes. Memory, i.e., computer-readable media, includes any available media that can be accessed by a general-purpose or special-purpose computer.
[0019] (Work support system) The functions of the work support system will be explained using FIG. The operation support device 20 includes a control unit 21 that communicates with the construction machine 10 .
[0020] The control unit 21 functions as an information acquisition unit 211 and an image generation unit 212 by executing the task assistance program. The information acquisition unit 211 acquires various types of work information from the construction machine 10. In this embodiment, the work information acquired includes status (rotation angle) information of the boom 10d, arm 10e, and bucket 10f of the construction machine 10, captured images, and shape information of the work area.
[0021] The image generating unit 212 generates and outputs guidance information (guide image) that supports the operation of the operator based on the acquired information. Here, the image generating unit 212 changes the display form of the guide image according to the distance between the work area and the work tool.
[0022] (Work support processing) Next, the work support process will be described with reference to FIGS. First, the control unit 21 of the operation support device 20 executes an initial setting process (step S11). Specifically, the camera 11 and the distance measuring device 12 are installed on the construction machine 10. As described above, the camera 11 and the distance measuring device 12 are fixed to the front side of the rotating part 10c of the construction machine 10 so as to overlook the excavation area as the working area.
[0023] As shown in FIG. 5, the position of the blade tip of the bucket 10f in the captured image can be identified from the position of the fixed camera 11 and the state (rotation angle) of the boom 10d, arm 10e, and bucket 10f from the sensor 13. In addition, as the distance to the subject (e.g., bucket 10f) increases, the number of pixels representing the subject in the captured image decreases. Therefore, a reference device M1 with a known reference length is placed in front of the rotating part 10c of the construction machine 10 at a predetermined distance from the rotating part 10c.
[0024] Then, the control unit 21 of the operation support device 20 is set to an initial setting mode. In this initial setting mode, the information acquisition unit 211 of the control unit 21 acquires a captured image of the reference unit M1 captured by the camera 11. Next, the image generation unit 212 calculates the number of pixels in which the reference unit M1 appears in the captured image. Next, the image generation unit 212 generates a conversion function that converts the number of pixels into a number that represents a length in the captured image. Then, the image generation unit 212 stores the generated conversion function in memory. Note that the conversion function may be manually created using an image and stored in memory.
[0025] Next, during work, the control unit 21 of the operation support device 20 executes a process of acquiring work information (step S12). Specifically, the information acquisition unit 211 of the control unit 21 acquires status information (rotation angles) of the boom 10d, arm 10e, and bucket 10f of the construction machine 10 from the control device 10a. This status information makes it possible to identify the position of the bucket 10f in the captured image. In addition, the information acquisition unit 211 acquires the captured image and shape information of the excavation area from the camera 11 and distance measuring device 12, respectively.
[0026] Next, the control unit 21 of the operation assistance device 20 executes a height calculation process (step S13). In this embodiment, the location directly below the cutting edge position is identified as the working area where the bucket 10f comes into contact. Then, the distance from the cutting edge (power tool) to the working area is calculated as the height. Therefore, the image generation unit 212 of the control unit 21 calculates the cutting edge position of the bucket 10f from the status information of the boom 10d, the arm 10e, and the bucket 10f.
[0027] Here, as shown in Fig. 6, the horizontal plane on which the construction machine 10 is grounded is taken as the reference plane P1. Then, the height h1 from the reference plane and the horizontal distance d1 from the front surface of the rotating part 10c are calculated as the cutting edge position. Furthermore, the image generating unit 212 identifies the shape SH1 of the ground surface in the working area based on the shape information acquired from the distance measuring device 12. Next, the image generating unit 212 identifies the height h2 of the ground surface at the horizontal distance d1 where the cutting edge of the bucket 10f is located in the shape SH1. Then, the image generation unit 212 calculates the height h3 based on the difference between the height h1 and the height h2.
[0028] Next, the control unit 21 of the operation support device 20 executes the display process of the guide image (step S14). Specifically, the image generation unit 212 of the control unit 21 generates a horizontal bar-shaped guide image corresponding to the blade tip of the bucket 10f according to the horizontal distance d1. In this case, the image generation unit 212 determines the reference length L0 of the guide image when the blade tip of the bucket 10f contacts the work area using a conversion function according to the horizontal distance d1. Next, the image generation unit 212 changes the guide length Lx according to the height h3. Here, when the height h3 = 0, an adjustment function f(h3) that monotonically increases or decreases so that the guide length Lx becomes the reference length L0 is used.
[0029] As shown in FIG. 7(a), when the height is h31 from the ground surface, the guide length Lx of the guide image G1 is the length L1. On the other hand, as shown in FIG. 7(b), when the height is h32 lower than h31, the guide length Lx of the guide image G1 monotonically decreases to the length L2 (<L1). Thereby, the length is changed as the display form of the guide image.
[0030] Then, in the monitor image in which the image generation unit 212 displays the captured image on the display device H13, a guide image is generated and superimposed on the work area where the bucket 10f contacts below the blade tip of the bucket 10f. Specifically, the guide image is generated with the number of pixels corresponding to the guide length Lx at a position corresponding to the number of pixels of the height h3 below the bucket 10f.
[0031] As shown in FIG. 8(a), in the monitor image 500, the guide image G1 is displayed below the bucket 10f. Then, as shown in FIG. 8(b), in the monitor image 510, when the bucket 10f approaches the ground surface, the guide image G1 becomes shorter. In this case, when the bucket 10f contacts the work area, the guide image G1 converges to the same length as the lateral width of the blade tip of the bucket 10f.
[0032] (action) According to this embodiment, the guide image G1 whose length changes according to the height of the bucket 10f is displayed on the monitor image, so that the operator can easily grasp the distance to the contact position of the bucket 10f.
[0033] According to this embodiment, the following effects can be obtained. (1) In this embodiment, the control unit 21 of the operation support device 20 executes an initial setting process (step S11). The image captured by the camera 11 differs depending on the placement position of the camera 11. By performing the initial setting, a conversion function for generating the guide image G1 can be calculated.
[0034] (2) In this embodiment, the control unit 21 of the operation assistance device 20 executes a process for acquiring work information (step S12). This allows the user to operate the construction machine 10 from a remote location while checking the monitor image.
[0035] (3) In this embodiment, the control unit 21 of the operation assistance device 20 executes a height calculation process (step S13), which makes it possible to identify the position of the blade tip of the bucket 10f of the construction machine 10.
[0036] (4) In this embodiment, the control unit 21 of the operation assistance device 20 executes a display process of a guide image (step S14). This allows the operator to easily grasp the distance to the contact position of the bucket 10f. For example, when operating the bucket 10f of the construction machine 10, the operator can easily recognize the relative positional relationship between the excavation point and the bucket 10f. Then, the operation information input by the operator to the input device H12 can be instructed to the control device 10a of the construction machine 10 via the remote operation device 15. As a result, productivity and safety can be improved.
[0037] This embodiment can be modified as follows: This embodiment and the following modifications can be combined with each other to the extent that there is no technical contradiction. In the above embodiment, the operation assistance device 20 is used. The hardware configuration is not limited to this. For example, a separate operator terminal may be provided in a server-client configuration. The operation assistance device 20 may also be configured by cloud computing. When the remote operation device 15 is used, an external operation assistance device 20 is utilized. For this reason, the remote operation device 15 and the operation assistance device 20 are configured separately, but they may also be configured as an integrated device. In the above embodiment, the sensor 13 detects the states (rotation angles) of the boom 10d, the arm 10e, and the bucket 10f. As long as the position of the bucket 10f can be identified, there is no limitation to using the sensor 13. Information obtained from the drive devices (hydraulic cylinders, etc.) of the boom 10d, the arm 10e, and the bucket 10f may also be used.
[0038] In the above embodiment, a backhoe is assumed as the construction machine 10, but the construction machine 10 is not limited to a backhoe. It can be applied to a construction machine 10 equipped with a work tool on an arm. For example, it can be used for demolition construction machines (nibblers) and lifting machines (cranes). In this case, a work area is specified according to the movement of the work tool of each construction machine 10, and a guide image of a size according to the distance from the work tool is displayed in this work area.
[0039] In the above embodiment, the control unit 21 of the operation support device 20 executes an initial setting process (step S11). In this case, a reference unit M1 for a reference length is placed at a predetermined position. If the length is known, it is not necessary to use the reference unit M1. For example, the size of the tool (e.g., the width of the bucket 10f) may be used as the reference length. In this case, the boom 10d and the arm 10e are operated to place the bucket 10f at a predetermined distance, and an image of this bucket 10f is used.
[0040] In the above embodiment, a LIDAR (Light Detection and Ranging) device is used as the distance measuring device 12. However, as long as the shape of the work area can be specified, the distance measuring device is not limited to a LIDAR device.
[0041] In the above embodiment, it is assumed that the construction machine 10 is placed on the reference plane P1, which is a horizontal plane. For example, it may be assumed that work is performed on an inclined surface. In this case, the construction machine 10 is provided with an inclinometer that measures the inclination angle of the swivel unit 10c. Then, the height is calculated taking this inclination angle into consideration.
[0042] In the above embodiment, when the bucket 10f is in a high position, a long guide image G1 is displayed, and when the bucket 10f comes into contact with the working area, the guide image G1 is converged to the same length as the width of the blade tip of the bucket 10f. The method of displaying the guide image G1 is not limited to this. For example, when the distance from the installation surface is far, a short guide image may be displayed, and as the bucket 10f approaches, the guide image may be monotonically increased to extend to the length corresponding to the width of the blade tip.
[0043] The thickness (display form) of the guide image G1 may be changed according to the distance. In this case, the thickness is changed so as to monotonically increase or decrease and converge to a predetermined thickness corresponding to the cutting edge when the guide image G1 comes into contact with the working area. Also, the brightness (display form) of the guide image G1 may be changed. For example, as the distance decreases, the brightness of the guide image may be monotonically changed to brighten the guide image and approach the upper brightness limit.
[0044] Next, the technical ideas that can be understood from the above-described embodiment and other examples will be described below. (a) the control unit, Acquire shape information of the work area from the construction machine; The work support system according to claim 1, wherein the distance is calculated based on the shape information.
[0045] (b) the control unit A guide image having a larger size is generated as the distance increases, 2. The work support system according to claim 1, wherein the size is converged to a size of the power tool when the power tool comes into contact with the working area.
[0046] (c) the control unit A smaller guide image is generated as the distance increases, 2. The work support system according to claim 1, wherein the size is converged to a size of the power tool when the power tool comes into contact with the working area.
[0047] (d) the control unit Acquire a photographed image including a reference device of a predetermined size; The work support system according to any one of claims 1, (a) to (c), characterized in that a conversion function for creating the guide image is generated based on the size of a reference device in the captured image.
[0048] (e) The work support system according to any one of (a) to (d), characterized in that the control unit controls the construction machine to be a backhoe and the work tool to be a bucket. [Explanation of symbols]
[0049] A1...work support system, G1...guide image, 10...construction machinery, 10b...traveling unit, 10c...rotating unit, 10d...boom, 10e...arm, 10f...bucket, 11...camera, 12...distance measuring device, 13...sensor, 15...remote control device, 20...operation support device, 21...control unit, 111...information acquisition unit, 112...image generation unit.
Claims
1. A work support system including a control unit connected to a construction machine equipped with a work tool, The control unit: Acquiring an image of a work area from the construction machine; Calculating a distance from the work tool to the working area; a guide image of a size according to the distance is outputted at a position where the work tool contacts the work area, so as to be superimposed on the image;
2. A method for performing work assistance using a work assistance system including a control unit connected to a construction machine equipped with a work tool, comprising: The control unit: Acquiring an image of a work area from the construction machine; Calculating a distance from the work tool to the working area; a guide image of a size according to the distance being outputted at a position where the work tool comes into contact with the work area, the guide image being superimposed on the image;
3. A program for performing work assistance using a work assistance system having a control unit connected to a construction machine having a work tool, The control unit, Acquiring an image of a work area from the construction machine; Calculating a distance from the work tool to the working area; a work support program that functions as a means for outputting a guide image of a size corresponding to the distance, superimposed on the image, at a position where the work tool abuts against the work area.
Citation Information
Patent Citations
Display control device and display method
JP2022055808A
