System for supporting remote construction for construction machine

Abstract

One aspect of the present disclosure relates to a remote construction support system for a construction machine, comprising: a communication device that controls communication with the construction machine; a remote operation lever that remotely operates the construction machine; a remote display that outputs a photographed image of a surrounding area photographed by a photographing device; and an image control device that superimposes and displays an image of a work device projected onto the ground on the photographed image , wherein the image control device determines whether to display the projected image, on the basis of at least one of a position of an attachment with respect to the ground, an operation direction of the remote operation lever, and an operation amount of the remote operation lever.

Description

Remote construction support system for construction equipment

[0001] The present disclosure relates generally to construction machinery. In a specific aspect, the present disclosure relates to a remote construction support system for construction machinery. The present disclosure may be applied to large vehicles such as trucks, buses, and construction equipment. While the present disclosure may be described with respect to specific vehicles, the present disclosure is not limited to any particular vehicle.

[0002] An excavator is a type of construction machine that performs various tasks, such as digging the ground at construction sites, loading work to transport soil, excavating work to create foundations, demolition work to dismantle buildings, leveling work to prepare the ground, and leveling work to level the ground.

[0003] In the case of work using an excavator, depending on the work location, environment, and conditions, work is often done in dangerous areas or under poor conditions, and in such cases, it is often very difficult for the worker to board and operate the excavator.

[0004] Recently, a remote construction support system for construction equipment has been used that allows the operator to control the excavator without having to board the excavator directly.

[0005] This conventional remote construction support system for construction machinery installs multiple cameras on an excavator, and displays the images taken by the excavator on a display, allowing the operator to observe the surrounding work space while the excavator is in operation.

[0006] However, there was a limitation in that it was difficult for workers to intuitively grasp the distance between the work device and the work object on a 2D display.

[0007] According to a first aspect of the present disclosure, a remote construction support system for a construction machine including an undercarriage, an upper swing body rotatably coupled to the undercarriage, a work device mounted on the upper swing body, a sensor device for detecting attitude information of the construction machine, and a photographing device for photographing a surrounding area of ​​the construction machine, the system including a communication device for controlling communication with the construction machine, a remote operation lever for remotely operating the construction machine, a remote display for outputting a photographed image of a surrounding area photographed by the photographing device, and an image control device for superimposing a projection image of the work device onto the ground on the photographed image, wherein the image control device determines whether to display the projection image based on at least one of a position of the attachment with respect to the ground, an operation direction of the remote operation lever, and an operation amount of the remote operation lever. The first aspect of the present disclosure can pursue a remote construction support system for a construction machine capable of intuitively grasping the distance between a work device and a work object. The technical advantage is that the projection image of the work device is displayed only in work situations that require the projection image based on the position of the bucket, the direction of operation of the remote control lever, and the amount of operation of the remote control lever, so that not only can the distance between the work device and the work object be intuitively identified, but the usability of the projection image is improved.

[0008] Optionally, in some examples, the image control device may display a projection image of the attachment projected onto the ground.

[0009] Optionally, in some examples, the image control device may display the projection image when the attachment is positioned higher than the ground during boom-down operation of the remote control lever.

[0010] Optionally, in some examples, the image control device may not display the projection image if the attachment is positioned below the ground during boom-down operation of the remote control lever.

[0011] Optionally, in some examples, the image control device can display the projection image when the attachment is positioned below a predetermined height and the boom-up operation amount is below a predetermined value during boom-up operation of the remote control lever.

[0012] Optionally, in some examples, the image control device may not display the projection image if the attachment is positioned higher than a predetermined height during boom-up operation of the remote control lever or if the boom-up operation amount exceeds a predetermined value.

[0013] Optionally, in some examples, the attachment is a bucket, and the projection image may include an image representing the bucket tip.

[0014] Optionally, in some examples, the image control device may display an image representing the swing range of the work device when the work device is at its maximum length and an image representing the operating range of the work device.

[0015] The above-described aspects, appended claims, and / or examples disclosed herein above and hereinafter may be suitably combined with one another as will be apparent to one of ordinary skill in the art.

[0016] Additional features and advantages are set forth in the following description, claims, and drawings, and in part will be readily apparent to those skilled in the art from the foregoing or may be recognized by practicing the teachings herein.

[0017] With reference to the accompanying drawings, a more detailed description of embodiments of the present disclosure, cited as examples, follows below.

[0018] Figure 1 is a block diagram showing the basic configuration of a construction machine according to one aspect of the present disclosure.

[0019] Figure 2 is a perspective view illustrating a construction machine according to one aspect of the present disclosure.

[0020] FIG. 3 is a block diagram illustrating a remote construction support system for construction machinery and a basic configuration of construction machinery according to one aspect of the present disclosure.

[0021] FIG. 4 is an example of a remote display and remote control lever according to one aspect of the present disclosure.

[0022] FIG. 5A is a schematic drawing showing the posture of a construction machine according to one aspect of the present disclosure, and FIG. 5B is an example of an image displayed on a remote display when the construction machine is in the state of FIG. 5A.

[0023] Figures 6a and 6b are other examples of images displayed on a remote display.

[0024] FIGS. 7A to 7C are schematic drawings showing different postures of a construction machine according to one aspect of the present disclosure.

[0025] Figure 8 is an example of a case where a projection image is not displayed on a remote display.

[0026] The aspects described below represent information necessary to enable a person skilled in the art to practice the present disclosure.

[0027] FIG. 1 is a block diagram showing the basic configuration of a construction machine according to one aspect of the present disclosure, and FIG. 2 is a perspective view showing a construction machine according to one aspect of the present disclosure.

[0028] Referring to FIGS. 1 and 2, a construction machine (100) such as an excavator has a cab (110), a lower body (120), an upper swivel body (130) that is swivellably installed on the lower body (120), a work device (140) that is movably installed on the upper body (130) in the vertical direction, and a hydraulic actuator (150).

[0029] The work device (140) is formed as a multi-joint and includes a boom (141) whose rear end is rotatably supported on an upper swivel body (130), an arm (142) whose rear end is rotatably supported on the front end of the boom (141), and a bucket (143) as an attachment rotatably installed on the front end side of the arm (142).

[0030] In addition to the bucket (143), there are various attachments such as a breaker, drill, grab, crusher, tilt-rotator, grapple, and thumb bucket, and the optimal attachment can be attached to the arm (142) depending on the type of specific work being performed. Hereinafter, an embodiment in which a bucket (143) is mounted as an attachment at the tip of the arm (142) will be described.

[0031] The hydraulic actuator (150) includes a boom actuator (151), an arm actuator (152), and a bucket actuator (153). When operating fluid is supplied according to the operator's operation of the operating lever, the boom actuator (151), the arm actuator (152), and the bucket actuator (153) operate the boom (141), the arm (142), and the bucket (143), respectively.

[0032] The construction machine (100) may further include a sensor device (160), a photographing device (170), an electronic control device (180), and a communication means (190).

[0033] The sensor device (160) may include a GNSS (Global Navigation Satellite System) sensor and an attitude measurement sensor.

[0034] The GNSS sensor can detect the current location and attitude of the construction machine (100) in real time. Specifically, the GNSS sensor can measure the current latitude, longitude, and altitude of the construction machine (100) in real time, thereby detecting the current location of the construction machine in real time. In addition, the GNSS sensor can detect the current attitude of the construction machine (100) in real time. The real-time information on the current location and attitude of the construction machine (100) detected by the GNSS sensor may be digital data.

[0035] The attitude measurement sensor is configured as a plurality of inertial measurement units (IMUs) and angle sensors, and can measure the position, attitude and / or angle of the upper swivel body (130), boom (141), arm (142) and bucket (143), and the inclination of the main body of the construction machine (100). For example, an inertial measurement device may be placed on each of the boom (141), arm (142) and bucket (143), thereby detecting the angles of each of the boom (141), arm (142) and bucket (143).

[0036] Information measured by the sensor device (160) is provided to the electronic control device (180) or to the remote construction support system of the construction machine through a communication means (190).

[0037] The photographing device (170) photographs the surrounding area of ​​the construction machine (100) using one or more cameras and Lidar installed on the construction machine. This photographing device (170) acquires images of the front, rear, left, and right sides of the construction machine (100) while driving and working, and transmits the acquired images to a remote construction support system.

[0038] Information measured by the photographing device (170) is provided to the electronic control device (180) or to the remote construction support system via a communication means (190).

[0039] The electronic control device (180) is configured by an operation processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes operation processing according to the software on the data.

[0040] The electronic control device (180) generates an electrical signal corresponding to the operation of the remote operation lever of the remote construction support system to control the operation of the lower driving body (120), upper swing body (130), and work device (140).

[0041] In addition, the electronic control device (180) can calculate the distance between the bucket (143) and the ground based on digital data such as the current latitude, longitude, altitude, and current attitude of the construction machine (100) received from the sensor device (160), the position and / or attitude of the upper swivel body (130), boom (141), arm (142), and bucket (143) detected by the attitude measurement sensor, and the inclination of the construction machine (100). The distance information between the bucket (143) and the ground can be provided to the remote construction support system of the construction machine through the communication means (190).

[0042] The communication means (190) is configured to communicate with a remote construction support system located outside the construction machine (100) and is connected to a commercial communication network. Examples of commercial communication networks include a 3G mobile communication network, a 4G LTE mobile communication network, and a 5G mobile communication network. The communication means (190) is connected to a commercial communication network, i.e., a mobile communication network, and provides an Internet connection service to the construction machine (100). However, the communication means (190) is not limited thereto, and may be connected to a dedicated communication network including a private WiFi network or a private 5G network.

[0043] In addition, the communication means (190) is connected to the internal network of the construction machine (100) and communicates with the electronic control device (180). The internal network of the construction machine (100) may be any one of CAN (Controller Area Network), Ethernet, FlexRay, LIN (Local Interconnect Network), etc., but it is preferable to use Ethernet.

[0044] FIG. 3 is a block diagram illustrating a remote construction support system for construction machinery and a basic configuration of construction machinery according to one aspect of the present disclosure.

[0045] Referring to FIG. 3, a remote construction support system (200) for a construction machine according to one embodiment of the present invention is a system for remotely controlling a construction machine (100) in a remote location, and includes a communication device (210), a remote operation lever (220), a remote display (230), and an image control device (240) configured to communicate with the construction machine (100) through a commercial communication network.

[0046] FIG. 4 is an example of a remote display and operating lever according to one aspect of the present disclosure.

[0047] Referring to FIG. 4, the remote control lever (220) may be an electric joystick, and preferably, the remote control lever (220) may generate an electric signal in proportion to the amount of manipulation by the operator.

[0048] The remote control lever (220) can control the rotation of the upper swivel body, the up and down movement of the boom, and the rotation of the arm and bucket. The remote control lever (220) can be configured to include a first remote control lever (221) and a second remote control lever (222).

[0049] The first remote control lever (221) is used to operate the arm and the upper swivel body. When the first remote control lever (221) is operated toward the front of the operator, the arm actuator is operated to perform an arm-out operation, that is, to open the arm. When the first remote control lever (221) is operated toward the rear of the operator, the arm actuator is operated to perform an arm-in operation, that is, to close the arm. In addition, when the first remote control lever (221) is operated toward the left of the operator, the swivel hydraulic motor is operated to perform a left swivel operation, that is, to turn the upper swivel body to the left. When the first remote control lever (221) is operated toward the right of the operator, the swivel hydraulic motor is operated to perform a right swivel operation, that is, to turn the upper swivel body to the right.

[0050] The second remote control lever (222) is used to operate the boom and the bucket. When the second remote control lever (222) is operated toward the front of the worker, the boom actuator is operated to perform a boom-down operation, that is, to lower the boom. When the second remote control lever (222) is operated toward the rear of the worker, the boom actuator is operated to perform a boom-up operation, that is, to raise the boom. In addition, when the second remote control lever (222) is operated toward the left of the worker, the bucket actuator is operated to perform a bucket-in operation, that is, to close the bucket. When the second remote control lever (222) is operated toward the right of the worker, the bucket actuator is operated to perform a bucket-out operation, that is, to open the bucket.

[0051] However, this is not limited thereto, and the operation methods of the first remote control lever (221) and the second remote control lever (222) can of course be changed by the operator. For example, the first remote control lever (221) can be used to operate the boom and bucket, and the second remote control lever (222) can be used to operate the arm and the upper swivel body.

[0052] The remote display (230) outputs a captured image of a surrounding area captured by a shooting device of a construction machine and a projected image generated by an image control device (240), and may include one or more displays.

[0053] For example, the remote display (230) may include a first display (231) positioned in front of the seat, a second display (232) positioned above the first display (231), a third display (233) positioned on the left side of the first display (231), a fourth display (234) positioned on the right side of the first display (231), and a fifth display (235) positioned below the first display (231).

[0054] Here, the first display (231) can output a captured image for the front view of the construction machine, the second display (232) can output captured images for the rear view and upper view of the construction machine, the third display (233) can output captured images for the left mirror view of the construction machine, the fourth display (234) can output captured images for the right mirror view of the construction machine, and the fifth display (235) can output captured images for the lower view of the construction machine.

[0055] The image control device (240) is configured by an operation processing device (a single core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes operation processing according to the software on the data.

[0056] The image control device (240) generates a projection image of the work device on the ground or work area and displays it on the remote display (230).

[0057] FIG. 5A is a drawing schematically showing the posture of a construction machine according to one aspect of the present disclosure, and FIG. 5B is an example of an image displayed on a remote display when the construction machine is in the state of FIG. 5A.

[0058] Referring to FIGS. 5A and 5B, the image control device (240) generates a projection image (BI) that projects the bucket (143) of the work device (140) onto the ground. The image control device (240) determines the position coordinates of the work device using data acquired from the sensor device of the construction machine, obtains the coordinates at which the bucket is projected onto the ground, and converts these into 2D coordinates to generate the projection image (BI). The projection image (BI) generated by the image control device (240) is displayed on the first display (231) of the remote display (230) together with the surrounding area of ​​the construction machine (100) captured by the capture device.

[0059] The image control device (240) can display a bucket tip image (BTI) indicating the bucket tip on the bucket projection image (BI). At this time, the bucket tip image (BTI) can be generated in various shapes, such as a dotted line or triangle.

[0060] The image control device (240) can generate different colors of the bucket tip image (BTI) depending on the distance between the bucket (143) and the ground. For example, various methods can be applied, such as a gradient method in which the color gradually changes according to the distance. In addition, the image control device (240) can generate a distance information image (DI) that shows the correlation between distance and color. Here, the image control device (240) can calculate the position of the bucket with respect to the ground using data acquired from the sensor device of the construction machine.

[0061] Figures 6a and 6b are other examples of images displayed on a remote display.

[0062] Referring to FIG. 6A, the image control device (240) can generate a swing radius image (SI) indicating a trajectory during a swing based on the maximum working range of the work device. The image control device (240) can generate a motion range image (RI) indicating the motion range of the work device. Through the swing radius image (SI) and the motion range image (RI), the worker can easily determine how far work is possible on the 2D screen from the current position, and can easily identify whether there are hazardous materials or people within the working radius. In addition, the image control device (240) can display a projection image (BI) of the bucket within the motion range image (RI).

[0063] Referring to FIG. 6b, when a swing operation is input to the remote control lever, the image control device (240) can display an image (BSI) indicating the swing range of the bucket according to the swing operation.

[0064] On the other hand, if the projection image is always displayed on the remote display, it will be perceived as unnecessary noise in work situations where the projection image is not needed, reducing its usefulness.

[0065] Accordingly, the image control device (240) determines whether to display the projection image based on the position of the bucket with respect to the ground, the operating direction of the remote control lever, and the operating amount of the remote control lever.

[0066] FIGS. 7a to 7c are schematic drawings showing different postures of a construction machine according to one aspect of the present disclosure, and FIG. 8 is an example of a case where a projection image is not displayed on a remote display.

[0067] As shown in Fig. 7a, in a boom-down situation, the image control device (240) generates a projection image (BI) of the bucket (143) projected onto the ground and displays it on a remote display so that the operator can intuitively grasp the relative positions of the bucket and the ground. Furthermore, in dark-in and dark-out situations, the image control device (240) generates a projection image (BI) of the bucket (143) projected onto the ground.

[0068] As shown in Fig. 7b, when the bucket (143) is positioned on the ground or lower than the ground, if a projection image of the bucket (143) is displayed on the remote display, the projection image and the captured image of the bucket (143) taken by the photographing device may overlap, making it difficult to determine the position of the bucket (143). Therefore, the image control device (240) does not display a projection image when the bucket (143) is positioned below the ground level during boom-down operation of the remote control lever.

[0069] As shown in Fig. 7c, in a situation where the boom (141) is raised to separate the bucket (143) from the ground, information on the distance between the bucket (143) and the ground is generally unnecessary. However, in a grading operation where the worker must slightly raise the boom (141), information on the distance between the bucket (143) and the ground is essential.

[0070] Accordingly, the image control device (240) displays a projection image (BI) of the bucket when the bucket (143) is positioned below a predetermined height threshold value (PHT) and the boom-up operation amount is below a predetermined operation amount threshold value. At this time, the image control device (240) can determine whether the bucket (143) is positioned within the threshold value (PHT) based on the tip of the bucket (143).

[0071] In addition, as shown in FIG. 8, the image control device (240) does not display a projection image of the bucket when the bucket (143) is positioned above a predetermined height threshold (PHT) or when the boom-up operation amount exceeds a predetermined operation amount threshold.

[0072] As described above, the remote construction support system for a construction machine according to one embodiment of the present invention displays a projection image only in a work situation requiring a projection image of a work device based on the position of the bucket, the operation direction of the remote operation lever, and the operation amount of the remote operation lever, so that the utility of the projection image can be improved.

[0073] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly dictates otherwise. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items. It will also be understood that the terms “comprises,” “comprising,” “includes,” and / or “including,” when used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0074] While terms such as "first" and "second" may be used herein to describe various components, it is to be understood that these components are not limited by these terms. These terms are used solely to distinguish one component from another. For example, a first component could be referred to as a "second component," and similarly, a second component could be referred to as a "first component," without departing from the scope of the present disclosure.

[0075] Relative terms such as "below" or "above" or "upper" or "lower" or "horizontal" or "vertical" may be used herein to describe one element's relationship to another, as illustrated in the drawings. It will be understood that these terms, and those discussed above, are intended to encompass different orientations of the device in addition to the orientations depicted in the drawings. When an element is referred to as being "connected" or "coupled" to another element, it will be understood that it may be directly connected or coupled to the other element, or that intervening elements may be present. Conversely, when an element is referred to as being "directly connected" or "directly coupled" to another element, no intervening elements are present.

[0076] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. Furthermore, it is to be understood that terms used herein should be interpreted to have a meaning consistent with their meaning within the context of this specification and related technologies, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

Claims

1. A remote construction support system for a construction machine, comprising: a lower body, an upper swivel body rotatably connected to the lower body, a work device mounted on the upper swivel body and including a boom, an arm, and an attachment, a sensor device for detecting posture information of the construction machine, and a photographing device for photographing an area around the construction machine. A communication device that controls communication with the above construction machine; A remote control lever for remotely operating the above construction machine; A remote display that outputs a captured image of a surrounding area captured by the above-mentioned shooting device; and An image control device that displays a projection image projected onto the ground by the above work device by superimposing it on the above photographed image; The above image control device, A remote construction support system for construction machinery, which determines whether to display the projection image based on at least one of the position of the attachment with respect to the ground, the operating direction of the remote operation lever, and the operating amount of the remote operation lever.

2. In paragraph 1, The above image control device, A remote construction support system for construction machinery that displays a projection image of the above attachment projected onto the ground.

3. In paragraph 2, The above image control device, A remote construction support system for a construction machine that displays the projection image when the attachment is positioned higher than the ground during boom-down operation of the remote control lever.

4. In paragraph 2, The above image control device, A remote construction support system for construction machinery, which does not display the projection image when the attachment is located below the ground during boom-down operation of the remote control lever.

5. In paragraph 2, The above image control device, A remote construction support system for a construction machine, which displays the projection image when the attachment is located below a predetermined height and the boom-up operation amount is below a predetermined value during boom-up operation of the above remote control lever.

6. In paragraph 2, The above image control device, A remote construction support system for construction machinery, which does not display the projection image when the attachment is positioned higher than a predetermined height or the boom-up operation amount exceeds a predetermined value during boom-up operation of the remote control lever.

7. In paragraph 2, The above attachment is a bucket, The above projection image is, A remote construction support system for construction machinery, comprising an image representing a bucket tip.

8. In paragraph 1, The above image control device, A remote construction support system for construction equipment, which displays an image showing the swing range when the above work device is at its maximum length and an image showing the operating range of the above work device.

Images