Work assistance system and work assistance method

Abstract

The present invention is a work assistance system that enables an operator to recognize a positional relationship between a work mechanism and a target object with high accuracy. This work assistance system comprises: an image generation unit; and a change processing unit. The image generation unit generates a work assistance image in which index images (M1, M2) indicating ground projection locations (P21, P22), which are locations where designated locations (P11, P12) of a tip attachment (445) are projected onto the ground surface, are superimposed on the work environment image. When at least one of the work machine, the work site, and the tip attachment (445) is a change target, the change processing unit executes specification information change processing for changing specification information of the work machine serving as a foundation for estimation of the ground projection location so as to adapt to the change target.

Description

Work support system and work support method

[0001] The present invention relates to a technique for assisting an operator in remotely operating a work machine such as a hydraulic excavator.

[0002] Patent Literature 1 discloses a technology for generating a display image in which a captured image and a cutting edge position image are superimposed to improve work efficiency. The captured image includes a bucket and a work object photographed by a camera mounted on a work machine. The cutting edge position image is an image in which the bucket cutting edge is projected vertically toward the work object.

[0003] In this technology, in order to generate the image, it is necessary to identify the position of the ground projection point, which is the point where a specified point on the end attachment is projected onto the ground, and specification information of the work machine including the end attachment is used for this identification. However, if at least one of the work machine, work site, and end attachment that are operated by the operator is changed, the actual position of the ground projection point will change, which may reduce the accuracy with which the operator can recognize the position of the ground projection point through the image.

[0004] Patent No. 6777375

[0005] An object of the present invention is to provide a system and method that allows an operator to maintain accuracy in recognizing the position of a ground projection point regardless of changes to at least one of the work machine, the work site, and the end attachment.

[0006] What is provided is a work support system that includes an image generation unit and a change processing unit. The image processing unit estimates a ground projection point, where a designated point, which is a pre-designated point on the end attachment of the work machine, is projected onto the ground, based on specification information of the work machine, and generates a work support image in which an index image indicating the ground projection point is superimposed on a work environment image acquired by an imaging device. The work support image is an image of the surroundings of the work machine, including the end attachment and the ground. The change processing unit executes specification information change processing when at least one of the work machine, the work site where the work machine performs work, and the end attachment is subject to change. The specification information change processing is processing for adapting the specification information, which is the basis for estimating the ground projection point, to the subject of change, and may be processing for changing the specification information, or processing for having the operator operating the work machine change the specification information.

[0007] Also provided is a work support method that includes the steps of: acquiring, by an imaging device, a work environment image that is an image of the surroundings of a work machine and includes the end attachment of the work machine and the ground; estimating a ground projection point, where a designated point that is a pre-designated point on the end attachment is projected onto the ground, based on specification information of the work machine, and generating a work support image in which an index image indicating the ground projection point is superimposed on the work environment image; and, if at least one of the work machine, the work site where the work machine performs work, and the end attachment is subject to change, executing a specification information change process to adapt the specification information that forms the basis for estimating the ground projection point to the subject to change.

[0008] FIG. 1 is a diagram showing a work support system according to an embodiment of the present invention. FIG. 2 is a plan view showing a remote control device constituting the work support system. FIG. 3 is a side view of a work machine according to the embodiment. FIG. 4 is a flowchart showing processing performed by the work support system. FIG. 5 is a side view showing a first example of a simulated image included in a specification information setting image provided in the embodiment. FIG. 6 is a side view showing a second example of a simulated image included in a specification information setting image provided in the embodiment. FIG. 7 is a front view showing a third example of a simulated image included in a specification information setting image provided in the embodiment. FIG. 8 is a perspective view showing a fourth example of a simulated image included in a specification information setting image provided in the embodiment. FIG. 9 is a perspective view showing a fifth example of a simulated image included in a specification information setting image provided in the embodiment. FIG. 10 is a diagram showing a first example of an index image superimposed on a work environment image in the embodiment. FIG. 11 is a diagram showing a second example of an index image superimposed on the work environment image.

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described with reference to the drawings. Fig. 1 shows a work machine 40 and a remote control device 20 for remotely controlling the work machine 40.

[0010] 3 shows a crawler hydraulic excavator as an example of the work machine 40. The work machine 40 shown in this example includes a lower traveling body 41 including a pair of left and right crawlers, an upper rotating body 42 that is rotatably mounted on the lower traveling body 41 via a swing drive mechanism, and a working mechanism 44. The upper rotating body 42 includes a cab 43 (operator's compartment), which forms the left front portion of the upper rotating body 42. The working mechanism 44 is connected to the right front portion of the upper rotating body 42 so as to be able to perform work operations. The cab 43 may be configured to be driven to move up and down relative to the upper rotating body 42 by a lifting drive mechanism (not shown) that has a link mechanism and an actuator.

[0011] The working mechanism 44 includes a boom 441, an arm 443, a tip attachment 445, and a plurality of hydraulic cylinders. The boom 441 is attached to the upper rotating body 42 so as to be able to rise and fall. The arm 443 is rotatably connected to the tip of the boom 441. The tip attachment 445 is rotatably connected to the tip of the arm 443. Examples of the tip attachment 445 include a bucket, a grapple, an irrigation hook, a breaker, and a lifting magnet. Each of the plurality of hydraulic cylinders is an actuator attached to the working mechanism 44 and capable of extending and retracting to move the working mechanism 44.

[0012] The plurality of hydraulic cylinders include a boom cylinder 442, an arm cylinder 444, and a tip attachment cylinder 446. The boom cylinder 442 is interposed between the boom 441 and the upper rotating body 42, and extends and retracts upon receiving a supply of hydraulic oil to rotate the boom 441 in the hoisting direction. The arm cylinder 444 is interposed between the arm 443 and the boom 441, and extends and retracts upon receiving a supply of hydraulic oil to rotate the arm 443 about a horizontal axis relative to the boom 441. The tip attachment cylinder 446 is interposed between the tip attachment 445 and the arm 443, and extends and retracts upon receiving a supply of hydraulic oil to rotate the tip attachment 445 about a horizontal axis relative to the arm 443.

[0013] 1, the remote operation device 20 includes a remote control device 200, a remote input interface 210, and a remote output interface 220. In this embodiment, the remote control device 200, the remote input interface 210, and the remote output interface 220 constitute a work support system. The remote control device 200 is configured by a processing device, and the processing device includes, for example, a single-core processor or a multi-core processor or processor cores constituting the same. The remote control device 200 reads necessary data and software from a storage device such as a memory, and performs processing on the data in accordance with the software.

[0014] The remote input interface 210 includes a remote operation mechanism 211. The remote output interface 220 includes a remote image output device 221, which is a display device, a remote sound output device 222, and a remote wireless communication device 224.

[0015] The remote operation mechanism 211 includes multiple remote operation units, including a remote travel operation unit, a remote swing operation unit, a remote boom operation unit, a remote arm operation unit, and a remote tip attachment operation unit. Each of the remote operation units includes a remote operation lever to which a rotation operation can be applied. The remote operation lever of the remote travel operation device is a travel lever, and a travel operation for moving the undercarriage 41 of the work machine 40 is applied to the travel lever. The travel lever may also serve as a travel pedal. For example, a pedal for receiving a travel operation may be fixed to the base or lower end of the travel lever. The remote operation lever of the swing operation device is a swing lever, and a swing operation for operating a swing motor is applied to the swing lever. The swing motor is a hydraulic motor included in the swing drive mechanism. The remote operation lever of the boom operation device is a boom lever, and a boom operation for moving the boom cylinder 442 is applied to the boom lever. The remote control lever of the arm operating device is an arm lever, and an arm operation is applied to the arm lever to move the arm cylinder 444. The remote control lever of the end attachment operating device is a end attachment lever, and an end attachment operation is applied to the end attachment lever to move the end attachment cylinder 446.

[0016] In the example shown in Fig. 2, the remote control levers included in the remote control mechanism 211 are arranged around a seat St, which is a seating portion on which an operator sits. The seat St shown in Fig. 2 is a high-back chair or the like including a headrest 2200 and a pair of armrests 2221, 2222. However, the seat St may also be a low-back chair without a headrest or a chair without a backrest. The seating portion may have any shape that allows an operator to sit on it.

[0017] Specifically, the remote travel operation unit includes, as the remote control levers, a pair of left and right travel levers 2110 corresponding to the pair of left and right crawlers, respectively, and the pair of left and right travel levers 2110 are arranged side by side in front of the seat St. The remote control mechanism 211 may include a remote control lever that combines the functions of multiple remote control levers. In the example shown in FIG. 2 , the remote control mechanism 211 includes a left operation lever 2111 provided in front of the left frame of the seat St and a right operation lever 2112 provided in front of the right frame of the seat St. The left operation lever 2111 is configured to function as the arm lever when operated in the forward / backward direction and as the swing lever when operated in the left / right direction. Similarly, the right operation lever 2112 is configured to function as the boom lever when operated in the forward / backward direction and as the tip attachment lever when operated in the left / right direction. Alternatively, the remote control mechanism 211 may be configured so that the lever patterns of the respective remote control levers can be arbitrarily changed according to an operation command from an operator.

[0018] The remote image output device 221 can be configured with a single image output unit or multiple image output units. The remote image output device 221 illustrated in FIG. 2 includes multiple image output units, namely, a central image output unit 2210, a left image output unit 2211, and a right image output unit 2212. The central image output unit 2210 is disposed in front of the seat St, the left image output unit 2211 is disposed diagonally forward and to the left of the seat St, and the right image output unit 2212 is disposed diagonally forward and to the right of the seat St. Each of the image output units 2210, 2211, and 2212 has a substantially rectangular screen, i.e., an image display area. The shapes and sizes of the screens of the image output units 2210, 2211, and 2212 may be the same or different. The remote image output device 221 may alternatively be composed of a single curved or bendable image output unit, or may be composed of two or four or more image output units arranged to surround the front of the sheet St.

[0019] 2 , the right edge of the left image output unit 2211 is adjacent to the left edge of the central image output unit 2210 so that the screens of the central image output unit 2210 and the left image output unit 2211 are inclined relative to each other at a predetermined angle θ1. Furthermore, the left edge of the right image output unit 2212 is adjacent to the right edge of the central image output unit 2210 so that the screens of the central image output unit 2210 and the right image output unit 2212 are inclined relative to each other at a predetermined tilt angle θ2. Each of the predetermined tilt angles θ1 and θ2 is set to satisfy, for example, 120°≦θ1≦150°. The predetermined tilt angles θ1 and θ2 may be the same or different from each other.

[0020] The screens of the image output units 2210, 2211, and 2212 may be parallel to the vertical direction or may be inclined relative to the vertical direction. At least one of the image output units 2210, 2211, and 2212 may be divided into a plurality of image output sections. For example, the central image output unit may be composed of image output sections adjacent to each other above and below, and each of the image output sections may have a substantially rectangular screen.

[0021] The remote sound output device 222 includes one or more speakers. The remote sound output device 222 illustrated in Fig. 2 is configured with a plurality of speakers arranged behind the seat St, behind the left armrest, and behind the right armrest. The specifications of the speakers may be the same or different from each other.

[0022] 1, the work machine 40 includes a real machine control device 400, a real machine input interface 410, a real machine output interface 420, and a real machine wireless communication device 422. The real machine control device 400 includes multiple components, each of which is configured by an arithmetic processing device. The arithmetic processing device is, for example, a single-core processor or a multi-core processor, or the processor cores that make up such a processor, and reads the necessary data and software from a storage device such as a memory, and performs arithmetic processing on the data in accordance with the software.

[0023] The real machine input interface 410 includes a real machine operation mechanism 411 , a real machine imaging device 412 , a real machine distance measuring device 414 , and a plurality of real machine sensors 416 .

[0024] The actual machine operation mechanism 411 includes a plurality of actual machine operation levers, which are arranged around a seat arranged inside the cab 43 in the same manner as the remote operation levers of the remote operation mechanism 211. An operation mechanism such as a robot is provided inside the cab 43, and the operation mechanism receives a signal corresponding to an operation given to the remote operation lever, and moves the actual machine operation lever based on the received signal.

[0025] The actual machine imaging device 412 is installed, for example, inside the cab 43, and captures an image including at least a part of the working mechanism 44 through the front window and the left and right side windows. Some or all of the front window and the left and right side windows may be omitted.

[0026] The actual distance measuring device 414 measures the distance from a reference position set on the work machine 40 to surrounding objects that exist around the work machine 40. The surrounding objects include a designated object OBJ, which will be described later. The actual distance measuring device 414 is configured, for example, by a LiDAR or TOF sensor.

[0027] The plurality of actual machine sensors 416 are configured to output signals corresponding to parameter values ​​relating to the state of the work machine 40, such as the hoisting angle of the boom 441 relative to the upper rotating body 42, the rotation angle of the arm 443 relative to the boom 441, the rotation angle of the tip attachment 445 relative to the arm 443, hydraulic oil pressure, and remaining fuel, and input these signals to the actual machine control device 400. The plurality of actual machine sensors 416 include an actual machine positioning device that measures the position of the work machine 40, such as latitude and longitude, and the actual machine positioning device is configured, for example, by GPS or GNSS and, if necessary, a gyro sensor or the like.

[0028] The processing performed by the task support system will be described with reference to the flowchart in Figure 4. In the flowchart, the portion indicated by a combination of "C" and a number indicates processing that determines whether specific data has been sent and / or received, and determines whether to execute processing in a branching direction based on the transmission and / or reception of the data.

[0029] In FIG. 4, the remote control device 20 determines whether or not there has been a change in at least one of the work machine 40, the work site, and the end attachment 445 (STEP 202).

[0030] Whether or not the work machine 40 has been changed may be determined, for example, based on whether or not a work machine designation change operation has been performed. The work machine designation operation is an operation given by an operator through the remote input interface 210 in order to change the work machine 40 that is the object of operation. Whether or not the work machine 40 has been changed may alternatively be determined based on whether or not communication between the work machine 40 before the change and the remote control device 20 has been terminated, and / or whether or not communication between the work machine 40 after the change and the remote control device 20 has been established. Whether or not the work machine 40 has been changed may alternatively be determined based on whether or not there has been a change in the actual machine identifier. The actual machine identifier is an identifier for identifying the work machine 40 that is the object of operation or its specification information, and is transmitted from the work machine 40 and received by the remote control device 20.

[0031] The remote control device 20 may determine whether the end attachment 445 has been changed based on the presence or absence of an attachment identifier. The attachment identifier is an identifier for identifying the end attachment 445 or its specification information, and is transmitted from an information processing terminal such as a smartphone or personal computer owned by the work machine 40 to be operated and / or a worker, and received by the remote control device 20. The attachment identifier may be read by a contactless reader from an RFID tag or barcode attached to the end attachment 445. The contactless reader may be included in the plurality of actual machine sensors 416, or may be mounted on the information processing terminal.

[0032] Alternatively, the remote control device 20 may determine whether the end attachment 445 has been changed based on whether there has been a change in end attachment information. The end attachment information is information that enables the end attachment 445 to be identified, such as a combination of a manufacturer name and model number or an attachment identifier, and is input into an information processing terminal by an information worker and received by the remote control device 20. To authenticate the information processing terminal, specifically to confirm the suitability of the information processing terminal as a source of information for the work machine 40 that is the target of operation, the result of a determination as to whether the position of the information processing terminal is within a designated area based on the position of the work machine 40 may be used, or the result of a determination as to whether communication has been established between the information processing terminal and the work machine 40 may be used.

[0033] The remote control device 20 may be configured to receive a work environment image from the work machine 40 to be operated or from an information processing terminal owned by a worker, analyze the work environment image, thereby recognizing the shape characteristics of the end attachment 445, and determine whether or not the end attachment 445 has been changed based on whether or not the shape characteristics have been changed. The work environment image can be captured by a camera mounted on the actual machine imaging device 412 or the information processing terminal.

[0034] The remote control device 20 may be configured to receive actual machine position information, which is information about the position of the work machine 40, and determine whether or not the work site (work area) containing that position has changed based on whether or not the work site has changed. The position of the work machine 40 can be measured by an actual machine positioning device included in the multiple actual machine sensors 416, and is identified by the latitude and longitude of the work machine 40, and also the altitude as necessary. A work site distribution map that represents the distribution pattern of work sites may be used for this identification. The remote control device 20 may be configured to receive work environment images captured by the actual machine imaging device 412 and analyze the work environment images to identify indicators or objects unique to the work site, and to determine whether or not the work site has changed based on whether or not the indicators, etc. have changed.

[0035] If it is determined that none of the work machine 40, the work site, and the end attachment 445 have been changed (NO in STEP 202), that is, if it is determined that none of the work machine 40, the work site, and the end attachment 445 are subject to change, the remote control device 20 executes a reception process (C21) for receiving three-dimensional image data via the remote wireless communication device 224, as will be described in detail later, and subsequent processes.

[0036] If it is determined that at least one of the work machine 40, the work site, and the end attachment 445 has been changed and is the change target (YES in STEP 202), the remote control device 200 executes specification information change processing. This processing is processing for adapting the specification information that forms the basis for estimating a ground projection location, which is the location where a designated location that has been designated in advance on the end attachment 445 is projected onto the ground, to the changed one of the work machine 40, the work site, and the end attachment 445, i.e., the change target. In other words, the remote control device 200 includes a change processing unit that performs the specification information change processing.

[0037] Specifically, the remote control device 200 displays a specification information setting image on the remote image output device 221 of the remote output interface 220 to allow the operator to confirm the specification information of the work machine 40. The specification information includes specification information of components such as the end attachment 445. The remote control device 200 reads the specification information and the specification information setting image corresponding to at least one of the actual machine identifier and the attachment identifier from a database, or searches within the database. The database is constituted, for example, by a storage device included in the remote control device 200 and / or an external database server.

[0038] The specification information setting image includes a simulated image that simulates the tip attachment 445 that conforms to the specification information. The image indicates that one or more first index points P1 are defined at one or more specified locations on the tip attachment 445. Furthermore, the simulated image may additionally indicate that a second index point P2 is defined by projecting the first index point P1 onto the ground in a pre-specified direction, for example, the vertical direction or the displacement direction of the tip attachment 445, or may additionally indicate an index image M indicating the position of the second index point P2.

[0039] 5A shows a first example of the simulated image of the tip attachment 445, which is a side view image showing a simulated grapple. This image shows that points corresponding to the central tips of the first claw portion 4451 and the second claw portion 4452 of the grapple are defined as a pair of first index points P11 and P12, respectively. As shown in FIG. 5A, the simulated image may additionally show that a pair of second index points P21 and P22 are further defined as points obtained by vertically projecting the pair of first index points P11 and P12 onto the ground, or may additionally show index images M1 and M2 indicating the respective positions of the pair of second index points P21 and P22.

[0040] FIG. 5B shows a second example of a simulated image of the tip attachment 445, which is a side-view image that simulates the bucket. FIG. 5C shows a third example of a simulated image of the tip attachment 445, which is a rear-view image that simulates the bucket. A front-view image may be output instead of the rear-view image. The simulated image of the third example shows that a point corresponding to the central tip of the bucket is defined as a first index point P1. Furthermore, as in the simulated images shown in FIGS. 5B and 5C, it may be shown that a second index point P2, which is a point obtained by projecting the first index point P1 vertically onto the ground, is defined, or an index image M indicating the position of the second index point P2 may be additionally displayed.

[0041] Fig. 6A shows a fourth example of the simulated image of the tip attachment 445, which is a three-dimensional image that simulates a grapple. Fig. 6B shows a fifth example of the simulated image of the tip attachment 445, which is a three-dimensional image that simulates a bucket. At least one of the attitude and display size of the three-dimensional image may be arbitrarily changed through an operation provided to the remote input interface 210 by an operator.

[0042] The simulated image shown in Fig. 6A shows that points corresponding to the central tips of the first claw portion 4451 and the second claw portion 4452 of the grapple, which is the tip attachment 445, are defined as a pair of first index points P11 and P12. The simulated image shown in Fig. 6B shows that points corresponding to the tips on both the left and right sides of the bucket, which is the tip attachment 445, are defined as a pair of first index points P11 and P12. In each of these simulated images, it may be additionally shown that points obtained by vertically projecting the pair of first index points P11 and P12 onto the ground are defined as a pair of second index points P21 and P22, or index images M1 and M2 indicating the positions of the pair of second index points P21 and P22 may be additionally displayed.

[0043] On the condition that an operator who has visually recognized the specification information setting image performs a confirmation operation on the remote input interface 210, the remote control device 200 confirms the specification information as accurate specification information as the basis for estimating a ground projection location, which is a location where a designated location on the tip attachment 445 is projected onto the ground. In detail, one or more first index points P1 corresponding to the specification information are automatically displayed on the specification information setting image, and the operator performs a confirmation operation on the remote input interface 210, thereby confirming the specification information as accurate specification information as the basis for estimating the ground projection location.

[0044] In the specification information setting image, specification information of the work machine 40 may be set by setting operations provided by the operator to the remote input interface 210. For example, an operator who visually recognizes the work machine 40 and the end attachment 445 reflected in the work environment image output to the remote image output device 221 may input information such as the manufacturer name and model number of the work machine 40 into the remote input interface 210, and specification information corresponding to the input information may be searched for within a database. Furthermore, in the specification information setting image, one or more first index points P1 corresponding to the specification information may be defined and confirmed by operations provided by the operator to the remote input interface 210, and further, the specification information may be confirmed as accurate specification information that serves as the basis for estimating a ground projection location, which is the location where a designated location on the end attachment 445 is projected onto the ground.

[0045] In the image for setting specification information, at least one of the direction in which the first index point P1 is projected onto the ground, the design of the index image M (shape, size, color or pattern, or any combination thereof), and the temporal change pattern of the design may be set by an operator through operations provided to the remote input interface 210.

[0046] With communication established between the remote control device 20 and the work machine 40 to be operated, the actual machine imaging device 412 of the work machine 40 acquires captured images of objects around the work machine 40, for example, the ground, earth, sand, materials, and structures present around the work machine 40, the actual machine ranging device 414 of the work machine 40 acquires three-dimensional images of the surrounding objects, and the actual machine wireless communication equipment 422 transmits three-dimensional image data representing the three-dimensional images to the remote control device 20 (STEP 412). The captured images or three-dimensional image data may be the captured images themselves, or may be image data representing a simulated environmental image generated based on the captured images.

[0047] For example, communication between the remote operation device 20 and the work machine 40 may be established when the remote control device 200 of the remote operation device 20 transmits a designation signal to the work machine 40 via the remote wireless communication device 224. If there are multiple operation target candidates, for example, if the multiple operation target candidates are multiple work machines 40, a designation signal may be transmitted to each of the multiple work machines 40. The remote control device 200 may be configured to determine whether or not a designation operation has been applied to the remote input interface 210 by the operator, and to transmit the designation signal when it is determined that the designation operation has been applied. The designation operation is, for example, an operation such as tapping applied to the remote input interface 210 by the operator to designate a work machine 40 to be remotely operated from among the multiple work machines 40.

[0048] The three-dimensional image is an image that includes information acquired by the real-space distance measuring device 414, and the information is information about the direction and distance from a reference position to a surrounding object or the real-space position of the object. The real-space position is defined by coordinate values ​​in a real-space coordinate system (e.g., latitude, longitude, and altitude) or coordinate values ​​in a real-space coordinate system based on the work machine 40. When a surrounding object is reflected in the captured image, the captured image includes pixel values ​​of each pixel of the three-dimensional image, and the pixel values ​​correspond to the real-space positions of the points on the surface of the surrounding object that correspond to each pixel.

[0049] The three-dimensional image data may be acquired and transmitted as a combination of separate data, such as an image captured through the actual imaging device 412 or a corresponding model image, and distance or real-space position data acquired through the actual ranging device 414.

[0050] The captured images are not limited to those acquired by the actual machine imaging device 412. The captured images may be acquired by any of an imaging device installed around the work machine 40, an imaging device mounted on an unmanned aerial vehicle, and an imaging device on equipment carried by a site worker. The distance or real space position, which is the pixel value of the three-dimensional image, may be acquired by a ranging device installed around the work machine 40 or a ranging device mounted on an unmanned aerial vehicle.

[0051] Instead of the combination of the actual machine imaging device 412 and the actual machine ranging device 414, captured images and three-dimensional images of the surrounding objects may be obtained by a stereo camera mounted on the work machine 40, for example, a pair of cameras arranged on the left and right.

[0052] 4, when the 3D image data is received through the remote wireless communication device 224 (C21), the remote control device 200 inputs a captured image corresponding to the 3D image data to the remote image output device 221, which is a display device (STEP 212). As a result, the remote image output device 221 outputs a captured image including the ground spreading out in front of the cab 43 and a tip attachment 445, such as a grapple, which is part of the working mechanism 44, as shown in FIG. 7, for example. The grapple is configured to pinch or grab a designated object with the first claw 4451 and the second claw 4452. Furthermore, the image shown in FIG. 8 as a captured image output by the remote image output device 221 includes not only the ground spreading out in front of the cab 43 but also the bucket, which is the tip attachment 445, which is part of the working mechanism 44.

[0053] The actual machine control device 400 of the work machine 40 transmits actual machine position / attitude data regarding the actual machine position and actual machine attitude acquired through the multiple actual machine sensors 416 to the image display system 10 via the actual machine wireless communication device 422 (STEP 414). The actual machine position / attitude data includes actual machine positioning data, and the actual machine positioning data is, for example, data regarding the real space position and real space attitude or real space orientation of the work machine 40 measured by an actual machine positioning device included in the multiple actual machine sensors 416. The actual machine position / attitude data further includes output signals from attitude angle sensors included in the multiple actual machine sensors 416. The attitude angle sensor is configured to output a signal corresponding to at least a portion of the rotation angle of the upper rotating body 42 relative to the lower running body 41, the elevation angle of the boom 441 relative to the upper rotating body 42, the rotation angle of the arm 443 relative to the boom 441, the rotation angle of the tip attachment 445 relative to the arm 443, and the relative position and / or attitude of the multiple members that make up the tip attachment 445 (for example, the opening angle between the first claw portion 4451 and the second claw portion 4452 of the grapple).

[0054] The process of transmitting the three-dimensional image data (STEP 412) and the process of transmitting the actual device position and attitude data (STEP 414) may be executed simultaneously.

[0055] When the actual machine position / attitude data is received via the remote wireless communication device 224 of the remote operation device 20 (C22), the remote control device 200 recognizes or identifies the real space position of the first index point P1 of the working mechanism 44 (STEP 214). Specifically, the remote control device 200 calculates the real space position of the first index point P1 in a forward kinematic manner based on the output signals of the attitude angle sensors included in the plurality of actual machine sensors 416, and the sizes of each component of the working mechanism 44 according to the specification information of the work machine 40 and the end attachment 445 set by the specification information setting process.

[0056] 5A and 6A, points corresponding to the central tip portions of the first claw portion 4451 and the second claw portion 4452 of the grapple which is the tip attachment 445 are defined as a pair of first index points P11 and P12. In the examples shown in Figures 5B and 5C, a point corresponding to the central tip portion of the bucket which is the tip attachment 445 is defined as the first index point P1. In the example shown in Figure 6B, a pair of points spaced apart from each other in the front-rear direction or left-right direction of the tip attachment 445 may be defined as the pair of first index points P11 and P12.

[0057] In the tip attachment 445 that does not include multiple members that move relative to each other, such as a bucket and a lifting magnet, the real spatial distance between the first index points P11 and P12 is constant. In the tip attachment 445 that includes multiple members that move relative to each other, such as a grapple and a nibbler, the real spatial distance between the first index points P11 and P12 defined on each of the multiple members varies in accordance with the relative movement.

[0058] If the three-dimensional image includes the first index point P1 of the working mechanism 44, the actual machine control device 400 may be configured to recognize the real-space position of the first index point P1 based on the three-dimensional image. Specifically, by analyzing the three-dimensional image, such as by grayscaling, edge extraction, or pattern matching, the average value of pixel values ​​of one or more pixels corresponding to the first index point P1 defined on the working mechanism 44 can be recognized as the real-space position of the first index point P1. Based on one of the real-space positions of each point of the working mechanism 44 recognized using the attitude angle sensor and the real-space positions that are pixel values ​​of the three-dimensional image, the other may be corrected.

[0059] The remote control device 200 recognizes or identifies the real-space position of the second index point P2 based on the real-space position of the first index point P1 and the real-space positions or three-dimensional shapes of each point constituting a point cloud on the surface of an object, such as the ground, included in the three-dimensional image (STEP 216). The second index point P2 is the point obtained by projecting the first index point P1 onto the surface of the object, such as the ground. The projection direction, in which the first index point P1 is projected onto the surface of the object, is the vertical direction in the examples shown in FIGS. 5A to 5C and 6A to 6B. In these examples, the real-space position of a point on the surface of the object that has the same horizontal position as the horizontal position of the first index point P1 or a horizontal position closest to that horizontal position, or the center of gravity of multiple points that have horizontal positions closest to the horizontal position of the first index point P1, is recognized as the second index point P2. The horizontal position is a position in the horizontal direction, and is specifically identified by a position in the x direction (longitude) and a position in the y direction (latitude).

[0060] As a result, the remote control device 200 recognizes or identifies, for example, the real space positions of the second index points P21 and P22 corresponding to the first index point P11 corresponding to the first claw portion 4451 of the grapple that is the tip attachment 445 and the first index point P12 corresponding to the second claw portion 4452 of the tip attachment 445. In addition, the remote control device 200 recognizes or identifies the real space position of the second index point P2 corresponding to the first index point P1 of the bucket that is the tip attachment 445.

[0061] If the work machine 40 or the upper rotating body 42 is inclined with respect to a vertical axis in real space, the direction in which the first index point P1 is projected onto the ground may also be determined to be a direction inclined with respect to the vertical axis. The inclination angle of the work machine 40 with respect to the vertical axis is measured by a machine body inclination angle sensor, for example a gyro sensor, included in the plurality of real machine sensors 416.

[0062] The remote control device 200 of the remote operation device 20 superimposes the index image M on the work environment image based on the real space position of the second index point P2 and at least one of the three-dimensional image and the pixel position (u, v) corresponding to the second index point P2 in the work environment image, and outputs a work support image, which is an image generated in this way, to the remote image output device 221 (STEP 218). That is, the remote control device 200 includes an image generation unit that generates the work support image.

[0063] Specifically, in the example shown in Fig. 7, the work support image is an image in which the index images M1 and M2 indicating the second index points P21 and P22, which are points obtained by projecting the first index points P11 and P12 onto an object surface such as the ground, are superimposed on the work environment image, which is the captured image. Similarly, in the example shown in Fig. 8, the work support image is an image in which the index image M indicating the second index point P2, which is a point obtained by projecting the first index point P1 onto an object surface such as the ground, is superimposed on the work environment image, which is the captured image.

[0064] Each of the index images M1, M2 or the index image M has a directional shape such as a triangular shape or an arrow shape that points toward the second index point P21, P22 or the second index point P2 along the vertical direction in real space. The first index point P1 and the second index point P2 may be displayed superimposed on the captured image, or the superimposed display of at least one of the first index point P1 and the second index point P2 may be omitted.

[0065] The remote control device 200 of the remote operation device 20 recognizes the operation mode of the remote operation mechanism 211, and transmits a remote operation command corresponding to the operation mode to the work machine 40 via the remote wireless communication device 224 (STEP 220).

[0066] When the actual machine control device 400 of the work machine 40 receives the operation command via the actual machine wireless communication device 422 (C44), it controls the operation of the work mechanism 44, etc. in accordance with the operation command (STEP 420). For example, if the tip attachment 445 is the grapple, the grapple clamps and holds a designated object in front of the work machine 40, for example, a natural object such as a stone or a tree, or an artificial object such as a block, and the upper rotating body 42 rotates, and then the tip attachment 445 releases the designated object and places it in a designated location. Also, if the tip attachment 445 is a bucket, the bucket scoops up and holds a designated object such as soil and sand in front of the work machine 40, the upper rotating body 42 rotates, and then the tip attachment 445 releases the designated object and places it in a designated location.

[0067] As explained above, the remote control device 200 that constitutes the work support system in the embodiment includes the change processing unit, and the change processing unit executes the specification information change processing when at least one of the work machine 40, the work site, and the end attachment 445 is the change target that has been changed. The specification information change processing is processing for changing the specification information of the work machine that forms the basis for estimating the ground projection location so that it matches the change target, and is processing for changing the specification information or processing for having the operator operating the work machine 40 change the specification information.

[0068] The specification information change process enables the ground projection point (e.g., second index point P2), which is the point where the specified point (e.g., first index point P1) on the tip attachment 445 is projected onto the ground, to be estimated based on specification information changed to suit the object of change, thereby enabling the operator to maintain high accuracy in recognizing the position of the ground projection point through the work support image regardless of changes to the object of change.

[0069] The work support system is not limited to being configured by the remote control device 200 of the remote operation device 20 according to the embodiment. The work support system may be configured by a computer having the function of communicating with both the remote operation device 20 and the work machine 40, or may be configured by the computer and the remote control device 200. Alternatively, the work support system may be configured by the remote control device 200 of the remote operation device 20 and the actual machine control device 400 of the work machine 40.

[0070] Various indices indicating the suitability of the real-space positions of the first index points P1, P11, and P12 can be set. For example, a first state in which the real-space position is appropriate in consideration of the task performed by the tip attachment 445 on the designated object OBJ, such as the task of grasping the designated object OBJ, a second state in which the real-space position is inappropriate, and a third state in which the real-space position is somewhat inappropriate may be represented by the color of the index images M, M1, and M2, such as black, white, or gray, while keeping the shapes of the index images M, M1, and M2 constant, or by differentiating the design of the index images M in other ways. Specifically, to represent the first to third states, the color, shape, or pattern of the index images M, or any combination thereof, or the temporal change in the design may be differentiated; the shape of the index images M may be differentiated without changing the color; or the presence or absence of blinking of the index images M and / or the length of the blinking cycle may be differentiated.

[0071] In the work support image, a figure representing the shape of the ground and / or the surface shape of the target object, for example, a meshed figure, may be superimposed on the work environment image in addition to the index image M. The width, color, and spacing between the meshes of the lines constituting the mesh may be set on the specification information setting image by operations applied to the remote input interface 210.

[0072] As described above, a system and method are provided that enable an operator to maintain high accuracy in recognizing the position of a ground projection point regardless of changes in at least one of the work machine, the work site, and the end attachment.

[0073] The provided work support system includes an image generation unit and a change processing unit. The image processing unit estimates a ground projection location, where a designated location, which is a location previously designated on the end attachment of the work machine, is projected onto the ground, based on specification information of the work machine, and generates a work support image in which an index image indicating the ground projection location is superimposed on a work environment image acquired by an imaging device. The work support image is an image of the surroundings of the work machine and includes the end attachment and the ground. The change processing unit executes specification information change processing when at least one of the work machine, the work site where the work machine performs work, and the end attachment is subject to change. The specification information change processing is processing for adapting the specification information, which is the basis for estimating the ground projection location, to the subject of change, and may be processing for changing the specification information, or processing for having the operator operating the work machine change the specification information.

[0074] The ground may be a curved and / or uneven surface that reproduces with high precision the shape of the ground surface around the work machine at the work site and the objects present on the ground surface, or it may be a flat or horizontal surface that roughly follows the ground surface.

[0075] The specification information change process enables new specification information that has been changed to suit the object of change to be used to estimate the ground projection point, which is the point where the specified point on the tip attachment is projected onto the ground, thereby enabling the operator to maintain high accuracy in recognizing the position of the ground projection point through the index image regardless of changes to the object of change.

[0076] Preferably, the work assistance system further comprises an output interface that outputs the specification information changed to suit the change target when at least one of the work machine, the work site, and the end attachment is the change target, and an input interface to which the operator can input a confirmation, and the change processing unit is configured to confirm the specification information as the basis for estimating the ground projection location upon receiving the confirmation input via the input interface. This makes it possible to confirm the specification information while respecting the operator's wishes.

[0077] Preferably, the work assistance system further comprises an output interface that outputs a specification setting image for setting the specification information when at least one of the work machine, the work site, and the end attachment is subject to the change, and an input interface that allows an operator to perform setting operations for setting the specification information, and the change processing unit is configured to set the specification information that forms the basis for estimating the ground projection location in accordance with the setting operations. This makes it possible to set the specification information while respecting the operator's wishes.

[0078] Also provided is a work support method, which includes the steps of: acquiring, by an imaging device, a work environment image of the surroundings of a work machine, the work environment image including the end attachment of the work machine and the ground; estimating, based on specification information of the work machine, a ground projection point where a designated point that is a pre-designated point on the end attachment is projected onto the ground; and generating a work support image in which an index image indicating the ground projection point is superimposed on the work environment image; and, if at least one of the work machine, the work site where the work machine performs work, and the end attachment is subject to change, executing a specification information change process to adapt the specification information that forms the basis for estimating the ground projection point to the subject to change.

Claims

1. A work support system comprising: an image generation unit that estimates a ground projection point, which is a point where a designated point, which is a point previously designated on the tip attachment of the work machine, is projected onto the ground based on specification information of the work machine, and generates a work support image in which an index image indicating the ground projection point is superimposed on a work environment image, the work environment image being an image of the surroundings of the work machine acquired by an imaging device and including the tip attachment and the ground; and a change processing unit that, when at least one of the work machine, a work site where the work machine performs work, and the tip attachment is a change target in which a change has been made, executes a specification information change process, which is a process for adapting the specification information that is the basis for estimating the ground projection point to the change target.

2. A work support system as described in claim 1, comprising: an output interface that outputs the specification information changed to match the change target when at least one of the work machine, the work site and the end attachment is the change target; and an input interface to which an operator can provide a confirmation input operation, and the change processing unit is configured to determine the specification information as the basis for estimating the ground projection location on the condition that the confirmation input operation has been provided to the input interface.

3. A work support system as described in claim 1, further comprising: an output interface that outputs a specification setting image for setting the specification information of the work machine when at least one of the work machine, the work site and the end attachment is subject to the change; and an input interface that allows an operator to provide setting operations for setting the specification information, wherein the change processing unit is configured to set the specification information that forms the basis for estimating the ground projection location in response to the setting operations.

4. A work support method comprising the steps of: acquiring, by an imaging device, a work environment image around a work machine, the work environment image including the end attachment of the work machine and the ground; estimating a ground projection point, where a designated point, which is a pre-designated point on the end attachment, is projected onto the ground, based on specification information of the work machine, and generating a work support image in which an index image indicating the ground projection point is superimposed on the work environment image; and, when at least one of the work machine, the work site where the work machine performs work, and the end attachment is a change target that has been changed, executing a specification information change process to adapt the specification information that is the basis for estimating the ground projection point to the change target.

Images