Work assistance system and work assistance method

Abstract

The present invention is a work assistance system that enables an operator to recognize a positional relationship between a work mechanism and a target object with high accuracy. The work assistance system includes an image generation part and a change processing part. The image generation part generates a work assistance image in which an index image (M1, M2) indicating a ground projection spot (P21, P22) at which a designated part (P11, P12) of a distal attachment (445) is projected onto the ground surface is superimposed on the work environment image. When at least one of a work machine, a work site, and the distal attachment (445) is a change target, the change processing part executes specification information change processing for changing specification information of the work machine as a basis for estimation of the ground projection spot to adapt the specification information to the change target.

Description

Technical Field

[0001] The present invention relates to a technique for assistance of a remote operation by an operator of a work machine such as a hydraulic excavator.Background Art

[0002] Patent Literature 1 discloses a technique for generating a display image obtained by superimposing a captured image and a blade edge position image to improve work efficiency. The captured image contains a bucket and a work target image-captured by a camera provided in the work machine. The blade edge position image is an image obtained by vertically projecting the blade edge of the bucket onto a work target.

[0003] The technique, in order to generate the image, requires determining the position of a ground projection spot at which a designated part in the distal attachment is projected on the ground surface, involving the use of specification information of a work machine including the distal attachment for the determination. However, the actual position of the ground projection spot is varied with the change of at least one of the work machine, which is the operation target by an operator, a work site, and the distal attachment, which may deteriorate the accuracy with which an operator recognizes the position of the ground projection spot through the image.Citation List Patent Literature

[0004] Patent Literature 1: Japanese Patent No. 6777375Summary of Invention

[0005] It is an object of the present invention to provide a system and a method that allow accuracy with which an operator recognizes the position of a ground projection spot to be kept regardless of a change of at least one of a work machine, a work site, and a distal attachment.

[0006] Provided is a work assistance system, including an image generation part and a change processing part. The image processing part estimates a ground projection spot, at which a designated part which is a part designated in advance in a distal attachment of a work machine is projected onto the ground, based on specification information of the work machine, and generates a work assistance image in which an index image indicating the ground projection spot is superimposed on a work environment image acquired by an imaging device. The work assistance image is an image of a periphery of the work machine, containing the distal attachment and the ground. The change processing part executes specification information change processing when at least one of the work machine, a work site where the work machine performs work, and the distal attachment is a change target that has been changed. The specification information change processing, which is processing for adapting the specification information as a basis for estimation of the ground projection spot to the change target, may be either processing for changing the specification information or processing for making an operator who operates the work machine change the specification information.

[0007] Also provided is a work assistance method, including: a step of acquiring, by an imaging device, a work environment image that is an image of a periphery of the work machine, the work environment image containing a distal attachment of the work machine and a ground; a step of estimating, based on specification information of the work machine, a ground projection spot, which is a spot at which a designated part which is a part designated in advance in the distal attachment is projected onto the ground, and generating a work assistance image in which an index image indicating the ground projection spot is superimposed on the work environment image; and a step of executing specification information change processing when at least one of the work machine, a work site where the work machine performs work, and the distal attachment is a change target that has been changed, the specification information change processing being processing for adapting the specification information as a basis for estimation of the ground projection spot to the change target.Brief Description of Drawings

[0008] FIG. 1 is a diagram showing a work assistance system according to an embodiment of the present invention. FIG. 2 is a plan view showing a remote operation device constituting the work assistance system. FIG. 3 is a side view of the work machine according to the embodiment. FIG. 4 is a flowchart showing a processing to be performed by the work assistance system. FIG. 5A is a side view showing a first example of a simulation image included in a specification information setting image provided in the embodiment. FIG. 5B is a side view showing a second example of the simulation image included in the specification information setting image provided in the embodiment. FIG. 5C is a front view showing a third example of the simulation image included in the specification information setting image provided in the embodiment. FIG. 6A is a perspective view showing a fourth example of the simulation image included in the specification information setting image provided in the embodiment. FIG. 6B is a perspective view showing a fifth example of the simulation image included in the specification information setting image provided in the embodiment. FIG. 7 is a diagram showing a first example of an index image superimposed on a work environment image in the embodiment. FIG. 8 is a diagram showing a second example of an index image superimposed on the work environment image. Detailed Description

[0009] There will be described an embodiment of the present invention with reference to the drawings. FIG. 1 shows a work machine 40 and a remote operation apparatus 20 for remotely operating the work machine 40.

[0010] FIG. 3 shows a crawler type hydraulic excavator that is an example of the work machine 40. The illustrated work machine 40 includes a lower traveling body 41 including a pair of left and right crawlers, an upper turning body 42 mounted on the lower traveling body 41 via a turning drive mechanism so as to be capable of turning, and a work mechanism 44. The upper turning body 42 includes a cab 43 (operation chamber), which forms a left front part of the upper turning body 42. The work mechanism 44 is connected to the right front part of the upper turning body 42 so as to be capable of performing a work motion. The cab 43 may be configured to be driven up and down with respect to the upper turning body 42 by a not-graphically-shown lifting drive mechanism including a link mechanism and an actuator.

[0011] The work mechanism 44 includes a boom 441, an arm 443, a distal attachment 445, and a plurality of hydraulic cylinders. The boom 441 is attached to the upper turning body 42 so as to be capable of derricking. The arm 443 is connected to a distal end of the boom 441 so as to be rotationally movable. The distal attachment 445 is connected to a distal end of the arm 443 so as to be rotationally movable. Examples of the distal attachment 445 include a bucket, a grapple, a reach hook, a breaker, and a lifting magnet. Each of the hydraulic cylinders is an actuator that is attached to the work mechanism 44 to be capable of performing expansion and contraction motions so as to move the work mechanism 44.

[0012] The plurality of hydraulic cylinders include a boom cylinder 442, an arm cylinder 444, and a distal attachment cylinder 446. The boom cylinder 442 is interposed between the boom 441 and the upper turning body 42, configured to be expanded and contracted by supply of hydraulic fluid to rotationally move the boom 441 in a derricking direction. The arm cylinder 444 is interposed between the arm 443 and the boom 441, configured to be expanded and contracted by supply of hydraulic fluid to rotationally move the arm 443 about a horizontal axis with respect to the boom 441. The distal attachment cylinder 446 is interposed between the distal attachment 445 and the arm 443, configured to be expanded and contracted by supply of hydraulic fluid to rotationally move the distal attachment 445 around a horizontal axis with respect to the arm 443.

[0013] As shown in FIG. 1, the remote operation apparatus 20 includes a remote control device 200, a remote input interface 210, and a remote output interface 220. In the present embodiment, the remote control device 200, the remote input interface 210, and the remote output interface 220 constitute a work assistance system. The remote control device 200 is composed of an arithmetic processing device, which includes, for example, a single-core processor or a multi-core processor or a processor core constituting the same. The remote control device 200 reads necessary data and software from a storage device such as a memory, and executes arithmetic processing according to the software for the data.

[0014] The remote input interface 210 includes a remote operation mechanism 211. The remote output interface 220 includes a remote image output device 221 that is a display device, a remote sound output device 222, and a remote wireless communication device 224.

[0015] The remote operation mechanism 211 includes a plurality of remote operation units, which include a remote traveling operation unit, a remote turning operation unit, a remote boom operation unit, a remote arm operation unit, and a remote distal-attachment operation unit. Each of the remote operation units includes a remote operation lever allowing a turning operation to be applied thereto. The remote operation lever of the remote traveling operation device is a traveling lever, to which a traveling operation for moving the lower traveling body 41 of the work machine 40 is applied. The traveling lever may also serve as a traveling pedal. For example, a pedal for receiving a traveling operation may be fixed to a base part or a lower end part of the traveling lever. The remote operation lever of the turning operation device is a turning lever, to which a turning operation for moving the turning motor is applied. The turning motor is a hydraulic motor included in the turning drive mechanism. The remote operation lever of the boom operation device is a boom lever, to which a boom operation for moving the boom cylinder 442 is applied. The remote operation lever of the arm operating device is an arm lever, to which an arm operation for moving the arm cylinder 444 is applied. The remote operation lever of the distal-attachment operation device is a distal attachment lever, to which a distal attachment operation for moving the distal attachment cylinder 446 is applied.

[0016] In the example shown in FIG. 2, the remote operation lever included in the remote operation mechanism 211 is disposed around a seat St, which is a seating part for an operator to sit thereon. The seat St shown in FIG. 2 is a high-back chair including a headrest 2200 and a pair of armrests 2221, 2222, or the like. The seat St, however, may be a chair with no headrest or a chair with no backrest. The seating part is allowed to be designed to have any form allowing an operator to sit thereon.

[0017] The remote traveling operation unit includes, specifically, as the remote operation lever, a pair of left and right traveling levers 2110 corresponding to the pair of left and right crawlers, respectively, which levers are arranged side by side on the front side of the seat St. The remote operation mechanism 211 may include a remote operation lever also having functions as a plurality of remote operation levers. In the example shown in FIG. 2, the remote operation mechanism 211 includes a left operation lever 2111 provided frontward of the left frame of the seat St and a right operation lever 2112 provided frontward of the right frame of the seat St, wherein the left operation lever 2111 is configured to function as the arm lever when operated in the front-rear direction, and configured to function as the turning lever when operated in the left-right direction. Similarly, the right operation lever 2112 is configured to function as the boom lever when operated in the front-rear direction, and configured to function as the distal attachment lever when operated in the left-right direction. Alternatively, the remote operation mechanism 211 may be configured to allow respective lever patterns of the remote operation levers to be arbitrarily changed by an operation instruction from an operator.

[0018] The remote image output device 221 may be composed of either a single image output unit or a plurality of image output units. The remote image output device 221 illustrated in FIG. 2 includes a plurality of image output units, namely, a center image output unit 2210, a left image output unit 2211, and a right image output unit 2212. The center image output unit 2210 is disposed frontward of the seat St, the left image output unit 2211 is disposed diagonally forward and leftward of the seat St, and the right image output unit 2212 is disposed diagonally forward and rightward of the seat St. Each of the image output units 2210, 2211, 2212 has a substantially rectangular screen, namely, an image display region. Respective shapes and sizes of the screens of the image output units 2210, 2211, and 2212 may be either the same or different. The remote image output device 221, alternatively, may be either composed of a single image output unit which is curved or curvable, or constituted by two or four or more image output units arranged to surround the front of the seat St.

[0019] In the example shown in FIG. 2, the right edge of the left image output unit 2211 is adjacent to the left edge of the center image output unit 2210 such that respective screens of the center image output unit 2210 and the screen of the left image output unit 2211 are inclined to each other at a predetermined angle θ1. Besides, the left edge of the right image output unit 2212 is adjacent to the right edge of the center image output unit 2210 such that respective screens of the center image output unit 2210 and the right image output unit 2212 are inclined to each other at a predetermined inclination angle θ2. Each of the predetermined inclination angles θ1 and θ2 is set, for example, 120° ≤ θ1 ≤ 150°. The predetermined inclination angles θ1 and θ2 may be either the same as or different from each other.

[0020] The screen of each of the image output units 2210, 2211, and 2212 may be either parallel to the vertical direction or inclined to the vertical direction. At least one of the image output units 2210, 2211, 2212 may be divided into a plurality of image output units. For example, the center image output unit may be constituted by vertically adjacent image output units with respective substantially rectangular screens.

[0021] The remote sound output device 222 includes one or more speakers. The remote sound output device 222 illustrated in FIG. 2 is composed of a plurality of speakers which are disposed behind the seat St, in the rear part of the left armrest, and in the rear part of the right armrest, respectively. Respective specifications of the speakers may be either the same as or different from each other.

[0022] As shown in FIG. 1, the work machine 40 includes a real-machine control device 400, a real-machine input interface 410, a real-machine output interface 420, and a real-machine wireless communication device 422. The real-machine control device 400 includes a plurality of components, each of which is composed of an arithmetic processing device. The arithmetic processing device, which is composed of, for example, a single-core processor, a multi-core processor, or a processor core constituting the same, reads necessary data and software from a storage device such as a memory, and executes arithmetic processing according to the software for the data.

[0023] The real-machine input interface 410 includes a real-machine operation mechanism 411, a real-machine imaging device 412, a real-machine ranging device 414, and a plurality of real-machine sensors 416.

[0024] The real-machine operation mechanism 411 includes a plurality of real-machine operation levers, which are disposed around the seat disposed inside the cab 43 in the same manner as the remote operation levers of the remote operation mechanism 211. In the cab 43 is provided an operation mechanism body such as a robot, which receives a signal corresponding to the operation applied to the remote operation lever and actuates the real-machine operation lever based on the received signal.

[0025] The real-machine imaging device 412 is installed, for example, inside the cab 43 to capture an image containing at least a part of the work mechanism 44 through a front window and left and right side windows. A part or all of the front window and the left and right side windows may be omitted.

[0026] The real-machine ranging device 414 measures a distance from a reference position set in the work machine 40 to a surrounding object present around the work machine 40. The surrounding object includes a designated object OBJ which will be below described. The real-machine ranging device 414 is composed of, for example, a LiDAR or a TOF sensor.

[0027] The plurality of real-machine sensors 416 are configured to output respective signals corresponding to the values of parameters related to a state of the work machine 40, such as a derricking angle of the boom 441 with respect to the upper turning body 42, a rotation angle of the arm 443 with respect to the boom 441, a rotation angle of the distal attachment 445 with respect to the arm 443, a hydraulic pressure, a fuel remaining amount, to input the signals to the real-machine control device 400. The plurality of real-machine sensors 416 include a real-machine positioning device that measures the position, for example, latitude and longitude, of the work machine 40, the real-machine positioning device composed of, for example, a GPS or GNSS and a gyro sensor as necessary.

[0028] The processing to be performed by the work assistance system will be described with reference to the flowchart of FIG. 4. In the flowchart, the part indicated by a combination of "C" and a number indicates processing of judging the presence or absence of specified data and determining whether to execute the subsequent processing in the branch direction on condition of transmission and / or reception of the specified data.

[0029] In FIG. 4, the remote operation apparatus 20 judges the presence or absence of a change of at least one of the work machine 40, a work site, and the distal attachment 445 (STEP202).

[0030] The presence or absence of the change of the work machine 40 may be judged based on, for example, the presence or absence of a work-machine designation change operation. The work-machine designation operation is an operation to be applied by an operator through the remote input interface 210 for changing the work machine 40 that is an operation target. The presence or absence of the change of the work machine 40, alternatively, may be judged based on whether or not the communication between the work machine 40 before the change and the remote operation apparatus 20 has been released and / or whether or not the communication between the work machine 40 after the change and the remote operation apparatus 20 has been established. The presence or absence of the change of the work machine 40, alternatively, may be judged based on the presence or absence of a change in a real-machine identifier. The real-machine identifier, being an identifier for identifying the work machine 40 that is the operation target or the specification information thereon, is transmitted from the work machine 40 and received by the remote operation apparatus 20.

[0031] The remote operation apparatus 20 may judge the presence or absence of the change of the distal attachment 445 based on the presence or absence of an attachment identifier. The attachment identifier, being an identifier for identifying the distal attachment 445 or the specification information thereon, is transmitted from an information processing terminal such as a smartphone or a personal computer owned by the work machine 40 that is the operation target and / or a worker and received by the remote operation apparatus 20. The attachment identifier may be read by a non-contact reader from an RFID tag or barcode attached to the distal attachment 445. The non-contact reader may be either included in the plurality of real-machine sensors 416 or installed on the information processing terminal.

[0032] Alternatively, the remote operation apparatus 20 may judge the presence or absence of the change of the distal attachment 445 based on the presence or absence of the change of distal attachment information. The distal attachment information, being information that enables the distal attachment 445 to be identified, such as a combination of a manufacturer name and a model number or an attachment identifier, is input to the information processing terminal by an information worker and received by the remote operation apparatus 20. To authenticate the information processing terminal, specifically, to confirm the suitability of the information processing terminal as the information provider of the work machine 40 that is the operation target, there may be used either the result of judgement on whether or not the position of the information processing terminal is present in a designated area based on the position of the work machine 40, or the result of judgement on the presence or absence of the establishment of the communication between the information processing terminal and the work machine 40.

[0033] The remote operation apparatus 20 may be configured to receive a work environment image from the work machine 40 that is the operation target or an information processing terminal owned by a worker and analyze the work environment image to thereby recognize a shape feature of the distal attachment 445 and judge the presence or absence of the change of the distal attachment 445 based on the presence or absence of a change in the shape feature. The work environment image can be captured by the real-machine imaging device 412 or a camera mounted on the information processing terminal.

[0034] The remote operation apparatus 20 may be configured to receive real-machine position information, which is information about the position of the work machine 40, and judge the presence or absence of the change of a work site (work area) including the position based on the presence or absence of the work site. The position of the work machine 40, which can be measured by a real-machine positioning device included in the plurality of real-machine sensors 416, is specified by the latitude and longitude of the work machine 40 and further the altitude as necessary. For the specification may be used a work site distribution map representing a distribution in the work site. The remote operation apparatus 20 may be configured to receive the work environment image captured by the real-machine imaging device 412 and analyze the work environment image to thereby discriminate an index or an object that is unique to the work site to judge the presence or absence of a change of the work site based on the presence or absence of the change in the index or the like.

[0035] When judging that neither of the work machine 40, the work site, and the distal attachment 445 has been changed (NO in STEP202), that is, when judging that neither of the work machine 40, the work site, and the distal attachment 445 is a change target, the remote operation apparatus 20 executes reception processing (C21) of receiving the three-dimensional image data through the remote wireless communication device 224 as specifically described later, and executes the subsequent processing.

[0036] When judging that at least one of the work machine 40, the work site, and the distal attachment 445 is the changed target that has been changed (YES in STEP202), the remote control device 200 executes specification information change processing. This processing is processing for adapting specification information as the basis for estimation of a ground projection spot, which is a spot at which a designated part designated in advance in the distal attachment 445 is projected onto the ground surface, to the change target that has been changed among the work machine 40, the work site, and the distal attachment 445. The remote control device 200, thus, includes a change processing part that performs the specification information change processing.

[0037] Specifically, the remote control device 200 makes the remote image output device 221 of the remote output interface 220 display a specification information setting image for allowing an operator to confirm the specification information of the work machine 40. The specification information includes specification information of the components of the distal attachment 445 and the like. The remote control device 200 reads the specification information and the specification information setting image that correspond to at least one of the real-machine identifier and the attachment identifier from a database, or searches them in the database. The database is composed of, for example, a storage device included in the remote control device 200 and / or an external database server.

[0038] The specification information setting image includes a simulation image that simulates the distal attachment 445 adapted to the specification information. The image indicates that one or more first index points P1 are defined at positions of one or more designated parts of the distal attachment 445. Furthermore, the simulation image may either additionally indicate that there is defined a second index point P2 at which the first index point P1 is projected on the ground in a designated direction designated in advance, for example, a vertical direction or a displacement direction of the distal attachment 445, or additionally indicate an index image M indicating the position of the second index point P2.

[0039] Shown in FIG. 5A is a first example of the simulation image of the distal attachment 445, being a side view image that simulates a grapple. This image indicates that the points corresponding to respective center tips of the first claw part 4451 and the second claw part 4452 of the grapple are defined as a pair of first index points P11 and P12, respectively. As shown in FIG. 5A, the simulation image may either additionally indicate that a pair of second index points P21 and P22 that are points at which the pair of first index points P11 and P12 are projected on the ground in the vertical direction are further defined or additionally indicate index images M1 and M2 indicating respective positions of the pair of second index points P21 and P22.

[0040] Shown in FIG. 5B is a second example of the simulation image of the distal attachment 445, being a side view image that simulatively shows the bucket. Shown in FIG. 5C is a third example of the simulation image of the distal attachment 445, being a back view image that simulatively shows the bucket. In place of the rear view image may be output a front view image. The simulation image according to the third example indicates that the point corresponding to the center tip of the bucket is defined as the first index point P1. Furthermore, as in the simulation image shown in each of FIG. 5B and FIG. 5C. it may be indicated that the second index point P2, which is a point at which the first index point P1 is projected onto the ground in the vertical direction or an index image M indicating the position of the second index point P2 may be additionally displayed.

[0041] Shown in FIG. 6A is a fourth example of the simulation image of the distal attachment 445, being a three-dimensional image that simulatively shows a grapple. Shown in FIG. 6B is a fifth example of the simulation image of the distal attachment 445, being a three-dimensional image that simulatively shows a bucket. At least one of the posture and the display size of the three-dimensional image may be arbitrarily changed through an operation applied to the remote input interface 210 by an operator.

[0042] The simulation image shown in FIG. 6A indicates that the points corresponding to respective center tips of the first claw part 4451 and the second claw part 4452 of the grapple, which is the distal attachment 445, are defined as a pair of first index points P11 and P12, respectively. The simulation image shown in FIG. 6B indicates that the points corresponding to respective tips on both right and left sides of the bucket, which is the distal attachment 445, are defined as a pair of first index points P11 and P12, respectively. In each of these simulation images, it may be additionally indicated that the points at which the pair of first index points P11 and P12 are projected on the ground in the vertical direction are defined as a pair of second index points P21 and P22, respectively, or index images M1 and M2 indicating the respective positions of the pair of second index points P21 and P22 may be additionally displayed.

[0043] On condition that a confirmation operation is applied to the remote input interface 210 by an operator who has visually recognized the specification information setting image, the remote control device 200 fixes the specification information as proper specification information as the basis for estimation of a ground projection spot which is a spot at which the designated part in the distal attachment 445 is projected on the ground. Specifically, one or more first index points P1 corresponding to the specification information are automatically displayed in the specification information setting image, and the specification information is determined as proper specification information as the basis for estimation of the ground projection spot by the application of a confirmation operation to the remote input interface 210 by an operator.

[0044] In the specification information setting image, specification information of the work machine 40 may be set by a setting operation applied to the remote input interface 210 by an operator. For example, it is also allowable that an operator who has visually recognized the work machine 40 and the distal attachment 445 contained in the work environment image that is output to the remote image output device 221 inputs information on a manufacturer name and a model number of the work machine 40 or the like to the remote input interface 210, causing specification information corresponding to the information that has been input to be searched in the database. Furthermore, in the specification information setting image, it is also allowable that, by an operation applied to the remote input interface 210 by an operator, one or a plurality of first index points P1 corresponding to the specification information are defined and determined and further the specification information is determined as proper specification information as the basis for estimation of a ground projection spot which is a spot at which the designated part in the distal attachment 445 is projected on the ground.

[0045] In the specification information setting image, at least one of the direction in which the first index point P1 is projected on the ground, the design (shape, size, color, or pattern or any combination thereof) of the index image M, and the mode of a temporal change in the design may be set by an operation applied to the remote input interface 210 by an operator.

[0046] In a state where the communication between the remote operation apparatus 20 and the work machine 40 that is the operation target to be operated by the remote operation apparatus 20 is established, the real-machine imaging device 412 of the work machine 40 acquires a captured image of an object around the work machine 40, for example, a ground surface, sediment, material, or a building present around the work machine 40; the real-machine ranging device 414 of the work machine 40 acquires a three-dimensional image of the surrounding object; and the real-machine wireless communication device 422 transmits three-dimensional image data that represents the three-dimensional image to the remote operation apparatus 20 (STEP 412). The captured image or the three-dimensional image data may be either a captured image itself or image data that represents a simulative environmental image generated based on the captured image.

[0047] For example, the communication between the remote operation apparatus 20 and the work machine 40 may be established along with the transmission of a designation signal from the remote control device 200 of the remote operation apparatus 20 to the work machine 40 through the remote wireless communication device 224. In the case of existence of a plurality of operation target candidates, for example, in the case where the plurality of operation target candidates are a plurality of work machines 40, the designation signal may be transmitted to each of the plurality of work machines 40. The remote control device 200 may be configured to judge the presence or absence of a designation operation applied to the remote input interface 210 by an operator and configured to transmit the designation signal when judging that the designation operation is present. The designation operation is, for example, an operation such as a tap applied to the remote input interface 210 by an operator in order to designate a work machine 40 to be the target of remote operation from among the plurality of work machines 40.

[0048] The three-dimensional image is an image containing information acquired by the real-machine ranging device 414, and the information is about a direction and a distance from a reference position to a surrounding object or a real-space position of the object. The real-space position is defined by coordinate values in a real-space coordinate system (for example, latitude, longitude, and altitude) or coordinate values in a real-machine coordinate system using the work machine 40 as a reference. When containing surrounding object, the captured image includes respective pixel values of the pixels of the three-dimensional image, and the pixel values correspond to respective real-space positions of the points corresponding to the pixels, respectively, on a surface of the surrounding object.

[0049] The three-dimensional image data may be acquired and transmitted as a combination of separated data, which includes data of a captured image or a model image corresponding to the captured image acquired through a real-machine imaging device 412 and data of a distance or a real-space position acquired through the real-machine ranging device 414.

[0050] The captured image is not limited to one acquired by the real-machine imaging device 412. The captured image may be acquired by either of an imaging device disposed around the work machine 40, an imaging device installed on an unmanned aircraft, and an imaging device of equipment to be carried by a field worker. The distance or the real-space position, which is the pixel value of the three-dimensional image, may be acquired by a ranging device disposed around the work machine 40 or a ranging device installed on the unmanned aircraft.

[0051] The captured image and a three-dimensional image of the surrounding object may be acquired by, in place of the combination of the real-machine imaging device 412 and the real-machine ranging device 414, a stereo camera mounted on the work machine 40, for example, a pair of left and right cameras.

[0052] In FIG. 4, when receiving the three-dimensional image data through the remote wireless communication device 224 (C21), the remote control device 200 inputs a captured image corresponding to the three-dimensional image data to the remote image output device 221, which is a display device (STEP212). This causes, for example, as shown in FIG. 7, the remote image output device 221 to output a captured image containing a ground surface extending frontward of the cab 43 and the distal attachment 445, for example, a grapple, which is a part of the work mechanism 44. The grapple is configured to sandwich or grasp a designated object between the first claw part 4451 and the second claw part 4452. Besides, the image illustrated in FIG. 8 as a captured image to be output by the remote image output device 221 contains not only the ground surface extending frontward of the cab 43 but also a bucket which is the distal attachment 445, which is a part of the work mechanism 44.

[0053] The real-machine control device 400 of the work machine 40 transmits the real-machine position-and-posture data about the real-machine position and the real-machine posture acquired through the plurality of real-machine sensors 416 to the image display system 10 through the real-machine wireless communication device 422 (STEP 414). The real-machine position-and-posture data includes real-machine positioning data, which is, for example, data about a real-space position and a real space posture or a real space azimuth of the work machine 40 measured by a real-machine positioning device included in the plurality of real-machine sensors 416. The real-machine position-and-posture data further includes an output signal of a posture angle sensor included in the plurality of real-machine sensors 416. The posture angle sensor is configured to output a signal corresponding to at least a part of the followings: the turning angle of the upper turning body 42 with respect to the lower traveling body 41; the derricking angle of the boom 441 with respect to the upper turning body 42; the rotation angle of the arm 443 with respect to the boom 441; the rotation angle of the distal attachment 445 with respect to the arm 443; and the relative position and / or posture of a plurality of members constituting the distal attachment 445 (for example, the opening angle between the first claw part 4451 and the second claw part 4452 of the grapple).

[0054] The transmission processing of the three-dimensional image data (STEP 412) and the transmission processing of the real-machine position / posture data (STEP 414) may be executed at the same time.

[0055] When receiving the real-machine position and orientation data through the remote wireless communication device 224 of the remote operation apparatus 20 (C22), the remote control device 200 recognizes or determines the real-space position of the first index point P1 of the work mechanism 44 (STEP214). Specifically, the remote control device 200 forward-kinematically calculates the real-space position of the first index point P1 based on the output signal of the posture angle sensor included in the plurality of real-machine sensors 416 and respective sizes of the components of the work mechanism 44 according to the specification information on the work machine 40 and the distal attachment 445 set by the specification information setting processing.

[0056] In the example shown in FIG. 5A and FIG. 6A, the points corresponding to respective center tips of the first claw part 4451 and the second claw part 4452 of the grapple, which is the distal attachment 445, are defined as a pair of first index points P11 and P12, respectively. Besides, in the example shown in FIG. 5B and FIG. 5C, the point corresponding to the center tip of the bucket, which is the distal attachment 445, is defined as the first index point P1. In the example shown in FIG. 6B, a pair of points spaced in the front-rear direction or the left-right direction of the distal attachment 445 may be defined as a pair of first index points P11 and P12, respectively.

[0057] In the distal attachment 445 with no plurality of members that make relative motions to each other, such as a bucket or a lifting magnet, the real-space distance between the first index points P11 and P12 is constant. In the distal attachment 445 that includes the plurality of members that make relative motions to each other, such as a grapple or a nibra, the real-space distance between first index points P11 and P12 that are defined for the plurality of members is varied with the relative motion.

[0058] When the three-dimensional image includes the first index point P1 of the work mechanism 44, the real-machine control device 400 may be configured to recognize the real-space position of the first index point P1 based on the three-dimensional image. Specifically, analysis processing of the three-dimensional image, for example, gray scale processing, edge extraction processing, pattern matching processing, allows the average value of the pixel values of one or a plurality of pixels corresponding to the first index point P1 defined in the work mechanism 44 to be recognized as the real-space position of the first index point P1. It is also allowable to correct one of the real-space position of each point of the work mechanism 44 recognized by use of the posture angle sensor and the real-space position that is the pixel value of the three-dimensional image, based on the other.

[0059] The remote control device 200 recognizes or determines the real-space position of the second index point P2 based on the real-space position of the first index point P1 and the real-space position of each point constituting the point group of the object surface such as the ground contained in the three-dimensional image or the three-dimensional shape (STEP216). The second index point P2 is the point at which the first index point P1 is projected onto an object surface such as the ground surface. In respective examples shown in FIGS. 5A to 5C and 6A to 6B, the projection direction, which is the direction in which the first index point P1 is projected on the object surface, is the vertical direction. In these examples, recognized as the second index point P2 among the points on the object surface is the real-space position of the point having the same horizontal position as the horizontal position of the first index point P1 or the horizontal position closest to the horizontal position, or the center of gravity of a plurality of points having the horizontal position closest to the horizontal position of the first index point P1. The horizontal position is a position in the horizontal direction, specifically, defined by a position in the x direction (longitude) and a position in the y direction (latitude).

[0060] The remote control device 200, thus, recognizes or determines, for example, respective real-space positions of the second index points P21, P22 corresponding to the first index point P11 corresponding to the first claw part 4451 and the first index point P12 corresponding to the second claw part 4452 of the grapple, which is the distal attachment 445. respectively. Besides, the remote control device 200 recognizes or determines the real-space position of the second index point P2 corresponding to the first index point P1 of the bucket that is the distal attachment 445.

[0061] When the work machine 40 or the upper turning body 42 is inclined to the vertical axis in the real space, the direction in which the first index point P1 is projected onto the ground may also be set to a direction inclined to the vertical axis. The inclination angle of the work machine 40 to the vertical axis is measured by a machine-body inclination angle sensor included in the plurality of real-machine sensors 416, for example, a gyro sensor.

[0062] The remote control device 200 of the remote operation apparatus 20 superimposes the index image M on the work environment image based on at least one of the real-space position of the second index point P2 and the three-dimensional image or the pixel position (u, v) corresponding to the second index point P2 in the work environment image, and makes the remote image output device 221 output a work assistance image which is the image generated as described above (STEP218). The remote control device 200, thus, includes an image generation part that generates the work assistance image.

[0063] Specifically, in the example shown in FIG. 7, the work assistance image is an image in which the index images M1 and M2 indicating the second index points P1 and P2, which are points at which the first index points P11 and P12 are projected onto an object surface such as a ground surface, respectively, are superimposed on the work environment image that is the captured image. Similarly, in the example shown in FIG. 8, the work assistance image is an image in which an index image M indicating the second index point P2, which is the point at which the first index point P1 is projected onto an object surface such as a ground surface, is superimposed on the work environment image that is the captured image.

[0064] Each of the index images M1 and M2 or the index image M has a directive shape such as a triangular shape or an arrow shape directed to the second index points P1 and P2 or the second index point P2 along the vertical direction in the real space. It may be done either to superimpose the first index point P1 and the second index point P2 on the captured image and display it or to omit the superimposition display of at least one of the first index point P1 and the second index point P2.

[0065] The remote control device 200 of the remote operation apparatus 20 recognizes the operation state of the remote operation mechanism 211, and transmits a remote operation command corresponding to the operation state to the work machine 40 through the remote wireless communication device 224 (STEP220).

[0066] When receiving the operation command through the real-machine wireless communication device 422 (C44), the real-machine control device 400 of the work machine 40 controls the motions of the work mechanism 44 and the like in accordance with the operation command (STEP 420). For example, when the distal attachment 445 is the grapple, work is executed in which the grapple holds a designated object located frontward of the work machine 40, for example, a natural product such as a stone or a tree or an artificial object such as a block, the upper turning body 42 turns, and then the distal attachment 445 releases the designated object to put the designated object onto a designated place. Besides, when the distal attachment 445 is a bucket, work is executed in which the bucket scoops and holds a designated object such as earth and sand laid frontward of the work machine 40, the upper turning body 42 turns, and then the distal attachment 445 releases the designated object to put the designated object onto a designated place.

[0067] As has been described, the remote control device 200 constituting the work assistance system in the embodiment includes the change processing part, which executes the specification information change processing when at least one of the work machine 40, the work site, and the distal attachment 445 is the changed target that has been changed. The specification information change processing is processing for changing specification information of the work machine as a basis for estimation of the ground projection spot to adapt the specification information to the change target, being processing of changing the specification information or processing of making an operator who operates the work machine 40 change the specification information.

[0068] The specification information change processing enables the ground projection spot (for example, the second index point P2), which is the spot at which the designated part (for example, the first index point P1) in the distal attachment 445 is projected on the ground, to be estimated based on the specification information that has been changed to be adapted to the change target, thereby enabling the accuracy with which an operator recognizes the position of the ground projection spot through the work assistance image to be kept high regardless of the change of the change target.

[0069] The work assistance system is not limited to one constituted by the remote control device 200 of the remote operation apparatus 20 according to the embodiment. The work assistance system may be constituted either by a computer having a function of communicating with each of the remote operation apparatus 20 and the work machine 40 or by the computer and the remote control device 200. Alternatively, the work assistance system may be constituted by the remote control device 200 of the remote operation apparatus 20 and the real-machine control device 400 of the work machine 40.

[0070] The index indicating the suitability of respective real-space positions of the first index points P1, P11, and P12 is allowed to be variously set. For example, in view of the work performed by the distal attachment 445 for the designated object OBJ, for example, the work of holding the designated object OBJ, the first state where the real-space position is appropriate, the second state where the real-space position is inappropriate, and the third state where the real-space position is slightly inappropriate, may be expressed either by respective colors of the index images M, M1, and M2, for example, black, white, gray, respectively, while keeping respective shapes of the index images M, M1, and M2 constant, or by the difference in the design of the index image M in another mode. Specifically, in order to represent each of the first to third states, may be differentiated any of the follows: the color, shape, or pattern of the index image M, or any combination thereof; the mode of the temporal change in the design; only the shape of the index image without the change of the color thereof; and the presence / absence of blinking of the index image M and / or the length of the blinking cycle.

[0071] In the work assistance image, there may be superimposed on the work environment image, in addition to the index image M, a figure representing the shape of the ground and / or a figure representing the surface shape of the target object, for example, a figure provided with a mesh. The width or color of a diagram constituting the mesh, the interval between the meshes, and the like may be set on the specification information setting image by an operation applied to the remote input interface 210.

[0072] As has been described, there are provided a system and a method that allow accuracy with which an operator recognizes the position of a ground projection spot to be kept high regardless of a change of at least one of a work machine, a work site, and a distal attachment.

[0073] Provided is a work assistance system including an image generation part and a change processing part. The image processing part estimates a ground projection spot, at which a designated part which is a part designated in advance in a distal attachment of a work machine is projected onto the ground, based on specification information of the work machine, and generates a work assistance image in which an index image indicating the ground projection spot is superimposed on a work environment image acquired by an imaging device. The work assistance image is an image of a periphery of the work machine, containing the distal attachment and the ground. The change processing part executes specification information change processing when at least one of the work machine, a work site where the work machine performs work, and the distal attachment is a change target that has been changed. The specification information change processing is processing for adapting the specification information as a basis for estimation of the ground projection spot to the change target, which processing may be either processing for changing the specification information or processing for making an operator who operates the work machine change the specification information.

[0074] The ground may be either a curved surface and / or an uneven surface that reproduces a shape including a ground surface around the work machine at the work site and an object present on the ground surface with high accuracy or a flat surface or a horizontal surface roughly along the ground surface.

[0075] The specification information change processing enables new specification information that has been changed to be adapted to the change target to be used for the estimation of the ground projection spot, which is the spot at which the designated part in the distal attachment is projected on the ground, thereby enabling the accuracy with which an operator recognizes the position of the ground projection spot through the index image to be kept high regardless of the change of the change target.

[0076] Preferably, the work assistance system further includes: an output interface that outputs the specification information that has been changed to be adapted to the change target when at least one of the work machine, the work site, and the distal attachment is the change target; and an input interface that allows a confirmation input operation to be applied to the input interface by the operator, wherein the change processing part is configured to fix the specification information as the basis for estimation of the ground projection spot on condition that the confirmation input operation is applied to the input interface. This allows the specification information to be determined in consideration of the intention of the operator.

[0077] The work assistance system, alternatively, may further include: an output interface that outputs a specification setting image for setting the specification information when at least one of the work machine, the work site, and the distal attachment is the change target; and an input interface that allows a setting operation for setting the specification information to be applied to the input interface by an operator, wherein the change processing part is configured to set the specification information as the basis for estimation of the ground projection spot in accordance with the setting operation. This allows the specification information to be set in consideration of the intention of the operator.

[0078] Also provided is a work assistance method including: a step of acquiring, by an imaging device, a work environment image that is an image of a periphery of the work machine, the work environment image containing a distal attachment of the work machine and a ground; a step of estimating, based on specification information of the work machine, a ground projection spot, which is a spot at which a designated part which is a part designated in advance in the distal attachment is projected onto the ground, and generating a work assistance image in which an index image indicating the ground projection spot is superimposed on the work environment image; and a step of executing specification information change processing when at least one of the work machine, a work site where the work machine performs work, and the distal attachment is a change target that has been changed, the specification information change processing being processing for adapting the specification information as a basis for estimation of the ground projection spot to the change target.

Claims

1. A work assistance system comprising: an image processing part that estimates a ground projection spot, at which a designated part which is a part designated in advance in a distal attachment of a work machine is projected onto the ground, based on specification information of the work machine and generates a work assistance image in which an index image indicating the ground projection spot is superimposed on a work environment image acquired by an imaging device, the work assistance image being an image of a periphery of the work machine, the work assistance image containing the distal attachment and a ground; and a change processing part that executes specification information change processing when at least one of the work machine, a work site where the work machine performs work, and the distal attachment is a change target that has been changed, the specification information change processing being processing for adapting the specification information as a basis for estimation of the ground projection spot to the change target.

2. The work assistance system according to claim 1, further comprising: an output interface that outputs the specification information that has been changed to be adapted to the change target when at least one of the work machine, the work site, and the distal attachment is the change target; and an input interface that allows a confirmation input operation to be applied to the input interface by an operator, wherein the change processing part is configured to fix the specification information as the basis for estimation of the ground projection spot on condition that the confirmation input operation is applied to the input interface.

3. The work assistance system according to claim 1, further comprising: an output interface that outputs a specification setting image for setting the specification information when at least one of the work machine, the work site, and the distal attachment is the change target; and an input interface that allows a setting operation for setting the specification information to be applied to the input interface by an operator, wherein the change processing part is configured to set the specification information as the basis for estimation of the ground projection spot in accordance with the setting operation.

4. A work assistance method comprising: a step of acquiring, by an imaging device, a work environment image that is an image of a periphery of a work machine, the work environment image containing a distal attachment of the work machine and a ground; a step of estimating, based on specification information of the work machine, a ground projection spot, which is a spot at which a designated part which is a part designated in advance in the distal attachment is projected onto the ground, and generating a work assistance image in which an index image indicating the ground projection spot is superimposed on the work environment image; and a step of executing specification information change processing when at least one of the work machine, a work site where the work machine performs work, and the distal attachment is a change target that has been changed, the specification information change processing being processing for adapting the specification information as a basis for estimation of the ground projection spot to the change target.

Images