Excavator construction support system

Abstract

The invention provides a construction support system for a shovel, which can further improve the working efficiency of the shovel. This construction support system is provided with: a shovel; an environment detection device for detecting environment information of a working site of the excavator; and a simulation device for acquiring the environment information detected by the environment detection device and generating a three-dimensional virtual space model of the work site during the work of the shovel. Therefore, the construction support system of the excavator can update the environment information according to the operation of the excavator and set the operation content according to the environment information, so that the operation efficiency of the excavator can be further improved.

Description

Technical Field

[0001] The present invention relates to a construction support system for an excavator. Background Art

[0002] Conventionally, in order to improve the work efficiency of an excavator, a construction support system for supporting the excavation operation etc. of an excavator has been known. For example, in Patent Document 1, a construction support system is disclosed which evaluates the work quality of a plurality of operators, learns the parameters of an operation model of an excavator based on the operation data of the best operator, and supports the operator based on the learned operation model.

[0003] Moreover, such a construction support system for an excavator can set the operation content of the excavator corresponding to the sandy soil information by investigating and pre-identifying the sandy soil information such as the soil quality at the position (excavation target) where the excavator performs excavation.

[0004] Prior Art Documents

[0005] Patent Documents

[0006] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2016-156193 Summary of the Invention

[0007] Technical Problem to be Solved by the Invention

[0008] In an actual work site, even if the sandy soil information is grasped in advance, the sandy soil information is likely to change depending on the excavation position, excavation depth, etc. Therefore, even if the construction support system sets the operation content of the excavator according to the sandy soil information, sometimes the actual excavator cannot perform excavation in a predetermined manner.

[0009] The present invention provides a construction support system for an excavator which updates environmental information as the excavator operates and sets the operation content according to the environmental information, thereby being able to further improve the work efficiency of the excavator.

[0010] Means for Solving the Technical Problem

[0011] According to an embodiment of the present invention, there is provided a construction support system for an excavator, which includes: an excavator; an environment detection device that detects environmental information of the work site of the excavator; and a simulation device that acquires the environmental information detected by the environment detection device during the operation of the excavator and generates a three-dimensional virtual space model of the work site.

[0012] Advantageous Effects of the Invention

[0013] According to an embodiment, environmental information is updated as the excavator operates and the operation content is set according to the environmental information, thereby being able to further improve the work efficiency of the excavator. Description of the Drawings

[0014] Figure 1 is a diagram illustrating the construction support system according to the first embodiment.

[0015] Figure 2 is a functional block diagram showing a structural example of the construction support system.

[0016] Figure 3 is a diagram showing an example of the three-dimensional virtual space model generated by the virtual space generation unit.

[0017] Figure 4 is an explanatory diagram showing the excavation operation of the virtual excavator.

[0018] Figure 5 is an explanatory diagram showing the sand discharge operation of the virtual excavator.

[0019] Figure 6 is an explanatory diagram showing the excavation operation of the excavator controlled by the construction support system according to the first embodiment.

[0020] Figure 7 is a diagram illustrating the construction support system according to the second embodiment.

[0021] Figure 8 is a functional block diagram showing a structural example of the construction support system.

[0022] Figure 9 is an explanatory diagram showing the excavation operation of the excavator controlled by the construction support system according to the second embodiment.

[0023] Figure 10 is an explanatory diagram showing the excavation operation of the excavator controlled by the construction support system according to the modification example. Detailed Embodiment Modes

[0024] Hereinafter, modes for implementing the present invention will be described with reference to the drawings. In each drawing, the same reference numerals may be given to the same structural parts and redundant explanations may be omitted.

[0025] 〔First Embodiment〕

[0026] First, with reference to Figure 1 , the construction support system SYS1 according to the first embodiment of the present invention will be described. Figure 1 is a diagram illustrating the construction support system SYS1 according to the first embodiment.

[0027] The excavator 100 applicable to the construction support system SYS1 includes a lower traveling body 1, an upper revolving body 3 rotatably mounted on the lower traveling body 1 via a revolving mechanism 2, an excavation attachment AT, and a cab 10.

[0028] The lower traveling body 1 of the excavator 100 according to the present embodiment has a pair of left and right crawlers 1C. The crawlers 1C are driven by a traveling actuator mounted on the lower traveling body 1, namely a traveling hydraulic motor 2M.

[0029] On the lower traveling body 1, an upper swing body 3 is swingably mounted via a swing mechanism 2. The swing mechanism 2 is driven by a swing actuator mounted on the upper swing body 3, namely a swing hydraulic motor 2A. Additionally, the swing actuator may be an electric actuator (swing electric generator).

[0030] An arm 4 is mounted on the upper swing body 3. A boom 5 is mounted at the front end of the arm 4, and a bucket 6 as an end attachment is mounted at the front end of the boom 5. The arm 4, the boom 5, and the bucket 6 constitute an excavation attachment AT as an example of an attachment. The arm 4 is driven by an arm cylinder 7, the boom 5 is driven by a boom cylinder 8, and the bucket 6 is driven by a bucket cylinder 9. The arm cylinder 7, the boom cylinder 8, and the bucket cylinder 9 constitute attachment actuators. The end attachment may be a face shovel bucket.

[0031] The arm 4 is pivotally supported relative to the upper swing body 3 so as to be rotatable up and down. An arm angle sensor S1 is mounted on the arm 4. The arm angle sensor S1 can detect the rotation angle of the arm 4, namely the arm angle α. The arm angle α is, for example, the rising angle starting from the state where the arm 4 is lowered to the maximum extent. Therefore, the arm angle α becomes maximum when the arm 4 is raised to the maximum extent.

[0032] The boom 5 is pivotally supported relative to the arm 4 so as to be rotatable. A boom angle sensor S2 is mounted on the boom 5. The boom angle sensor S2 detects the rotation angle of the boom 5, namely the boom angle β. The boom angle β is, for example, the opening angle starting from the state where the boom 5 is closed to the maximum extent. Therefore, the boom angle β becomes maximum when the boom 5 is opened to the maximum extent.

[0033] The bucket 6 is pivotally supported relative to the boom 5 so as to be rotatable. A bucket angle sensor S3 is mounted on the bucket 6. The bucket angle sensor S3 detects the rotation angle of the bucket 6, namely the bucket angle γ. The bucket angle γ is the opening angle starting from the state where the bucket 6 is closed to the maximum extent. Therefore, the bucket angle γ becomes maximum when the bucket 6 is opened to the maximum extent.

[0034] Each of the arm angle sensor S1, the boom angle sensor S2, and the bucket angle sensor S3 may use only an acceleration sensor, or may use an acceleration sensor and a gyro sensor in combination. Alternatively, the arm angle sensor S1 may be a stroke sensor mounted on the arm cylinder 7, or may be a rotary encoder, a potentiometer, an inertial measurement device, etc. The same applies to the boom angle sensor S2 and the bucket angle sensor S3.

[0035] On the upper revolving body 3, a cockpit 10 serving as an operation room is provided, and a power source such as an engine 11 is mounted. Further, a space recognition device 70, an orientation detection device 71, and a positioning device 72 are mounted on the upper revolving body 3. At the same time, various sensors of the excavator 100, such as a body tilt sensor S4 and a rotational angular velocity sensor S5, are also mounted. Moreover, an operation device 21, an output device 22, a controller 30, etc. are provided inside the cockpit 10. In addition, in this specification, for the sake of convenience of explanation, the side of the upper revolving body 3 where the excavation attachment device AT is mounted is taken as the front, and the side where the counterweight is mounted is taken as the rear.

[0036] The space recognition device 70 is a device for recognizing the three-dimensional actual space (environmental information) around the excavator 100. Further, the space recognition device 70 is configured to measure the azimuth and distance from the space recognition device 70 or the excavator 100 to the recognized object. The space recognition device 70 includes, for example, an ultrasonic sensor, a millimeter-wave radar, a single-lens reflex camera, a stereo camera, a LiDAR, a distance image sensor, an infrared sensor, etc., or any combination thereof. In the present embodiment, the space recognition device 70 includes a front sensor 70F mounted at the front end of the upper surface of the cockpit 10, a rear sensor 70B mounted at the rear end of the upper surface of the upper revolving body 3, a left sensor 70L mounted at the left end of the upper surface of the upper revolving body 3, and a right sensor 70R mounted at the right end of the upper surface of the upper revolving body 3. The space recognition device 70 may mount an upper sensor (not shown) for recognizing an object existing in the upper space of the upper revolving body 3 on the excavator 100.

[0037] The orientation detection device 71 is a device for detecting information regarding the relative relationship between the orientation of the upper revolving body 3 and the orientation of the lower traveling body 1. The orientation detection device 71 may be constituted, for example, by a combination of a geomagnetic sensor mounted on the lower traveling body 1 and a geomagnetic sensor mounted on the upper revolving body 3. Alternatively, the orientation detection device 71 may be constituted by a combination of a GNSS receiver mounted on the lower traveling body 1 and a GNSS receiver mounted on the upper revolving body 3. The orientation detection device 71 may be a rotary encoder, a rotational position sensor, etc., or any combination thereof. In a structure in which the upper revolving body 3 is rotationally driven by a rotary electric generator, the orientation detection device 71 may be constituted by a resolver. The orientation detection device 71 may also be mounted, for example, on a center joint provided in association with the slewing mechanism 2 that realizes the relative rotation between the lower traveling body 1 and the upper revolving body 3.

[0038] The orientation detection device 71 may also be constituted by a camera mounted on the upper revolving body 3. At this time, the orientation detection device 71 performs known image processing on the captured information captured by the camera to extract the image of the lower traveling body 1 included in the captured information. Further, the orientation detection device 71 determines the longitudinal direction of the lower traveling body 1 from the image of the lower traveling body 1 and derives the angle between the direction of the front-rear axis formed on the upper revolving body 3 and the longitudinal direction of the lower traveling body 1. The direction of the front-rear axis of the upper revolving body 3 is derived based on the mounting position of the camera. In particular, since the crawler 1C protrudes from the upper revolving body 3, the orientation detection device 71 can determine the longitudinal direction of the lower traveling body 1 by detecting the image of the crawler 1C. This orientation detection device 71 may be incorporated into the controller 30. Further, the camera may apply the space recognition device 70.

[0039] The positioning device 72 is configured to measure the position of the upper revolving body 3. In the present embodiment, the positioning device 72 is a GNSS receiver, which detects the position of the upper revolving body 3 and outputs the detected value to the controller 30. The positioning device 72 may be a GNSS compass. At this time, since the positioning device 72 can detect the position and orientation of the upper revolving body 3, it also functions as the orientation detection device 71.

[0040] The body tilt sensor S4 detects the tilt of the upper revolving body 3 with respect to a specified plane. In the present embodiment, the body tilt sensor S4 is an acceleration sensor that detects the tilt angle of the upper revolving body 3 around the front-rear axis and the tilt angle around the left-right axis with respect to the horizontal plane. The front-rear axis and the left-right axis of the upper revolving body 3 are orthogonal to each other, for example, and pass through a point on the rotation axis of the excavator 100, that is, the excavator center point.

[0041] The rotational angular velocity sensor S5 detects the rotational angular velocity of the upper revolving body 3. In the present embodiment, the rotational angular velocity sensor S5 is a gyro sensor, but it may also be a resolver, a rotary encoder, etc., or any combination thereof. The rotational angular velocity sensor S5 can detect the rotational speed. The rotational speed can be calculated based on the rotational angular velocity.

[0042] Hereinafter, at least one of the boom angle sensor S1, the arm angle sensor S2, the bucket angle sensor S3, the body tilt sensor S4, and the rotational angular velocity sensor S5 is also referred to as a posture detection device. The posture of the excavation attachment AT is detected, for example, based on the respective outputs of the boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3.

[0043] The operating device 21 is a device provided in the cab 10 for an operator to operate the excavator 100. For example, the operating device 21 has an operating lever and an operating pedal for controlling the drive of the actuator of the excavator 100. The actuator includes at least one of a hydraulic actuator and an electric actuator.

[0044] Moreover, the operating device 21 has an information input device (for example, a right control box, a left control box) for the operator of the excavator 100 to input information to the controller 30. The information input device can be, for example, a switch panel provided adjacent to the display device of the output device 22. Alternatively, the information input device can be a touch panel used as a display device, or a voice input device such as a microphone disposed in the cab 10. And, the information input device can be a communication device for obtaining information from the outside.

[0045] The output device 22 includes at least one of a display device and a voice output device. The display device is a liquid crystal display disposed in the cab 10. Additionally, the display device can be a display of a mobile terminal such as a smart phone. The voice output device includes at least one of a device for outputting voice to the operator in the cab 10 and a device for outputting voice to the workers outside the cab 10. Additionally, the voice output device can be a speaker of a mobile terminal.

[0046] The controller 30 is a control device for controlling the excavator 100. In the present embodiment, the controller 30 is constituted by a computer including one or more processors, a memory (volatile memory, non-volatile memory), etc. The one or more processors read and execute programs corresponding to the respective functions from the memory. For example, the respective functions include a device guidance function for guiding the manual operation of the excavator 100 performed by the operator or a device control function for causing the excavator 100 to automatically (or autonomously) operate. The controller 30 can include a contact avoidance function for causing the excavator 100 to automatically operate or brake to avoid contact with an object existing around the excavator 100.

[0047] In other words, construction support includes expressions of causing the excavator 100 to automatically operate instead of the operator, assisting the operation of the operator of the excavator 100, and providing operation information to the operator of the excavator 100. The construction support system SYS1 uses the controller 30 mounted on the above-mentioned excavator 100 to execute various functions and the environment detection device 79 that provides information to the controller 30 for the construction support of the excavator 100 (refer to Figure 2)。However, in addition to being composed of the single excavator 100, the construction support system SYS1 can also apply a management device 200 that can communicate with the excavator 100 outside the excavator 100. By applying the management device 200, the construction support system SYS1 can also support operations in a coordinated manner by multiple excavators 100 (or other construction machinery). Moreover, the construction support system SYS1 can be equipped with an external-side space recognition device 300 having the same function as the space recognition device 70 outside the excavator 100.

[0048] Next, with reference to Figure 2 , an example of the construction support system SYS1 according to the first embodiment will be described. Figure 2 FIG. is a functional block diagram showing a structural example of the construction support system SYS1.

[0049] The excavator 100 includes a controller 30, a space recognition device 70, an orientation detection device 71, a positioning device 72, various solenoid valves 41, various actuators 42, and a communication device T1 to constitute the construction support system SYS1. On the other hand, the management device 200 includes a computer main body 210 that performs various processes of the construction support system SYS1 and a communication device T2.

[0050] In addition, the external-side space recognition device 300 detects the state of the work site where the excavator 100 is located. The detection of the state of the work site includes, for example, not only photographing of the work site but also measurement of distance, shape, orientation, etc. For example, the external-side space recognition device 300 includes ultrasonic sensors, millimeter-wave radars, single-lens reflex cameras, stereo cameras, LiDARs, distance image sensors, infrared sensors, etc. provided at the work site or any combination thereof. The external-side space recognition device 300 performs wireless or wired communication with at least one of the communication device T1 of the excavator 100 and the communication device T2 of the management device 200, and sequentially transmits the detection information of the detected state of the work site.

[0051] The controller 30 of the excavator 100 has a virtual space generation unit 31, an excavator state determination unit 32, an excavation state inference unit 33, an actuator drive unit 34, a determination unit 36, an operation prediction unit 37, an operation intervention unit 38, and an action simulator 39 as functional blocks. For ease of explanation, the virtual space generation unit 31, the excavator state determination unit 32, the excavation state inference unit 33, the actuator drive unit 34, the determination unit 36, the operation prediction unit 37, the operation intervention unit 38, and the action simulator 39 are distinguished, but physical distinction is not required, and they may be wholly or partially composed of shared software components or hardware components.

[0052] The computer main body 210 of the management device 200 has a determination unit 211, an operation prediction unit 212, an operation intervention unit 213, and an action simulator 214 as functional blocks. For ease of explanation, the determination unit 211, the operation prediction unit 212, the operation intervention unit 213, and the action simulator 214 are distinguished, but physical separation is not necessary, and they may be wholly or partially composed of shared software components or hardware components. Also, the determination unit 211, the operation prediction unit 212, the operation intervention unit 213, and the action simulator 214 have the same functions as the determination unit 36, the operation prediction unit 37, the operation intervention unit 38, and the action simulator 39 of the controller 30. In addition, the management device 200 may further include a virtual space generation unit 215 having the same function as the virtual space generation unit 31 of the controller 30 (refer to Figure 2 the dashed line). Hereinafter, each functional block of the controller 30 will be representatively described, and the description of each functional block of the computer main body 210 will be omitted. The construction support system SYS1 only needs to have the functions of the virtual space generation units 31 and 215, the determination units 36 and 211, the operation prediction units 37 and 212, the operation intervention units 38 and 213, and the action simulators 39 and 214 on at least one of the excavator 100 and the management device 200.

[0053] The virtual space generation unit 31 generates a three-dimensional virtual space model on the virtual three-dimensional coordinates within the virtual space generation unit 31 based on the detection information of the space recognition device 70, the orientation detection device 71, the positioning device 72, and the information of the operation device 21. The three-dimensional virtual space model is formed as a virtual rectangular parallelepiped, cube, sphere, or hemisphere according to the shooting range of the space recognition device 70. The three-dimensional virtual space model may be image information displayed on the display device of the output device 22. At this time, the three-dimensional virtual space model is a three-dimensional terrain image and is composed of computer graphics.

[0054] Typically, the three-dimensional virtual space model is information having a plurality of layers in which object information is superimposed on terrain information representing the terrain around the excavator 100 visible to the operator sitting in the cab 10. Hereinafter, the variable information (parameters) applied to the three-dimensional virtual space model will be referred to as environmental information. The environmental information includes terrain information and object information. The virtual space generation unit 31 performs known image processing on the detection information captured by at least one of the front sensor 70F, the rear sensor 70B, the left sensor 70L, and the right sensor 70R of the space recognition device 70, and extracts the terrain information and object information included in the detection information. At this time, for example, the object information included in the current captured information may be extracted by comparing a plurality of past captured information with the current captured information. Moreover, the virtual space generation unit 31 reproduces the virtual environmental information around the cab 10 (operator) in the three-dimensional virtual space model by arranging the extracted terrain information and object information in the three-dimensional virtual space model.

[0055] For example, the object information extracted for reproducing the three-dimensional virtual space model may include, for example, the soil to be excavated (including mounds, holes, walls, ditches, etc.), stationary objects other than the excavation target, the excavator 100 itself, construction machinery such as other excavators, vehicles, animals including humans, plants, and the like. Further, when the virtual space generation unit 31 arranges the object information in the three-dimensional virtual space model, it can utilize information such as the distance and azimuth between the excavator 100 and the object information measured by ultrasonic sensors, millimeter-wave radars, LiDARs, and the like. Thereby, various objects at the work site where the excavator 100 is located and the coordinates of the objects are accurately reproduced in the three-dimensional virtual space model.

[0056] In addition, the virtual space generation unit 31 can receive the detection information of the external space recognition device 300 and generate a three-dimensional virtual space model photographed by the external space recognition device 300. Alternatively, the virtual space generation unit 31 may also be configured to generate a single three-dimensional virtual space model by processing the detection information of the space recognition device 70 of the excavator 100 and the detection information of the external space recognition device 300.

[0057] Figure 3 FIG. 9 is a diagram showing an example of the three-dimensional virtual space model 50 generated by the virtual space generation unit 31. For example, a virtual excavator 51 that reproduces the excavator 100 in the actual space is arranged in the three-dimensional virtual space model 50 of the virtual space generation unit 31. The virtual excavator 51 corresponds to the shape, position, posture, etc. of the excavator 100 in the actual work site. The position and posture of the virtual excavator 51 in the three-dimensional virtual space model 50 are determined by an excavator state determination unit 32 described later, for example, based on the detection information of at least one of the space recognition device 70 and the orientation detection device 71. In addition, the position and posture of the virtual excavator 51 can be determined or adjusted by using the detection information of a posture detection device or a positioning device 72 or the like.

[0058] Topographic information existing around the virtual excavator 51, that is, the excavation target 52 (a mound in FIG. 9), object information such as a virtual dump truck 53 (discharge target), etc., including its shape itself, are arranged in the three-dimensional virtual space model 50. In addition, in FIG. 9, the illustration of other environmental information is omitted, but object information extracted from the detection information is appropriately arranged in the three-dimensional virtual space model 50. Further, the virtual space generation unit 31 can generate the three-dimensional virtual space model 50 by leaving an image that is not easily extracted from the detection information as the background of the three-dimensional virtual space model 50. Figure 3 which is a mound), object information such as a virtual dump truck 53 (discharge target), etc., including its shape itself, are arranged in the three-dimensional virtual space model 50. In addition, in Figure 3 FIG. 9, the illustration of other environmental information is omitted, but object information extracted from the detection information is appropriately arranged in the three-dimensional virtual space model 50. Further, the virtual space generation unit 31 can generate the three-dimensional virtual space model 50 by leaving an image that is not easily extracted from the detection information as the background of the three-dimensional virtual space model 50.

[0059] Moreover, the virtual space generation unit 31 according to the present embodiment associates (adds) various additional information with the terrain information or object information reproduced in the three-dimensional virtual space model 50.

[0060] For example, in addition to adding the shape, position, or posture to the virtual excavator 51, additional information of the excavator such as identification numbers, types, operation times, types or postures of the excavation attachment device AT, and information of the sand and soil such as the weight, volume, and density of the sand and soil loaded in the bucket 6 (additional information) are added. The additional information of the excavator can be inferred based on the detection information of the environment detection device 79, or the information pre-stored in the controller 30 can be used. Moreover, the information of the sand and soil input to the excavator 100 can be detected by a part of the environment detection device 79, such as a pressure sensor or a load sensor, which detects the load borne by the excavator 100 during excavation and is inferred based on this information.

[0061] In other words, the environment detection device 79 refers to a device that detects information that affects the environment information constituting the three-dimensional virtual space model 50 inside or outside the excavator 100. The environment detection device 79 may include various sensors such as a space recognition device 70, an orientation detection device 71, a positioning device 72, a posture detection device (boom angle sensor S1, arm angle sensor S2, bucket angle sensor S3, body tilt sensor S4, and swing angular velocity sensor S5), a pressure sensor (not shown), a load sensor, and an operation sensor of the operation device 21. Moreover, the environment detection device 79 may include the detection information of a device provided outside the excavator 100 (an external side space recognition device 300, other construction machinery or vehicles (for example, a dump truck)).

[0062] Moreover, for example, additional information of the excavation target 52 in the three-dimensional virtual space model 50 such as the accumulated amount, weight, density, hardness, and soil quality of the sand and soil is added. The additional information of the excavation target is inferred by the excavation state inference unit 33 described later based on the detection information of the space recognition device 70. Alternatively, the additional information of the excavation target can also be inferred from the pressure or load borne by the bucket 6 of the excavator 100 when the excavator 100 operates, the weight or image of the sand and soil recognized in the dump truck when discharging the sand and soil to the dump truck, etc. Alternatively, a part of the additional information of the excavation target can be extracted from the design data of the work site pre-stored in the controller 30.

[0063] In addition to the additional shape, position, and posture of the virtual dump truck 53 in the three-dimensional virtual space model 50, additional information of the discharge object such as identification number, type, dimensions of the cargo box, state of the cargo box, and operation time is also added. Moreover, information such as the weight, volume, and density of the sand in the cargo box loaded in the cargo box of the virtual dump truck 53 (discharge object) can be added to the cargo box of the virtual dump truck 53. The additional information of the discharge object can be inferred from the object information extracted from the detection information of the space recognition device 70, or the information pre-stored in the controller 30 can be used. By detecting and receiving information such as the weight or image of the sand that changes when the excavator 100 discharges the sand in an actual dump truck, the sand information is added to the virtual dump truck 53 in the three-dimensional virtual space model 50.

[0064] In addition, various information can be cited as the additional information added to the environmental information other than the above. For example, when there are people or other construction machinery around the excavator 100, these object information are configured in the three-dimensional virtual space model 50. Moreover, it is preferable to add additional information such as the relative distance from the excavator 100 to the object information. Also, it is preferable to add additional information such as identification number, type, and operation time to the information of other construction machinery.

[0065] The virtual space generation unit 31 can merge the design data of the work site stored in the controller 30 into the three-dimensional virtual space model 50. The design data has the completed excavation shape of the work site, and the virtual space generation unit 31 superimposes and displays graphics such as computer graphics representing the position of the completed shape on the three-dimensional virtual space model 50. Moreover, the design data includes the state of the sand (position, shape, soil quality, hardness) at the excavation location that has been pre-investigated, and these information can be added to the terrain information of the three-dimensional virtual space model 50.

[0066] Furthermore, the virtual space generation unit 31 sequentially updates the environmental information of the three-dimensional virtual space model 50 according to the detection information of the space recognition device 70, the orientation detection device 71, the positioning device 72, the information of the operation device 21, and the communication information from the external space recognition device 300 or the dump truck. For example, when the excavator 100 excavates the excavation object at the work site, it is preferable to change the environmental information of the three-dimensional virtual space model 50 according to the detection information at that time. Even if the sand information (additional information) of the excavation object 52 has been pre-added, the sand information may be different when the excavator 100 actually excavates. In this case, the virtual space generation unit 31 changes or corrects the sand information according to the detection information detected when the excavator 100 actually excavates. Thus, the virtual space generation unit 31 can gradually generate a three-dimensional virtual space model 50 that is closer to the work site environment.

[0067] Return Figure 2, the excavator state determination unit 32 is configured to determine the state of the excavator 100 including the position and orientation of the excavator 100 (such as the posture of the excavation attachment AT). The position of the excavator 100 is, for example, the latitude, longitude, and altitude of a reference point on the excavator 100. The excavator state determination unit 32 determines the position of the excavator 100 based on the output of the positioning device 72 and determines the orientation of the excavator 100 based on the output of the orientation detection device 71. The position and orientation of the excavator 100 determined by the excavator state determination unit 32 are reflected in the virtual excavator 51 of the three-dimensional virtual space model 50.

[0068] The excavation state inference unit 33 infers additional information of the excavation object required during the operation of the excavator 100 based on the detection information of the environment detection device 79. In addition, the "operation" of the excavator 100 in this specification refers to a concept including a series of actions such as the excavation action of the excavation object, i.e., sand and soil, the transportation of sand and soil accompanying the ascending and slewing actions of the excavator 100, the discharging of sand and soil into the dump truck cargo box, and the return of the descending and slewing actions of the excavator 100. Examples of the additional information of the excavation object include the hardness, soil density, and soil quality of the ground to be excavated. When performing the excavation action of the excavation object in the actual space, a reaction force of excavation is applied to the excavation attachment AT from the excavation object. Therefore, the excavation state inference unit 33 can infer the additional information of the excavation object based on the pressure detected by the pressure sensors in the hydraulic paths of the respective cylinders of the excavation attachment AT or the load detected by the load sensors arranged at appropriate positions of the excavation attachment AT. In addition, the excavation state inference unit 33 can infer the sand and soil information based on the detection information of various sensors not only during the excavation of the excavation object but also when holding the sand and soil by the excavation attachment AT. As described above, the additional information of the excavation object inferred by the excavation state inference unit 33 is added to the sand and soil information of the virtual excavator 51 in the three-dimensional virtual space model 50 and is appropriately reflected in the additional information of the excavation object 52 excavated by the virtual excavator 51.

[0069] The actuator drive unit 34 is configured to drive various solenoid valves 41 and various actuators 42 mounted on the excavator 100. In addition to outputting the operation signal of the operation device 21, the actuator drive unit 34 outputs a working signal to the corresponding solenoid valve 41 or the corresponding actuator 42 according to the control signal processed by the controller 30 itself. By driving with the solenoid valve 41 and the actuator 42 that receive the working signal, the excavator 100 performs various actions (excavation action, ascending and slewing action, discharging of sand and soil action, descending and slewing action, etc.) during operation.

[0070] On the other hand, the determination unit 36 of the controller 30 is configured to determine whether there is a matter that should be notified to the operator of the excavator 100 regarding the situation around the excavator 100. For example, the determination unit 36 determines whether there is a matter that should be notified to the operator of the excavator 100 based on at least one of the detection information of the environment detection device 79 provided in the excavator 100 and the set operation content of the excavator 100. Further, the determination unit 36 may be configured to determine whether there is a matter that should be notified to the operator of the excavator 100 based on the detection information of the external space recognition device 300. In addition, the external space recognition device 300 may be a sensor (such as a camera, LiDAR, etc.) installed on other construction machinery, or a sensor (such as a camera, LiDAR, etc.) installed on a flying body such as a multi-axis aircraft (drone) flying over the work site. The determination unit 36 may utilize the three-dimensional virtual space model 50 of the virtual space generation unit 31 in the determination, or may utilize the simulation result of the action simulator 39 described later.

[0071] For example, the determination unit 36 determines at least one of the position, posture, and operation content of other construction machinery or vehicles based on the detection information of the space recognition device 70 or the external space recognition device 300. Whether there is a matter to be notified may also be determined by comparing with past cases and based on the presence or absence of the same or similar situations. Further, for example, when the determination unit 36 recognizes a person near the excavator 100 (within the operation range of the excavation attachment AT), it determines that there is a matter that should be notified to the operator.

[0072] Moreover, the determination unit 36 may determine that there is a matter that should be notified to the operator when it detects a downhill slope around the excavator 100. At this time, the determination unit 36 may determine the downhill slope based on the terrain information of the design data pre-stored in the controller 30. Or, when the determination unit 36 detects an object (such as an electric wire) that may affect the operation outside the range covered by the three-dimensional virtual space model 50, it determines that there is a matter that should be notified to the operator.

[0073] When it is determined that there is a matter that should be notified to the operator of the excavator 100, the determination unit 36 performs a process for attracting the operator's attention. As an example, the determination unit 36 sends information related to the notification matter to the output device 22. Thereby, the output device 22 can notify the operator of the information related to the notification matter received from the determination unit 36.

[0074] The operation prediction unit 37 is configured to predict the operation signal after a specified time based on the operation signal from the operating device 21 or the signal from the management device 200. This is to suppress the delay caused by the processing overload or the reduction in operation responsiveness caused by the communication delay. The specified time is, for example, several milliseconds to tens of milliseconds. For example, the operation prediction unit 37 predicts the operation signal after a specified time based on the change of the operation signal (the inclination angle of the operating lever of the operating device 21) within the past specified time. As an example, when the operation prediction unit 37 detects that the inclination angle of the operating lever has an increasing trend within the past specified time, it predicts that the inclination angle after a specified time will be greater than the current inclination angle. As a result, the excavator 100 can reduce the delay of the operation signal and move the excavator 100.

[0075] The operation intervention unit 38 is configured to determine whether to intervene in the operation performed by the operator of the shovel 100 based on the detection information of the environment detection device 79 and intervene in the operation as needed. For example, the operation intervention unit 38 plays a contact avoidance function of preventing the shovel 100 from contacting other objects by intervening in the operation of the operator. The operation intervention unit 38 can use the three-dimensional virtual space model 50 of the virtual space generation unit 31 in the determination of the operation intervention, and can also use the simulation results of the motion simulator 39 described later.

[0076] The operation intervention unit 38 determines that the operation should be intervened when it is detected that the excavator 100 is in contact with an object existing around the excavator 100. As an example, the operation intervention unit 38 determines that the operation should be intervened when it is detected that there is a person on the left side of the excavator 100 and the left turn operation (the operation of pushing the left operating lever to the left) has been started. At this time, the operation intervention unit 38 invalidates the operation signal generated according to the left turn operation to prevent the upper rotating body 3 from rotating to the left. In addition, the operation intervention unit 38 can also determine whether the excavator 100 is in contact with an object based on the detection information of the external side space recognition device 300 and intervene in the operation.

[0077] Furthermore, the operation intervention unit 38 preferably cancels the braking action (stopping, deceleration, etc.) during intervention when a cancel condition (for example, the operation lever is temporarily returned to the neutral position or a cancel button is pressed) is satisfied during intervention by the operation intervention unit 38 .

[0078] The motion simulator 39 is configured to simulate the motion of the excavator 100 in the three-dimensional virtual space model 50. In addition, the "motion of the excavator 100" refers to various motions classified in the operation based on the excavator 100. For example, the excavation operation is composed of a plurality of motions such as the excavation motion, the lifting and rotating motion, the sand and soil discharge motion, and the lowering and rotating motion. The motion simulator 39 is configured to simulate the motion of the excavator 100 according to the automatic control button 23 (refer to Figure 6) The opening operation is performed to simulate the actions of the excavator 100. For example, the automatic control button 23 is provided on the operating device 21 in the cab 10.

[0079] Specifically, the action simulator 39 uses the three-dimensional virtual space model 50 generated by the virtual space generation unit 31 to simulate the operations (excavation actions, raising and slewing actions, discharging sand actions, lowering and slewing actions, etc.) of the virtual excavator 51. In other words, the controller 30 of the excavator 100 functions as the simulation device of the present invention. The information simulated by the action simulator 39 can be displayed on the display device of the output device 22. The terrain information of the three-dimensional virtual space model 50 can be changed according to the actions of the simulated virtual excavator 51.

[0080] Figure 4 It is an explanatory diagram showing the excavation action of the virtual excavator 51. As Figure 4 shown, for example, when simulating the excavation action, the action simulator 39 generates a plurality of excavation trajectories (simulated action information) for the excavation attachment 51a of the virtual excavator 51 to act on the excavation target 52 of the three-dimensional virtual space model 50. Using the detection information of the posture detection device of the excavator 100 in the actual space, a plurality of excavation trajectories are generated according to the actual possible action range of the excavation attachment AT of the excavator 100 in the actual space and the additional information of the excavation target 52.

[0081] For example, the action simulator 39 sets the action range (including the moving direction, moving distance, posture, etc.) of the excavation attachment 51a of the virtual excavator 51 at the position of the excavation target 52. Moreover, the action simulator 39 envisions a plurality of excavation modes in which the order of the excavation positions and the excavation amount (or excavation depth) of the excavation target 52 within the action range of the excavation attachment 51a are appropriately changed. In the envisioning of the plurality of modes, the density, hardness, soil quality, etc. of the sand in the sand information of the excavation target 52 are utilized.

[0082] Moreover, the action simulator 39 generates a plurality of excavation trajectories of the virtual excavator 51 corresponding to the action range of the excavation attachment 51a and the plurality of excavation modes. In Figure 4 the example of (a) in the upper right figure on the right, it shows a mode in which the position to be excavated initially is set to the upper part of the excavation target 52, and the surface of the excavation target 52 is successively cut downward, and then the inner part of the excavation target 52 is excavated. On the other hand, in Figure 4In the example of (b) in the upper right figure, a mode is shown in which the position to be initially excavated is set to the middle of the excavation target 52, and fine excavation is performed on its periphery. For example, the mode of (a) is the case where the sandy soil of the excavation target 52 is soft, and the mode of (b) is the case where the sandy soil of the excavation target 52 is hard. However, as described above, the density, hardness, soil quality, etc. of the excavation target in the actual space sometimes differ from the information before the excavation operation of the excavator 100. Therefore, even when the motion simulator 39 is associated with the additional information of the excavation target, it is preferable to pre-generate a plurality of excavation trajectories when changing the density, hardness, and soil quality of the excavation target 52.

[0083] Then, after the motion simulator 39 generates a plurality of excavation trajectories, it appropriately evaluates (simulates) the plurality of excavation trajectories and selects the optimal excavation trajectory. In the evaluation of the optimal excavation trajectory, for example, it is preferable to apply an objective function for efficiently excavating the excavation target 52 and perform the evaluation using constraint conditions such as the operation speed, operation time, and operation safety of the excavation attachment 51a. Thus, the motion simulator 39 can obtain the optimal excavation trajectory corresponding to the shape, position, and additional information of the currently acquired excavation target.

[0084] When the controller 30 automatically controls the excavator 100 through the construction support system SYS1, it provides this optimal excavation trajectory to the actuator drive unit 34 as simulated motion information. Thus, the actuator drive unit 34 can make the excavation attachment 51a in the actual space move along the excavation trajectory by controlling various solenoid valves 41 and various actuators 42 (refer to Figure 4 the lower figure).

[0085] When the excavation attachment AT in the actual space performs an excavation operation along the excavation trajectory, the controller 30 acquires the detection information of the environment detection device 79 (including a pressure sensor and a load sensor) and the communication information from the dump truck. Moreover, the virtual space generation unit 31 updates the terrain information or the shape, position, posture, and additional information attached to them of the object information in the three-dimensional virtual space model 50 based on this information.

[0086] Moreover, the motion simulator 39 corrects the excavation trajectory based on the terrain information, object information, and additional information of the updated three-dimensional virtual space model 50. In addition, the motion simulator 39 can reselect the previously calculated plurality of excavation trajectories, or can also recalculate the excavation trajectory. Moreover, when the motion simulator 39 newly provides the optimal excavation trajectory, the actuator drive unit 34 switches to this excavation trajectory and makes the excavation attachment AT in the actual space move.

[0087] Figure 5 is an explanatory diagram showing the sand discharge operation of the virtual excavator 51. As Figure 5As shown, when simulating the sand discharging operation, the motion simulator 39 generates a plurality of sand discharging trajectories (simulation motion information) for causing the excavation attachment 51a of the virtual excavator 51 to operate on the virtual dump truck 53 of the three-dimensional virtual space model 50. Based on various sensors of the excavator 100 in the actual space, a plurality of sand discharging trajectories are generated according to the actual possible motion range of the excavation attachment AT of the excavator 100 in the actual space and the unloading information of the virtual dump truck 53.

[0088] For example, the motion simulator 39 sets the motion range of the excavation attachment 51a of the virtual excavator 51 during sand discharging (including the moving direction, moving distance, posture, etc. of the excavation attachment 51a) at a position above the cargo box of the virtual dump truck 53. Moreover, the motion simulator 39 envisions a plurality of sand discharging patterns in which the position and loading amount of the sand loaded into the cargo box of the virtual dump truck 53 within the motion range of the excavation attachment 51a are appropriately changed. In the envisioning of the plurality of patterns, the accumulated amount, density, hardness, soil quality, etc. of the sand in the sand information of the excavator 100 and / or the dump truck are utilized.

[0089] Moreover, the motion simulator 39 generates a plurality of sand discharging trajectories of the virtual excavator 51 corresponding to the motion range of the excavation attachment 51a and the plurality of sand discharging patterns. For example, in the Figure 5 upper right figure, the sand discharging position is set at the rear of the cargo box of the virtual dump truck 53, and a trajectory is shown in which after sand discharging, the sand in the cargo box is evenly spread forward by the excavation attachment 51a.

[0090] After generating a plurality of sand discharging trajectories, the motion simulator 39 appropriately evaluates (simulates) the plurality of sand discharging trajectories and selects the best sand discharging trajectory. The best sand discharging trajectory preferably applies, for example, an objective function for evenly loading the sand into the cargo box of the virtual dump truck 53, and is evaluated using constraint conditions such as the motion speed, operation time, and operation safety of the excavation attachment 51a. Thus, the motion simulator 39 can obtain the best sand discharging trajectory according to the sand information currently acquired.

[0091] When the controller 30 makes the excavator 100 perform automatic control through the construction support system SYS1, the best sand discharging trajectory is provided to the actuator drive unit 34. Thus, the actuator drive unit 34 can control various solenoid valves 41 and various actuators 42 to make the excavation attachment AT in the actual space move along the sand discharging trajectory (refer to the Figure 5 lower figure).

[0092] When the earth-discharging auxiliary device AT in the actual space performs the earth-discharging operation along the earth-discharging trajectory, the controller 30 acquires the detection information of the environment detection device 79. Moreover, based on this information, the virtual space generation unit 31 updates the shape, position, posture, and additional information attached to them of the terrain information or object information in the three-dimensional virtual space model 50.

[0093] The motion simulator 39 corrects the optimal earth-discharging trajectory according to the terrain information, object information, and additional information of the updated three-dimensional virtual space model 50. In addition, the motion simulator 39 can reselect multiple previously calculated earth-discharging trajectories, or can also recalculate the earth-discharging trajectory. Then, when a new optimal earth-discharging trajectory is provided, the actuator drive unit 34 switches to this earth-discharging trajectory to make the earth-digging auxiliary device AT in the actual space act.

[0094] The construction support system SYS1 of the excavator 100 according to the first embodiment is basically configured as shown above. The following will refer to Figure 6 to describe its operation. Figure 6 It is an explanatory diagram showing the control of the operation of the excavator 100 based on the construction support system SYS1 according to the first embodiment. In addition, hereinafter, an example of the automatic control (equipment control function) of the excavator 100 by the controller 30 will be described. The construction support system SYS1 is not limited to this, and can also act as an equipment guidance function for guiding the trajectory generated by the controller 30 to the operator. And the construction support system SYS1 can also perform the automatic control or guidance of the excavator 100 through the management device 200.

[0095] The controller 30 starts the automatic control of the operation of the excavator 100 according to the operator's operation of turning on the automatic control button 23 of the excavator 100. After starting, the virtual space generation unit 31 generates a three-dimensional virtual space model 50 based on the stored environment information before the operation of the excavator 100 (step S1). The environment information stored before the operation is the information stored in the controller 30, and the three-dimensional virtual space model 50 of the previous time (for example, the previous day) can be continued to be used. As described above, in addition to attaching the terrain information and object information to the terrain information and object information of the three-dimensional virtual space model 50, various information such as additional information of the virtual excavator 51, sand information (additional information) of the virtual excavator 51, additional information of the excavation target 52, additional information of the dump truck (discharge target), and sand information (additional information) of the virtual dump truck 53 are also attached.

[0096] Then, the motion simulator 39 simulates the operation of the virtual excavator 51 in the three-dimensional virtual space model 50 (step S2). That is, the motion simulator 39 generates the optimal trajectories in the operations (excavation motion, rising and slewing motion, discharging sand motion, falling and slewing motion) respectively according to the environmental information. Thereby, the controller 30 can obtain the continuous excavation trajectory, rising and slewing trajectory, discharging sand trajectory, and falling and slewing trajectory on the operation time axis.

[0097] The actuator drive unit 34 receives the information of the excavation trajectory, rising and slewing trajectory, discharging sand trajectory, and falling and slewing trajectory generated in the motion simulator 39 (step S3). Thereby, the actuator drive unit 34 controls various solenoid valves 41 and various actuators 42 etc. corresponding to these trajectories in the excavator 100 in the real space (step S4). That is, the excavator 100 in the real space automatically performs the excavation motion, rising and slewing motion, discharging sand motion, and falling and slewing motion as actual operations.

[0098] The actuator drive unit 34 can correct the operation of the excavation attachment AT (position, speed, acceleration, etc.) by feeding back the detection information of various sensors of the excavator 100 during the excavation motion. And during the rising and slewing motion, discharging sand motion, and falling and slewing motion, the actuator drive unit 34 can perform feedback control on each motion according to the detection information of various sensors of the excavator 100.

[0099] During actual operation, the controller 30 obtains the information of the environment detection device 79 (including the space recognition device 70, the orientation detection device 71, the positioning device 72, and the excavation motion information of various sensors or the operation of the operation device 21, the communication information from the external side space recognition device 300 or the dump truck) at a specified moment. This specified moment can be set sequentially (every specified time) during the operation of the excavator 100, or can be set to obtain the log stored in that motion as the end of one motion. The virtual space generation unit 31 updates the terrain information, object information, and additional information of the three-dimensional virtual space model 50 according to this information (step S5). For example, the virtual space generation unit 31 updates the additional information (ground hardness, soil density, soil quality, etc.) of the excavation object 52 of the three-dimensional virtual space model 50 to the information inferred in the excavation state inference unit 33. And the additional information of the excavator or the sand information of the virtual excavator 51, or the additional information or sand information of the virtual dump truck 53 (discharge object) is also appropriately updated.

[0100] Thereby, the motion simulator 39 corrects various trajectories of the operation (excavation trajectory, rising and slewing trajectory, discharging sand trajectory, falling and slewing trajectory) according to the updated information (step S6). The corrected trajectory is sent to the actuator drive unit 34 and reflected in the control performed by the actuator drive unit 34 (step S7).

[0101] By repeating the above actions during the operation, the construction support system SYS1 enables the excavator 100 to accurately operate according to the real-time changing situation at the actual job site in the real space. As a result, the construction support system SYS1 enables the excavator 100 to operate efficiently and precisely.

[0102] In addition, the construction support system SYS1 according to the present invention is not limited to the above-described embodiment, and various modifications can be adopted. As an example, when the controller 30 of the construction support system SYS1 recognizes that there is someone around the excavator 100 based on the environmental information, it is preferable to generate a trajectory (action information) for the virtual excavator 51 to avoid people within the three-dimensional virtual space model 50. For example, when there is someone on the left side of the virtual excavator 51, it is preferable to switch the ascending rotation trajectory or the descending rotation trajectory of the virtual excavator 51 from the left-handed rotation trajectory to the right-handed rotation trajectory.

[0103] 〔Second Embodiment〕

[0104] Next, with reference to Figure 7 and Figure 8 , the construction support system SYS2 for the excavator 100 according to the second embodiment will be described. Figure 7 FIG. is an example of the construction support system SYS2 according to the second embodiment. Figure 8 FIG. is a functional block diagram showing a structural example of the construction support system SYS2.

[0105] The construction support system SYS2 according to the second embodiment is different from the construction support system SYS1 according to the first embodiment in that the excavator 100 is remotely operated by a remote operation room RC provided at a position away from the excavator 100. In addition, the structure other than remotely operating the excavator 100 is basically the same as that of the first embodiment, and thus its detailed description is omitted.

[0106] The remote operation room RC includes a remote controller 30R, a sound output device A2, an indoor camera device C2, a display device RD, and a communication device T3. And, a driver's seat DS for the operator who remotely operates the excavator 100 is provided in the remote operation room RC.

[0107] The remote controller 30R is an arithmetic device that performs various operations. The remote controller 30R, like the controller 30 of the excavator 100, is composed of a computer including one or more processors and a memory. Various functions of the remote controller 30R are realized by the processor executing a program stored in the memory.

[0108] The sound output device A2 is a device that outputs sound and is configured to play the sound collected by a sound collection device (not shown) installed in the excavator 100.

[0109] The indoor camera device C2 is a device that captures the inside of the remote operation room RC. For example, the indoor camera device C2 is a camera installed inside the remote operation room RC and captures the operator OP sitting on the driver's seat DS.

[0110] The display device RD is a device that displays information related to the situation around the excavator 100. For example, the display device RD is a multi-display composed of a total of nine displays arranged in three vertical rows and three horizontal columns, and is configured to be able to display the states of the spaces in front of, on the left side, and on the right side of the excavator 100. Alternatively, the display device RD can be a head-mounted display wearable by the operator.

[0111] The communication device T3 is configured to enable communication between the communication device T1 of the excavator 100, the communication device T2 of the management device 200, the external space recognition device 300, and the like.

[0112] Moreover, the remote operation room RC has a structure substantially the same as that of the driver's seat in the cab 10 installed in the excavator 100 around the driver's seat DS. Specifically, a left control box is arranged on the left side of the driver's seat DS, a right control box is arranged on the right side of the driver's seat DS, a left operating lever is arranged at the front end of the upper surface of the left control box, and a right operating lever is arranged at the front end of the upper surface of the right control box. Also, a travel lever and a travel pedal are arranged in front of the driver's seat DS. The left operating lever, the right operating lever, the travel lever, and the travel pedal constitute the operating device 21R of the remote operation room.

[0113] An operation sensor 29R for detecting the operation content of the operation device 21R is provided in the operation device 21R. As the operation sensor 29R, for example, an inclination sensor that detects the inclination angle of the operating lever, an angle sensor that detects the swing angle of the operating lever around the swing axis, etc. can be cited, and the operation sensor 29R can also be composed of other sensors such as a pressure sensor, a current sensor, a voltage sensor, or a distance sensor. The operation sensor 29R outputs the detected information related to the operation content of the operation device 21R to the remote controller 30R. The remote controller 30R generates an operation signal based on the received information and sends the generated operation signal to the excavator 100.

[0114] As Figure 8 shown, the remote controller 30R has an operator state determination unit 61, an image synthesis unit 62, and an operation signal generation unit 63 as functional blocks. For ease of explanation, the operator state determination unit 61, the image synthesis unit 62, and the operation signal generation unit 63 are shown separately, but physical separation is not required, and they can also be composed of general software components or hardware components as a whole or in part.

[0115] The operator state determination unit 61 is configured to determine the state of the operator located in the remote operation room RC. The operator state determination unit 61 determines the position and line-of-sight direction of the operator's eyes (operator's viewpoint) based on the shooting information of the indoor imaging device C2. Specifically, the operator state determination unit 61 performs appropriate image processing on the image captured by the indoor imaging device C2, and determines the position of the operator's viewpoint and the coordinates of the line-of-sight direction in the operation room coordinate system.

[0116] The operator state determination unit 61 may also derive the position of the operator's viewpoint and the line-of-sight direction of the operator OP based on the output of other devices other than the indoor imaging device C2, such as LiDAR installed in the remote operation room RC, or an inertial measurement device installed in a head-mounted display as the display device RD. In addition, the inertial measurement device may include a positioning device. Moreover, the operator state determination unit 61 transmits information related to the position of the operator's viewpoint E1 and the line-of-sight direction of the operator OP to the excavator 100 via the communication device T3.

[0117] The image synthesis unit 62 is configured to generate a composite image by synthesizing the detection information (shooting information) of the space recognition device 70 received from the excavator 100 or the three-dimensional virtual space model 50 with other images. The space recognition device 70 of the excavator 100 captures the shooting information of each sensor by converting the position and line-of-sight direction of the operator's viewpoint in the operation room coordinate system transmitted from the remote controller 30R into coordinates in the excavator coordinate system, and transmits it to the remote controller 30R.

[0118] Moreover, the other image may be an image generated based on design data, that is, a design surface image. In the present embodiment, the image synthesis unit 62 superimposes and displays a graphic such as a computer graphic representing the position of the design surface as a design surface image on the environment information according to the design data prestored in the memory of the remote controller 30R. The image synthesis unit 62 determines the position where the design surface image should be superimposed and displayed according to the position and orientation of the excavator 100 determined by the excavator state determination unit 32 of the controller 30.

[0119] The operation signal generation unit 63 is configured to generate an operation signal to be transmitted to the excavator 100. The operation signal generation unit 63 generates an operation signal based on the output of the operation sensor 29R in the remote operation room RC. The excavator 100 basically receives the operation signal generated when the operator in the remote operation room RC operates while observing the image of the display device RD, and performs an action corresponding to the operation signal.

[0120] In addition, the remote operation of the excavator 100 is not limited to the operation performed by the operator in the remote operation room RC. For example, as Figure 7As shown, it can also be implemented through the application of the mobile terminal 400. At this time, it can be set to automatically control most of the operations of the excavator 100 in advance to simplify the operations on the mobile terminal 400.

[0121] In the construction support system SYS2 of the excavator 100 to which the remote operation room RC is applied, the trajectory of construction support can be generated in the controller 30 of the excavator 100 or in the remote controller 30R of the remote operation room RC. Therefore, the remote controller 30R can be provided with a virtual space generation unit 65, a determination unit 66, an operation prediction unit 67, an operation intervention unit 68, and an action simulator 69 having the same functions as the virtual space generation unit 31, the determination unit 36, the operation prediction unit 37, the operation intervention unit 38, and the action simulator 39 of the controller 30. Alternatively, the construction support system SYS2 can also generate the trajectory of construction support through the management device 200.

[0122] When performing construction support during remote operation, the remote controller 30R, like the controller 30, generates a plurality of trajectories for each action of the divided operation, and selects the best trajectory by evaluating each trajectory. Moreover, the remote controller 30R transmits the information of the trajectory of each action to the excavator 100, and the actuator drive unit 34 of the controller 30 controls the action of the excavator 100 along the trajectory of each action.

[0123] The construction support system SYS2 according to the second embodiment is basically configured as shown above. Hereinafter, reference will be made to Figure 9 to describe its operation. Figure 9 It is an explanatory diagram showing the operation of the excavator 100 controlled by the construction support system SYS2 according to the second embodiment. In addition, hereinafter, an example of automatically controlling the excavator 100 (equipment control function) by the remote controller 30R will be described. The construction support system SYS2 is not limited to this, and can also operate as an equipment guidance function for guiding the trajectory generated (simulated) by the remote controller 30R to the operator.

[0124] When the remote controller 30R operates the operation device 21R while the operator observes the shooting information taken by the space recognition device 70 of the excavator 100, it automatically controls the operation of the excavator 100. The virtual space generation unit 65 generates a three-dimensional virtual space model 50 based on the detection information of the environment detection device 79 (the detection information of the space recognition device 70, the orientation detection device 71, the positioning device 72, and various sensors, or the information of the operation device 21, the communication information from the external space recognition device 300 or the dump truck) (step S11).

[0125] Then, the motion simulator 69 simulates the operation of the virtual excavator 51 in the three-dimensional virtual space model 50 (step S12). As a result, the remote controller 30R can obtain the continuous excavation trajectory, ascending swing trajectory, soil discharge trajectory, and descending swing trajectory (simulation motion information) on the operation time axis.

[0126] The remote controller 30R sends the operation instructions of the excavation trajectory, ascending swing trajectory, soil discharge trajectory, and descending swing trajectory generated in the motion simulator 69 to the controller 30 (step S13). Moreover, the actuator drive unit 34 of the controller 30 controls various solenoid valves 41, various actuators 42, etc. corresponding to these trajectories in the excavator 100 in the real space (step S14). In addition, the actuator drive unit 34 can also correct the actions (position, speed, acceleration, etc.) of the excavation attachment AT by feeding back the detection information of various sensors of the excavator 100 to the actuator drive unit 34 during each action of the operation.

[0127] During actual operation, the controller 30 acquires the detection information of the environment detection device 79 or the communication information from the dump truck and sends it to the remote controller 30R. The virtual space generation unit 65 of the remote controller 30R updates the terrain information, object information, and additional information of the three-dimensional virtual space model 50 based on this information (step S15). As a result, the motion simulator 69 corrects various trajectories of the operation (excavation trajectory, ascending swing trajectory, soil discharge trajectory, descending swing trajectory) according to the updated information (step S16). The corrected trajectory is sent from the remote controller 30R to the controller 30 and reflected in the control performed by the actuator drive unit 34 of the controller 30 (step S17).

[0128] As described above, even when the remote controller 30R is provided, the construction support system SYS2 can make the excavator 100 accurately operate according to the real-time changing situation at the operation site in the real space by repeating the above actions during the operation. As a result, the construction support system SYS2 can make the excavator 100 operate efficiently and precisely.

[0129] The construction support system SYS2 according to the second embodiment is not limited to the above structure, and various modifications can be adopted. Hereinafter, with reference to Figure 10 the operation of the construction support system SYS2 according to the modification will be described. Figure 10It is an explanatory diagram showing the operation of controlling the excavator 100 based on the construction support system SYS2 according to the modified example. In the construction support system SYS2 according to the modified example, while observing the three-dimensional virtual space model 50 displayed on the display device RD in the remote operation room RC, the operator operates the operation device 21R, and thus the operation signal of the operation sensor 29R is sent from the remote controller 30R to the controller 30.

[0130] At this time, the virtual space generation unit 65 of the remote controller 30R reproduces the environmental information (topographic information, object information) in the three-dimensional virtual space model 50 based on the detection information of the environmental detection device 79 sent from the controller 30. Based on the topographic information or object information of the three-dimensional virtual space model 50, the operator can operate the excavation action, the raising and slewing action, the soil discharge action, and the lowering and slewing action of the operation. At this time, the motion simulator 69 can generate various trajectories of the operation and guide the operator along the trajectory. And when the operation of the operator deviates significantly from the generated trajectory, the operation intervention unit 68 can intervene in the operation to correct the operation signal of the operation sensor 29R.

[0131] Thus, based on the operation signal received from the remote controller 30R, the controller 30 can control the operation of the excavator 100 in the actual space. At this time, based on the detection information detected by various sensors of the excavator 100, the controller 30 performs feedback control of each action.

[0132] Moreover, during the operation, the controller 30 sends the detection information of the environmental detection device 79, the communication information of the dump truck, etc. to the remote controller 30R. Thus, the virtual space generation unit 65 of the remote controller 30R can update the environmental information of the three-dimensional virtual space model 50 during the operation for the generation of the trajectory of the motion simulator 69 or the operation of the operator.

[0133] The construction support systems SYS1 and SYS2 of the excavator 100 according to the embodiments disclosed this time are illustrative in all aspects and are not restrictive. The embodiments can be deformed and improved in various ways without departing from the scope and gist of the technical solution. The matters described in the above-mentioned multiple embodiments can adopt other structures within the non-contradictory scope and can be combined within the non-contradictory scope.

[0134] This application claims the priority of the basic application No. 2022-203642 filed with the Japan Patent Office on December 20, 2022, the entire content of which is incorporated herein by reference.

[0135] Reference Signs

[0136] 21, 21R - Operating device, 30 - Controller, 30R - Remote controller, 50 - Three - dimensional virtual space model, 51 - Virtual excavator, 70 - Space recognition device, 79 - Environment detection device, 100 - Excavator, 200 - Management device, 300 - External - side space recognition device, RC - Remote operation room, SYS1, SYS2 - Construction support system.

Claims

1. An earthmover construction support system, comprising: An earthmover; An environment detection device that detects environmental information of the work site of the earthmover; and A simulation device that acquires the environmental information detected by the environment detection device during the operation of the earthmover and generates a three-dimensional virtual space model of the work site.

2. The earthmover construction support system according to claim 1, wherein The environment detection device acquires the environmental information at a specified time during the operation of the earthmover.

3. The earthmover construction support system according to claim 1, wherein The environmental information includes operation information acquired during the operation of the earthmover.

4. The earthmover construction support system according to claim 3, wherein The operation information includes excavation operation information acquired along with the excavation operation of the earthmover.

5. The earthmover construction support system according to claim 4, wherein The environmental information includes sand information inferred from the excavation operation information.

6. The earthmover construction support system according to any one of claims 1 to 5, wherein The simulation device provides simulated operation information simulated by a virtual earthmover in the three-dimensional virtual space model reproduced in the simulation device to the earthmover.

7. The earthmover construction support system according to claim 6, wherein Based on the simulated operation information, the earthmover is controlled at the work site.

8. The earthmover construction support system according to claim 6, wherein The simulation device generates a plurality of trajectories for changing conditions for the excavation operation or the sand discharging operation of the virtual earthmover, and based on the results of simulating the plurality of trajectories, provides the best trajectory among the plurality of trajectories as the simulated operation information.

9. The earthmover construction support system according to claim 6, wherein When the simulation device identifies that there are people around the earthmover based on the environmental information, the simulation device generates the simulated operation information for avoiding the people.

10. The earthmover construction support system according to any one of claims 1 to 5, further comprising a remote operation room arranged at a position far from the earthmover, wherein The simulation device simulates the operation of a virtual earthmover in the three-dimensional virtual space model reproduced in the simulation device according to an operation instruction of an operation device in the remote operation room.

11. The earthmover construction support system according to any one of claims 1 to 5, wherein The simulation device is arranged in a controller that controls the earthmover or a management device that can communicate with the earthmover outside the earthmover.

12. The earthmover construction support system according to any one of claims 1 to 5, wherein The environmental information is stored in the simulation device before the operation of the earthmover, The simulation device generates the three-dimensional virtual space model based on the stored environmental information before the operation of the earthmover.

13. The earthmover construction support system according to any one of claims 1 to 5, wherein The simulation device extracts an object included in the environmental information and the position of the object as information, and configures the extracted object information in the three-dimensional virtual space model.

14. The construction support system for an excavator according to claim 13, wherein the simulation device adds additional information corresponding to the object information configured in the three-dimensional virtual space model to the object information.

15. The construction support system for an excavator according to claim 14, wherein the object information includes at least one of information on the shape or position of the excavator, information on the shape or position of the excavation target excavated by the excavator, and information on the shape or position of the discharge target at the target position where the excavation target is discharged.

16. The construction support system for an excavator according to claim 15, wherein the additional information corresponding to the information on the shape or position of the excavator includes at least one of the posture of the excavator, the identification number, the type, the working time, and the state of the attachment device.

17. The construction support system for an excavator according to claim 15, wherein the additional information corresponding to the information on the shape or position of the excavation target includes at least one of the stacking amount, the weight, the density, the hardness, and the soil quality of the excavation target.

18. The construction support system for an excavator according to claim 15, wherein the additional information corresponding to the information on the shape or position of the discharge target includes at least one of the posture of the discharge target, the identification number, the type, the size of the cargo box, the state of the cargo box, the working time, and the weight, volume, and density of the discharge target.

19. The construction support system for an excavator according to any one of claims 1 to 5, wherein the environmental detection device includes a space recognition device that recognizes the terrain information of the work site and the object information in the work site, the space recognition device is arranged on the excavator or at a position away from the excavator.

Images