Image Generation System

Abstract

To provide an image generation system configured to improve visibility of an imaging target.SOLUTION: A hydraulic shovel 100 includes a camera 20 for imaging an imaging target existing in a work site. An image generation system includes: a display device 50; a recording unit 270 for recording an image captured by the camera 20; and an image processing unit 26 which processes the captured image. An image acquisition unit 261 acquires a latest image which is a latest captured image. A point-of-view conversion unit 264 generates a conversion image by converting a past captured image recorded in the recording unit 270 so as to be adjusted to conform to an orientation of the camera 20 imaging the latest image. An image output unit 67 causes the display unit 50 to display a compensated image obtained by compensating a blind area in the latest image where the imaging target is covered by a blocking object, with the conversion image.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to an image generation system that generates an image to be displayed on a display device on a remote control side for remotely operating a work machine. [Background technology]

[0002] Japanese Patent Application Laid-Open Publication No. 2018-207244 (Patent Document 1) discloses a remote control system that is mounted on a work machine having a rotating body that rotates around a rotating axis and a work implement supported on the rotating body, and that includes an imaging device that captures images, a display device that is located outside the work machine, and a control device that is located outside the work machine and is capable of communicating with the work machine, wherein the control device has an image data acquisition unit that acquires images from the imaging device and a display control unit that displays the images acquired by the image data acquisition unit on the display device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-207244 Summary of the Invention [Problem to be solved by the invention]

[0004] When an imaging device is mounted on the body of a work machine, structures that make up the body may be within the field of view of the imaging device. These structures may appear in the captured image, reducing the visibility of the subject.

[0005] The present disclosure proposes an image generation system that can improve the visibility of an image capture target. [Means for solving the problem]

[0006] In accordance with the present disclosure, an image generation system is proposed that generates an image to be displayed on a display device on a remote control side for remotely operating a work machine. The work machine has an imaging device that captures an image of an object to be captured that is present at a work site. The image generation system includes a display device, an image recording unit that records images captured by the imaging device, and an image processing computer that processes the captured image. The image processing computer acquires a latest image that is the most recent captured image. The image processing computer generates a converted image by converting a previous captured image recorded in the image recording unit to match the orientation of the imaging device when the imaging device captured the latest image. The image processing computer displays, on the display device, a complemented image in which blind spots in the latest image where the object to be captured is blocked by an obstruction are complemented with the converted image. [Effects of the Invention]

[0007] According to the image generation system according to the present disclosure, it is possible to improve the visibility of the imaging target. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is an external view of a hydraulic excavator. [Figure 2] FIG. 1 is a diagram schematically illustrating an example of a remote control system for a hydraulic excavator. [Figure 3] FIG. 1 is a diagram schematically illustrating an example of a remote control device. [Figure 4] FIG. 1 is a functional block diagram showing an example of the configuration of an image generation system. [Figure 5] 1 is a flowchart illustrating an example of an image generation method. [Figure 6] FIG. 10 is a diagram showing an example of a latest image. [Figure 7] FIG. 7 is an enlarged view of region VII in FIG. [Figure 8] FIG. 10 is a diagram showing an example of a previous image. [Figure 9] FIG. 10 is a diagram showing an image in which the viewpoint is transformed. [Figure 10] FIG. 10 is a diagram illustrating a process of combining images. [Figure 11]FIG. 10 is a diagram illustrating an example of a composite image. [Figure 12] FIG. 10 is a diagram in which a plurality of past images and the latest image are combined. [Figure 13] FIG. 10 is a functional block diagram showing an example of the configuration of an image generation system according to a second embodiment. [Figure 14] FIG. 10 is a functional block diagram showing an example of the configuration of an image generation system according to a third embodiment. [Figure 15] 10 is a flowchart showing an example of an image generating method according to the third embodiment. [Figure 16] FIG. 10 is a schematic diagram showing an image capturing situation by a camera according to a fourth embodiment. [Figure 17] FIG. 13 is a diagram showing an example of a captured image according to the fourth embodiment. [Figure 18] FIG. 13 is a diagram showing an example of a composite image according to the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment will be described with reference to the drawings. In the following description, the same components are denoted by the same reference numerals. The names and functions of the components are also the same. Therefore, detailed description thereof will not be repeated.

[0010] [First embodiment] Fig. 1 is an external view of a hydraulic excavator 100 as an example of a work machine remotely controlled by a remote control system based on an embodiment. As shown in Fig. 1, in this example, the hydraulic excavator 100 will be mainly used as an example of the work machine.

[0011] The hydraulic excavator 100 is present at a work site and performs work at the work site. The hydraulic excavator 100 performs excavation work of objects such as earth and sand ore, and performs loading work of loading the excavated objects onto a transport machine such as a dump truck. The hydraulic excavator 100 of the embodiment is a work machine that can be remotely controlled. The hydraulic excavator 100 is controlled by radio signals from a remote location using a remote control system. The hydraulic excavator 100 is not equipped with a control function by an operator on board.

[0012] The hydraulic excavator 100 includes a main body 1. The main body 1 includes a revolving body 3 and a traveling body 5.

[0013] The traveling body 5 has a pair of crawler tracks 5Cr and a traveling motor 5M. The hydraulic excavator 100 is capable of traveling by rotation of the crawler tracks 5Cr. The traveling motor 5M is provided as a drive source for the traveling body 5. The traveling motor 5M is a hydraulic motor that is hydraulically operated. Note that the traveling body 5 may have wheels (tires).

[0014] The rotating body 3 is disposed on the running body 5 and is supported by the running body 5. The rotating body 3 is mounted on the running body 5 so as to be rotatable relative to the running body 5 about a rotation axis RX. The rotating body 3 has a vehicle compartment 4. The front of the vehicle compartment 4 is covered with a transparent window panel 4W. The window panel 4W is made of tempered glass or the like. A space 4S is formed within the vehicle compartment 4.

[0015] The passenger compartment 4 has multiple pillars that support the roof portion. The multiple pillars include a left front pillar 4FP1 and a right front pillar 4FP2. The left front pillar 4FP1 is located at the left front corner of the passenger compartment 4. The right front pillar 4FP2 is located at the right front corner of the passenger compartment 4. A window panel 4W is located between the left front pillar 4FP1 and the right front pillar 4FP2. The left edge of the window panel 4W is attached to the left front pillar 4FP1, and the right edge of the window panel 4W is attached to the right front pillar 4FP2.

[0016] The revolving unit 3 has an engine room 9 in which an engine is housed, and a counterweight provided at the rear of the revolving unit 3. In the engine room 9, an engine 31, a hydraulic pump 32, and the like, which will be described later, are arranged.

[0017] A handrail 19 is provided in front of the engine room 9 on the rotating unit 3. An antenna 21 is provided on the handrail 19. The antenna 21 is, for example, an antenna for GNSS (Global Navigation Satellite Systems). The antenna 21 has a first antenna 21A and a second antenna 21B provided on the rotating unit 3 so as to be spaced apart from each other in the vehicle width direction.

[0018] The hydraulic excavator 100 is equipped with a hydraulically operated work implement 2. The work implement 2 is supported on a revolving unit 3. The work implement 2 has a boom 6, an arm 7, and a bucket 8. The base end of the boom 6 is rotatably connected to the revolving unit 3 via a boom foot pin 13. The base end of the arm 7 is rotatably connected to the tip of the boom 6 via a boom tip pin 14. The bucket 8 is rotatably connected to the tip of the arm 7 via an arm tip pin 15. The arm 7 and the bucket 8 are each movable members that can move on the tip side of the boom 6.

[0019] The bucket 8 has multiple blades. The bucket 8 does not have to have blades. The tip of the bucket 8 may be formed from a straight steel plate. The bucket 8 is an example of an attachment that can be attached to the tip of the work implement 2. Depending on the type of work, the attachment can be changed to a breaker, grapple, lifting magnet, or the like.

[0020] In this embodiment, the positional relationship of each part of the hydraulic excavator 100 will be described with reference to the work implement 2.

[0021] The boom 6 of the work implement 2 rotates around a boom foot pin 13 relative to the revolving unit 3. A specific portion of the boom 6 that rotates relative to the revolving unit 3, for example, the tip of the boom 6, moves along an arc-shaped trajectory, and a plane that includes this arc is identified. When the hydraulic excavator 100 is viewed from above, this plane is represented as a straight line. The direction in which this straight line extends is the fore-and-aft direction of the main body 1 of the hydraulic excavator 100 or the fore-and-aft direction of the revolving unit 3, and will hereinafter also be referred to simply as the fore-and-aft direction. The left-and-right direction (vehicle width direction) of the main body 1 of the hydraulic excavator 100 or the left-and-right direction of the revolving unit 3 is a direction perpendicular to the fore-and-aft direction in a plan view, and will hereinafter also be referred to simply as the left-and-right direction. The up-and-down direction of the vehicle main body or the up-and-down direction of the revolving unit 3 is a direction perpendicular to the plane defined by the fore-and-aft direction and the left-and-right directions, and will hereinafter also be referred to simply as the up-and-down direction.

[0022] In the front-to-rear direction, the side where the work implement 2 protrudes from the main body 1 of the hydraulic excavator 100 is the front direction, and the direction opposite to the front direction is the rear direction. Looking from the front direction, the right and left sides in the left-right direction are the right direction and the left direction, respectively. In the up-down direction, the side with the ground is the bottom side, and the side with the sky is the top side.

[0023] The work implement 2 has a boom cylinder 10, an arm cylinder 11, and a bucket cylinder 12. The boom cylinder 10 drives the boom 6. The arm cylinder 11 drives the arm 7. The bucket cylinder 12 drives the bucket 8. Each of the boom cylinder 10, the arm cylinder 11, and the bucket cylinder 12 is a hydraulic cylinder driven by hydraulic oil.

[0024] The hydraulic excavator 100 is equipped with a camera 20. The camera 20 is an imaging device that captures an image of an object that is present at a work site around the hydraulic excavator 100 and acquires an image of the object. The camera 20 is mounted on the revolving body 3.

[0025] The imaging targets imaged by the camera 20 include construction targets to be constructed at the work site. The construction targets include excavation targets to be excavated by the work implement 2 of the hydraulic excavator 100. The excavation targets include the excavation target before excavation (i.e., the current topography), the excavation target during excavation, and the excavation target after excavation. The imaging targets imaged by the camera 20 include obstacles around the hydraulic excavator 100.

[0026] The object imaged by the camera 20 includes at least a part of the hydraulic excavator 100. The object imaged by the camera 20 includes at least a part of the work implement 2. The object imaged by the camera 20 includes other work machines arranged around the hydraulic excavator 100. The other work machines include transport machines that transport the object to be excavated by the hydraulic excavator 100. The other work machines include dump trucks.

[0027] The camera 20 has an optical system and an image sensor that receives light that has passed through the optical system. The image sensor includes a CCD (Couple Charged Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0028] The cameras 20 include a right front camera 20A, a right side camera 20B, a rear camera 20C, a left side camera 20D, and a front camera 20E.

[0029] The right front camera 20A and the right side camera 20B are arranged on the right edge of the upper surface of the revolving unit 3. The right front camera 20A is arranged further forward than the right side camera 20B. The right front camera 20A and the right side camera 20B are arranged side by side in the front and rear direction near the center of the revolving unit 3 in the fore-and-aft direction. The right front camera 20A captures an image of the area to the right front of the revolving unit 3. The optical axis of the right front camera 20A extends diagonally forward and to the right from the right front camera 20A. The right side camera 20B captures an image of the area to the right rear of the revolving unit 3. The optical axis of the right side camera 20B extends diagonally rear and to the right from the right side camera 20B.

[0030] The rear camera 20C is disposed at the rear end of the revolving unit 3 in the front-to-rear direction, and at the center of the revolving unit 3 in the left-to-right direction. A counterweight is installed at the rear end of the revolving unit 3 to balance the vehicle body during mining, etc. The rear camera 20C is disposed on the upper surface of the counterweight. The rear camera 20C captures images behind the revolving unit 3. The optical axis of the rear camera 20C extends rearward from the rear camera 20C. The left side camera 20D is disposed at the left edge of the upper surface of the revolving unit 3. The left side camera 20D is disposed near the center of the revolving unit 3 in the front-to-rear direction. The left side camera 20D captures images to the left of the revolving unit 3. The optical axis of the left side camera 20D extends leftward from the left side camera 20D.

[0031] The front camera 20E is disposed at the front of the revolving unit 3. The front camera 20E is disposed on the left side of the work implement 2. The front camera 20E is disposed inside the vehicle cabin 4. The front camera 20E is housed in the vehicle cabin 4. The front camera 20E is disposed in a space 4S within the vehicle cabin 4. The front camera 20E is disposed at a position that would be the eye position of the operator if the operator were to board the vehicle cabin 4. The front camera 20E captures an image in front of the revolving unit 3 through the window panel 4W. The optical axis of the front camera 20E extends forward from the front camera 20E. The image captured by the front camera 20E may include the terrain in front of the revolving unit 3 and at least a portion of the work implement 2.

[0032] The hydraulic excavator 100 is equipped with a communication device 22 and a vehicle body controller 26. The communication device 22 includes a communication antenna. The communication antenna is disposed, for example, above the engine room 9. The communication device 22 receives control signals transmitted from a remote location. The vehicle body controller 26 controls the engine 31, the work implement 2, the rotating bed 3, etc. based on the received control signals. The communication device 22 also transmits signals including information about the hydraulic excavator 100 to the remote location. The communication device 22 transmits, to the remote location, images of the object captured by the camera 20, position information and attitude information of the hydraulic excavator 100, etc.

[0033] 2 is a diagram schematically illustrating an example of a remote control system 200 for the hydraulic excavator 100. The remote control system 200 exists outside the hydraulic excavator 100. The hydraulic excavator 100 is remotely controlled by the remote control system 200. The remote control system 200 is provided, for example, in a remote control facility. The remote control facility may be installed at the work site where the hydraulic excavator 100 is located, or may be installed in a remote location away from the work site.

[0034] The remote operation system 200 mainly comprises a remote operation device 40, a display device 50, a remote controller 60, and a communication device 72. The remote operation device 40, the display device 50, the remote controller 60, and the communication device 72 are each provided separately from the hydraulic excavator 100.

[0035] The hydraulic excavator 100 is connected to the remote operation system 200 via a network 400. The network 400 includes at least one of the Internet, a local area network (LAN), a mobile phone communication network, and a satellite communication network. The network 400 may include a relay station that relays communicated data.

[0036] The communication device 22 of the hydraulic excavator 100 transmits a signal including information about the hydraulic excavator 100 to the remote operation system 200 via the network 400. The remote operation system 200 processes the image of the object captured by the camera 20, which is received by the communication device 72, in the remote controller 60 and displays the image on the display device 50.

[0037] FIG. 3 is a diagram schematically illustrating an example of a remote control device 40. The remote control device 40 and display device 50 shown in FIG. 3 are arranged in a remote control room provided in a remote control facility. The remote control device 40 is operated by an operator seated in an operator seat 45. The operator sits in the operator seat 45 so as to directly face the display screen of the display device 50. The operator visually confirms the status of the work site via the display device 50. The operator operates the remote control device 40 while looking at the display screen of the display device 50.

[0038] The remote control device 40 includes a left operating lever 41 and a right operating lever 42 that are operated to operate the work implement 2 and the rotating body 3, and a left traveling lever 43 and a right traveling lever 44 that are operated to operate the traveling body 5.

[0039] The left working lever 41 is located to the left of the operator's seat 45. An operator seated on the operator's seat 45 grips the left working lever 41 with his left hand and operates the left working lever 41. The left working lever 41 operates the arm 7 and the rotating body 3. The left working lever 41 receives input from the operator regarding the rotation direction of the rotating body 3 and the up and down movement of the arm 7. Operation of the left working lever 41 in the forward and backward directions corresponds to the rotation of the rotating body 3, and right and left rotation operations of the rotating body 3 are executed in response to the forward and backward operation. Operation of the left working lever 41 in the left and right directions corresponds to operation of the arm 7, and upward and downward movement of the arm 7 is executed in response to the left and right operation.

[0040] The right working lever 42 is located to the right of the operator's seat 45. An operator seated on the operator's seat 45 grips the right working lever 42 with his right hand and operates the right working lever 42. The boom 6 and bucket 8 are operated by the right working lever 42. The right working lever 42 receives input from the operator regarding the up and down movement of the boom 6 and the up and down movement of the bucket 8. Operation of the right working lever 42 in the forward and backward directions corresponds to operation of the boom 6, and the boom 6 is raised and lowered in response to the forward and backward operation. Operation of the right working lever 42 in the left and right directions corresponds to operation of the bucket 8, and the bucket 8 is moved downward and upward in response to the left and right operation.

[0041] A left console 48 is also disposed to the left of the operator's seat 45. A right console 49 is also disposed to the right of the operator's seat 45.

[0042] The left travel lever 43 and the right travel lever 44 are located in front of the operator seat 45. The left travel lever 43 and the right travel lever 44 are located side by side, with the left travel lever 43 on the left side and the right travel lever 44 on the right side. The left travel lever 43 and the right travel lever 44 receive input from the operator regarding the travel of the running unit 5. The left track 5Cr of the running unit 5 moves forward or backward in response to forward or backward operation of the left travel lever 43. The right track 5Cr of the running unit 5 moves forward or backward in response to forward or backward operation of the right travel lever 44.

[0043] The display device 50 is disposed in front of the operator's seat 45. The display device 50 is disposed farther from the operator's seat 45 than the left travel lever 43 and the right travel lever 44. The display device 50 includes a flat panel display such as a liquid crystal display or an organic EL display. The display device 50 displays images. The display device 50 is capable of displaying captured images taken by the camera 20 and acquired via the network 400. The display device 50 is also capable of displaying images after the captured images have been processed by the remote controller 60. Specifically, the display device 50 is capable of displaying an image in which blind spots have been compensated for by combining multiple captured images taken by the camera 20.

[0044] 3 shows an example in which one display screen constitutes display device 50, but display device 50 may include multiple display screens. Display device 50 may include a total of five display screens, with one display screen arranged adjacent to each of the top, bottom, left, and right of a central display screen, or a total of nine display screens arranged adjacent to each other in three columns and three rows, or may include any number of display screens arranged in any other manner. Display device 50 may include a display screen arranged in front of pilot seat 45, as well as display screens arranged to the left of pilot seat 45, to the right of pilot seat 45, and / or above pilot seat 45.

[0045] The display device 50 may be a monitor permanently installed in the remote control room, a head-mounted display, a head-up display, a personal computer monitor, or a mobile terminal such as a tablet computer or a smartphone.

[0046] 4 is a functional block diagram showing an example of the configuration of an image generation system that generates an image to be displayed on the display device 50 on the remote control side, based on an embodiment. The remote control system 200 is installed in a remote control facility outside the hydraulic excavator 100. The remote control system 200 is connected to the hydraulic excavator 100 via a network 400.

[0047] The hydraulic excavator 100 is configured such that a hydraulic pump 32 is driven by an engine 31, and hydraulic oil discharged from the hydraulic pump 32 is supplied to various hydraulic actuators 34 via a directional control valve 33. By controlling the supply and discharge of hydraulic pressure to the hydraulic actuators 34, the operation of the work implement 2, the rotation of the rotating body 3, and the traveling operation of the traveling body 5 are controlled. The hydraulic actuators 34 include the boom cylinder 10, arm cylinder 11, bucket cylinder 12, and traveling motor 5M shown in FIG. 1, as well as a swing motor.

[0048] The engine 31 is, for example, a diesel engine. The amount of fuel injected into the engine 31 is adjusted in accordance with the output of a control signal from the vehicle body controller 26, thereby controlling the output of the engine 31. The engine 31 has a drive shaft for coupling to the hydraulic pump 32.

[0049] The hydraulic pump 32 is connected to the drive shaft of the engine 31. The rotational driving force of the engine 31 is transmitted to the hydraulic pump 32, thereby driving the hydraulic pump 32. The hydraulic pump 32 is a variable displacement hydraulic pump that has a swash plate and changes its discharge capacity by changing the tilt angle of the swash plate. The hydraulic pump 32 supplies hydraulic oil used to drive the work machine 2, to travel the traveling body 5, and to rotate the rotating body 3. The hydraulic oil discharged from the hydraulic pump 32 is reduced to a constant pressure by a pressure reducing valve and supplied to the directional control valve 33.

[0050] The directional control valve 33 is a spool-type valve that switches the direction of hydraulic oil flow by moving a rod-shaped spool. The directional control valve 33 has spools that adjust the amount of hydraulic oil supplied to the boom cylinder 10, arm cylinder 11, bucket cylinder 12, travel motor 5M, and swing motor. The amount of hydraulic oil supplied to the hydraulic actuator 34 is adjusted by each spool moving axially in accordance with the output of a control signal from the vehicle body controller 26.

[0051] The remote controller 60 has a control signal generating unit 69. The control signal generating unit 69 generates a control signal for operating the hydraulic excavator 100 based on the operation of the remote operation device 40 by the operator. The communication device 72 transmits the control signal generated by the control signal generating unit 69 to the hydraulic excavator 100 via the network 400.

[0052] 1 is input to the vehicle body controller 26. The vehicle body controller 26 has an image processing unit 260. The image processing unit 260 processes the captured image input to the vehicle body controller 26. The communication device 22 transmits the processed image to the remote operation system 200 via the network 400.

[0053] The remote controller 60 has an image output unit 67. The image output unit 67 outputs the image processed by the image processing unit 260 to the display device 50, causing the display device 50 to display the image.

[0054] Fig. 5 is a flowchart showing an example of an image generating method based on the embodiment. A method for generating an image to be displayed on the display device 50 on the remote control side for remotely operating the hydraulic excavator 100 of the embodiment will be described below with reference to Figs. 4, 5 and subsequent figures. An example of capturing an image of an object while the revolving body 3 is revolving will be described below.

[0055] 5, first, in step S1, the latest image is acquired. The image processing unit 260 has an image acquisition unit 261. The image acquisition unit 261 acquires the latest image, which is the most recent image captured by the camera 20 of the hydraulic excavator 100.

[0056] FIG. 6 is a diagram showing an example of the latest image. FIG. 6 shows an image IMG captured by the forward camera 20E. The image IMG includes a portion of the work implement 2. The image IMG includes an image IFP1 of the left front pillar 4FP1 of the vehicle interior 4 and an image IFP2 of the right front pillar 4FP2. The image IMG includes a landscape LS. Shapes such as circles, crosses, triangles, and rectangles in FIG. 6 indicate portions of a simplified landscape LS. The landscape LS is blocked by images IFP1 and IFP2. The left front pillar 4FP1 and the right front pillar 4FP2 correspond to obstructions that block the image capture target in the embodiment.

[0057] 6 indicates the direction of rotation of the rotating unit 3. In this embodiment, the rotating unit 3 rotates to the right.

[0058] Figure 7 is an enlarged view of area VII in Figure 6. Area VII is set to include an image IFP1 of the left front pillar 4FP1, which is an obstructing object, and also to include a landscape LS that is not obstructed by the obstructing object. In the example shown in Figure 7, in image IMG1, the landscape LS that is not obstructed exists on both the left and right sides of image IFP1.

[0059] In step S2, a previous image, which is an image captured before the latest image, is read out. The hydraulic excavator 100 has a recording unit 270. Previous image data 278 is recorded in the recording unit 270. The recording unit 270 records an image captured by the camera 20 and acquired by the image acquisition unit 261 as a previous image. The image processing unit 260 has a movement information acquisition unit 262. The movement information acquisition unit 262 reads out the previous image from the recording unit 270. Typically, the movement information acquisition unit 262 reads out a previous image, which is a previous image captured immediately before the latest image.

[0060] FIG. 8 is a diagram showing an example of a previous image. Image IMG2 shown in FIG. 8 is an enlarged view of a region of the previous image corresponding to region VII shown in FIG. 6. As with image IMG1 in FIG. 7, in image IMG2, unobstructed scenery LS is present on both the left and right sides of image IFP1 of left front pillar 4FP1. Because the position of left front pillar 4FP1, which is an obstruction, does not change relative to front camera 20E, the position of image IFP1 of left front pillar 4FP1 remains unchanged between images IMG1 and IMG2. However, because the images were captured while the rotating unit 3 was rotating, the scenery LS included in images IMG1 and IMG2 has changed. A portion of the scenery LS that was not visible in image IMG1 is visible in image IMG2.

[0061] Specifically, in image IMG1 of Fig. 7, the left half of the circle is obscured by image IFP1 of left front pillar 4FP1. In contrast, in image IMG2 of Fig. 8, the circle is not obscured and the entire circle is shown. Image IMG1 includes only a portion of the cross, while image IMG2 shows the entire cross.

[0062] In step S3, the optical flow is calculated. The movement information acquisition unit 262 performs image processing on the latest image and the immediately preceding image. The movement information acquisition unit 262 calculates the movement vector of pixels corresponding to the same feature points included in two temporally consecutive images, specifically, image IMG1 (FIG. 7) and image IMG2 (FIG. 8). In this embodiment, the revolving unit 3 is turning right, so the scenery LS has moved leftward in image IMG1, which is a part of the latest images, compared to image IMG2, which is a part of the immediately preceding images. The movement information acquisition unit 262 compares image IMG1 and image IMG2, and calculates the amount of movement of the scenery LS from the time the immediately preceding image was captured to the time the latest image was captured, based on the difference between image IMG1 and image IMG2.

[0063] In step S4, a converted image is generated. The image processing unit 260 has a viewpoint conversion unit 264. The viewpoint conversion unit 264 converts the previous image to match the orientation of the camera 20 when the camera 20 captured the latest image. The viewpoint conversion unit 264 shifts the previous image by the amount of movement of the scenery LS calculated in step S3, so that the scenery LS included in the previous image exactly overlaps with the scenery LS included in the latest image.

[0064] Fig. 9 is a diagram showing an image in which the viewpoint has been converted based on the optical flow. The portion of image IMG3 shown in Fig. 9 surrounded by a dashed line corresponds to image IMG2 shown in Fig. 8 shifted to the left in the figure. The circles and crosses included in image IMG3 and the circles and crosses included in image IMG1, which is part of the latest image shown in Fig. 7, are positioned so as to overlap each other.

[0065] As the rotating body 3 rotates, the front camera 20E undergoes translation and rotation. Because the front camera 20E is placed near the rotation axis RX, the change in the scenery LS between the latest image and the immediately preceding image is dominated by the rotation of the front camera 20E. This makes it possible to utilize the rotation of the front camera 20E. A transformed image can be generated by a simple transformation that is uniquely determined from the amount of rotation of the front camera 20E. In addition, the transformed image can be approximated by simply moving the image in one direction.

[0066] In step S5, a blind spot area is detected. The image processing unit 260 has an area acquisition unit 263. The area acquisition unit 263 performs image processing on the latest image and the immediately preceding image. The area acquisition unit 263 determines a blind spot area caused by the structure of the vehicle interior 4. The area acquisition unit 263 may compare an image IMG1 that is a part of the latest image with an image IMG2 that is a part of the immediately preceding image, and recognize an area where there is little change and the amount of change is below a threshold as a blind spot area where an image IFP1 of the left front pillar 4FP1 is present and where the scenery LS is blocked by the left front pillar 4FP1. The area acquisition unit 263 may detect the boundary of an object blocking the scenery LS. The area acquisition unit 263 may detect the blind spot area as a bitmap.

[0067] Because the position of the left front pillar 4FP1 does not change relative to the front camera 20E, the left front pillar 4FP1 appears in the same position in the image even when the rotating unit 3 rotates. In other words, an object that appears in the same position even when the unit rotates is an obstruction. This makes it easy to determine the blind spot area.

[0068] In step S6, masking is performed. In step S7, the images are synthesized. The image processing unit 260 has an image synthesis unit 265. The image synthesis unit 265 masks blind spot areas in the latest images so that the blind spot areas are not used in synthesizing the images. FIG. 10 is a diagram showing the image synthesis process. Image IMG41 shown in FIG. 10 corresponds to the portion of image IMG1 shown in FIG. 7 that is to the left of image IFP1 of left front pillar 4FP1. Image IMG42 corresponds to the portion of image IMG1 shown in FIG. 7 that is to the right of image IFP1 of left front pillar 4FP1.

[0069] The image synthesis unit 265 synthesizes the images so as to superimpose the most recent images, images IMG41 and IMG42, on image IMG3, which is a converted image obtained by viewpoint-converting the immediately preceding image, to generate a synthetic image. FIG. 11 is a diagram showing an example of a synthetic image. In image IMG5 shown in FIG. 11, image IMG51 is located to the left of image IFP1 of the left front pillar 4FP1, and images IMG52 and IMG52 are located to the right of IFP1. Image IMG51 is the same as image IMG41 shown in FIG. 10. Image IMG52 is the same as image IMG42 shown in FIG. 10. Image IMG53 is a portion of image IMG3 shown in FIGS. 9 and 10. Specifically, image IMG53 corresponds to the portion of image IMG3 immediately to the right of image IFP1 of the left front pillar 4FP1.

[0070] In image IMG5 shown in Fig. 11, image IMG53 exists between image IMG51 and image IMG52. In the latest image shown in Fig. 7, image IMG53 is superimposed where image IFP1 of left front pillar 4FP1 exists. In the latest image (image IMG1, Fig. 7), the blind spot area where the scenery LS is blocked by left front pillar 4FP1 is complemented by image IMG53, which is part of the converted image (image IMG3, Figs. 9 and 10) obtained by converting the immediately preceding image (image IMG2, Fig. 8).

[0071] Fig. 12 is a diagram in which a plurality of past images and the latest image are combined. In image IMG6 shown in Fig. 12, images IMG61 and IMG62 are portions of the latest image also shown in Fig. 7. Image IMG63 is a portion of a converted image obtained by converting the immediately preceding image also shown in Fig. 9. Images IMG64 and IMG65 are portions of converted images obtained by similarly converting a past image captured before the immediately preceding image based on the amount of movement of the landscape LS between the time the past image was captured and the time the latest image was captured.

[0072] Images IMG61 and IMG62 are the most recent images. Image IMG63 is a converted image obtained by converting the image captured one image before the most recent image. Image IMG64 is a converted image obtained by converting the image captured two images before the most recent image. Image IMG65 is a converted image obtained by converting the image captured three images before the most recent image. All blind spots in the most recent image are complemented with converted images obtained by converting previously captured images. Image IMG6 is generated as a composite image in which the blind spots are complemented with previous images by overlaying the converted images on the blind spots in the most recent image. Image composition unit 265 combines images IMG61 and IMG62, which are part of the most recent image, with images IMG63, IMG64, and IMG65, which are previously captured images, to generate composite image IMG6.

[0073] In step S8, the image is output. The communication device 22 transmits the image IMG6 generated by the image synthesis unit 265 to the remote controller 60 via the network 400. The image output unit 67 of the remote controller 60 outputs the image IMG6 received by the communication device 72 from the vehicle body controller 26 to the display device 50, causing the display device 50 to display the image IMG6.

[0074] After the process of step S8, the process returns, and the process of generating a composite image using the newer captured image and displaying it on the display device 50 is repeated.

[0075] As shown in FIG. 4, the image generation system of the embodiment described above includes a remote-controlled display device 50, a recording unit 270 that records captured images captured by the camera 20, and an image processing unit 260 that processes the captured images. As shown in FIGS. 5 to 7, the image acquisition unit 261 acquires the latest captured image (image IMG1). As shown in FIGS. 5, 8 to 10, the viewpoint conversion unit 264 generates a converted image (image IMG3) by converting a previous captured image (image IMG2) recorded in the recording unit 270 to match the orientation of the camera 20 when the camera 20 captured the latest image. As shown in FIGS. 5, 10 to 12, the image output unit 67 causes the display device 50 to display a complemented image (image IMG6) obtained by complementing image IFP1, which shows a blind spot where the subject is blocked by the left front pillar 4FP1 in the latest image, with the converted image.

[0076] A scene LS that is obscured by an obstruction in the latest image IMG1 is not obscured by an obstruction in the previously captured image and is included in the captured image. An image IMG3 is generated by converting a previously captured image IMG2 to match the orientation of the camera 20 when the latest image IMG1 was captured, and the latest image IMG1 and the converted image IMG3 are superimposed. At this time, an image IFP1, which is a blind spot area in the latest image IMG1, is masked so that it is not used in the superimposition of images IMG1 and IMG3. This results in a complemented image in which the blind spot area is complemented by the previously captured image.

[0077] By displaying the complementary image on the display device 50, it is possible to present the operator with an image that looks as if the left front pillar 4FP1 is made transparent. Since it is possible to provide the operator with an image in which the object blocking the image capture target has seemingly been removed, it is possible to improve the visibility of the image capture target.

[0078] The front camera 20E captures an image from a viewpoint similar to that seen when an operator is inside the vehicle compartment 4 and operating the hydraulic excavator 100. By displaying an image for remote operation on the display device 50 based on the image captured by the front camera 20E, it is possible to present a clearer and more intuitive image to the operator remotely operating the hydraulic excavator 100.

[0079] 5, 11 and 12, the image processing unit 260 generates an image IMG6, which is a composite image obtained by superimposing the converted image IMG3 on the image IFP1, which is a blind spot area in the latest image, and causes the image IMG6 to be displayed on the display device 50. By generating a composite image obtained by combining a plurality of images in the image processing unit 260 and displaying the composite image on the display device 50, it is possible to reliably provide the operator with an image in which any obstructions that may be blocking the image capture target have been eliminated, thereby improving the visibility of the image capture target.

[0080] 12, the image processing unit 260 generates converted images from multiple previously captured images. Converted images are generated until blind spots in the latest image are completely filled in, and by filling in the blind spots with these converted images, it is possible to reliably provide the operator with an image in which any obstructions blocking the imaged object appear to be eliminated, thereby improving the visibility of the imaged object.

[0081] As shown in Fig. 1, the hydraulic excavator 100 has a rotating unit 3 that rotates around a rotation axis RX. As shown in Figs. 1 and 4, a camera 20 is mounted on the rotating unit 3. By utilizing a previous image captured while the rotating unit 3 was rotating to complement a blind spot area in the latest image, it is possible to reliably provide the operator with an image in which any obstructions that may be blocking the imaged object have been removed, thereby improving the visibility of the imaged object.

[0082] As shown in FIG. 1, the rotating unit 3 has a passenger compartment 4. A front camera 20E is housed in the passenger compartment 4. As shown in FIGS. 6 and 7, an image IFP1 of a left front pillar 4FP1 is included in the image captured by the front camera 20E, and the image IFP1 obscures the landscape LS that is the subject of the image capture. By using the image generation system of the embodiment to complement the blind spot area in the latest image, it is possible to reliably provide the operator with an image in which any obstructions that may be obscuring the subject of the image capture are seemingly eliminated, thereby improving the visibility of the subject of the image capture.

[0083] 5, 8 and 9, the image processing unit 260 generates a converted image by shifting the position of the previous captured image based on the rotation angle of the revolving unit 3 from when the previous captured image was taken to when the latest image was taken. In the hydraulic excavator 100, the direction in which the revolving unit 3 rotates is specified, so the process of shifting the position of the captured image based on the rotation angle to generate a converted image is easy, and a converted image with higher accuracy can be generated.

[0084] [Second embodiment] Fig. 13 is a functional block diagram showing an example of the configuration of an image generation system according to the second embodiment. Compared to Fig. 4, the image formation system of the second embodiment shown in Fig. 13 differs in that the image processing unit 260 has a complementary image generation unit 266 instead of the image synthesis unit 265. In the first embodiment, a synthesized image was generated by superimposing a converted image obtained by converting a previously captured image onto a blind spot area in the latest image, and the synthesized image was displayed on the display device 50, but the present invention is not limited to this example.

[0085] In the image forming system of the second embodiment, the area acquisition unit 263 acquires the blind spot area in the latest image. The viewpoint conversion unit 264 converts past images, including the immediately preceding image, according to the orientation of the camera 20 when the camera 20 captured the latest image, to generate a converted image. The complementary image generation unit 266 compares the latest image with the converted image, extracts a portion of the converted image that overlaps the blind spot area, and uses the extracted portion as a complementary image. The complementary image generation unit 266 converts past images, which are previously captured images, until the blind spot area can be completely complemented, and extracts the portion that overlaps the blind spot area as a complementary image. The complementary image generation unit 266 can generate multiple complementary images.

[0086] The communication device 22 transmits the latest image and the complementary image to the remote controller 60 via the network 400. The image output unit 67 of the remote controller 60 causes the display device 50 to display the latest image, and also causes the display device 50 to display the complementary image in an area corresponding to the blind spot of the latest image.

[0087] In this way, by outputting an image of a non-blind spot area and an image that complements a blind spot area to the display device 50, an image that appears to have no blind spot area can be displayed on the display device 50, similar to the case of synthesizing the composite image described in the first embodiment. Since there is no obstruction that blocks the image capture target in the image displayed on the display device 50, the visibility of the image capture target can be improved.

[0088] The converted image does not have to be the entire previous image converted. A converted image may be generated by converting only a portion of the previous image that corresponds to a blind spot area where the subject to be imaged is obscured by an obstruction in the latest image. The portion of the previous image that corresponds to the blind spot area may be converted pixel by pixel to generate multiple converted images in pixel units. These multiple converted images in pixel units can be used as multiple complementary images in pixel units. By transmitting the multiple complementary images in pixel units to the remote controller 60 sequentially or simultaneously, and having the image output unit 67 display the multiple complementary images on the display device 50, an image in which the obstruction appears to have been eliminated can be reliably provided to the operator.

[0089] [Third embodiment] 14 is a functional block diagram showing an example of the configuration of an image generation system according to the third embodiment. In the first and second embodiments, the vehicle controller 26 mounted on the hydraulic excavator 100 has the image processing unit 260. In contrast, in the image generation system according to the third embodiment, the remote controller 60 on the remote operation side has the image processing unit 600.

[0090] The hydraulic excavator 100 is equipped with a swing angle sensor 27. The swing angle sensor 27 detects the swing angle of the swing unit 3 relative to the traveling unit 5. The swing angle sensor 27 may be, for example, a resolver provided on the rotary shaft of the swing motor, or an IMU (Inertial Measurement Unit) mounted on the swing unit 3.

[0091] 4, the vehicle body controller 26 has an image acquisition unit 261 and a turning angle acquisition unit 267. The turning angle acquisition unit 267 acquires the angle at which the turning unit 3 has turned within a predetermined time from the detection value of the turning angle sensor 27.

[0092] The remote control system 200 includes a recording unit 610. Past image data 618 is recorded in the recording unit 610. The recording unit 610 records captured images transmitted to the remote control system 200 via the network 400 as past images.

[0093] The recording unit 610 also records camera direction data 611. The camera direction data 611 indicates the orientation of the front camera 20E when the front camera 20E captures an image. For example, the camera direction data 611 may indicate the direction of the optical axis of the front camera 20E in the ITRF (International Terrestrial Reference Frame) coordinate system. The camera direction data 611 is input to the remote operation system 200 and recorded in the recording unit 610.

[0094] The recording unit 610 also stores blind spot area data 612. The blind spot area data 612 is recorded in the recording unit 610 by the following procedure. Design values ​​for the shape of the vehicle interior 4 and the mounting position of the front camera 20E in the vehicle interior 4 are input to the remote control system 200 in advance. From these design values, the areas in the captured image where structures constituting the vehicle interior 4 exist are determined in advance. For example, the positions of the image IFP1 of the left front pillar 4FP1 and the image IFP2 of the right front pillar 4FP2 in the image IMG shown in FIG. 6 are determined. The positions of the structures constituting the vehicle interior 4 in the captured image are recognized as blind spot areas and recorded in the recording unit 610. Because the positions of the left front pillar 4FP1 and the right front pillar 4FP2 do not change relative to the front camera 20E, it is easy to determine the blind spot areas.

[0095] Fig. 15 is a flowchart showing an example of an image generation method according to the third embodiment. The image generation method according to the third embodiment will be described with reference to Fig. 14 and Fig. 15. As in the first embodiment, an example will be described in which an image of an object is captured while the revolving unit 3 is revolving.

[0096] 15, first, in step S11, the latest image is acquired. The image acquisition unit 261 acquires the latest image, which is the image most recently captured by the camera 20 of the hydraulic excavator 100.

[0097] In step S12, the rotation angle is acquired. The rotation angle acquisition unit 267 acquires the rotation angle of the rotating unit 3 from the detection value of the rotation angle sensor 27. The rotation angle acquisition unit 267 acquires the rotation angle of the rotating unit 3 from the time when the image immediately before the latest image was captured to the time when the latest image was captured.

[0098] The communication device 22 transmits the captured image captured by the camera 20 and acquired by the image acquisition unit 261 and the rotation angle of the rotating body 3 acquired by the rotation angle acquisition unit 267 to the remote control system 200 via the network 400.

[0099] In step S13, a previous image, which is an image captured before the latest image, is read. The image processing unit 600 has a viewpoint conversion unit 604. The viewpoint conversion unit 604 reads a previous image from the recording unit 610. Typically, the viewpoint conversion unit 604 reads a previous image, which is an image captured immediately before the latest image.

[0100] In step S14, the camera direction is acquired. The image processing unit 600 has a direction acquisition unit 601. The direction acquisition unit 601 reads out, from the recording unit 610, camera direction data 611 indicating the direction of the camera 20 when the camera 20 captured the previous image. The direction acquisition unit 601 reads out, from the recording unit 610, data indicating the direction of the front camera 20E.

[0101] In step S15, a converted image is generated. Based on the rotation angle of the revolving unit 3 and the orientation of the front camera 20E, the viewpoint conversion unit 604 converts the previous image, including the immediately preceding image, to match the orientation of the front camera 20E when the latest image was captured by the front camera 20E. The viewpoint conversion unit 604 shifts the previous image by the rotation angle of the revolving unit 3 from the time the previous image was captured to the time the latest image was captured, so that the scenery included in the previous image exactly overlaps with the scenery included in the latest image.

[0102] In step S16, the blind spot area is acquired. The image processing unit 600 has an area acquisition unit 603. The area acquisition unit 603 reads out blind spot area data 612 from the recording unit 610. The area acquisition unit 603 acquires the positions of the blind spot areas in the image captured by the front camera 20E, typically the image IFP1 of the left front pillar 4FP1 and the image IFP2 of the right front pillar 4FP2.

[0103] In step S17, masking is performed. In step S18, the images are synthesized. The image processing unit 600 has an image synthesis unit 265. The image synthesis unit 605 masks blind spots in the latest image so that the blind spots are not used in synthesizing the images. The image synthesis unit 605 synthesizes the images by superimposing a part of the latest image and a part of the converted image. If necessary, the image synthesis unit 605 synthesizes the latest image with multiple converted images obtained by converting multiple past images, to generate a synthesized image.

[0104] In step S19, the image is output. The image output unit 67 outputs the composite image generated in step S18 to the display device 50, causing the display device 50 to display the composite image.

[0105] After the process of step S19, the process returns, and the process of generating a composite image using the newer captured image and displaying it on the display device 50 is repeated.

[0106] As with the first embodiment, the image generation system of the third embodiment can also display an image with no apparent blind spots on the display device 50. Since there is no obstruction that blocks the image capture target in the image displayed on the display device 50, the visibility of the image capture target can be improved.

[0107] The image generation system that generates the images to be displayed on the display device 50 may be mounted on the vehicle controller 26 on the work machine side, on the remote controller 60 on the remote operation side, or on another computer such as a control computer.

[0108] [Fourth embodiment] 16 is a schematic diagram showing a situation in which images are captured by a camera according to the fourth embodiment. In the fourth embodiment, an example will be described in which scenes LSA, LSB, and LSC, which are typically arranged side by side on the same plane, are captured in order by a camera 20 rotating in a rotation direction RD.

[0109] FIG. 17 is a diagram showing an example of a captured image according to the fourth embodiment. Image IMG101 includes landscapes LSA and LSB. Image IMG101 is a captured image of landscapes LSA and LSB captured obliquely. Image IMG102 includes landscape LSB and parts of landscapes LSA and LSC. Image IMG102 is a captured image of landscape LSB captured from the front. Image IMG103 includes landscapes LSB and LSC. Image IMG103 is a captured image of landscapes LSB and LSC captured obliquely.

[0110] Fig. 18 is a diagram showing an example of a composite image according to the fourth embodiment. As shown in Fig. 16, camera 20 is rotated on the spot to successively capture a plurality of images IMG101, IMG102, and IMG103 shown in Fig. 17, and the images IMG101, IMG102, and IMG103 can be combined into a single image by joining the common portions of the images IMG101, IMG102, and IMG103 so that they overlap. Image IMG111 shown in Fig. 18 is a modified version of image IMG101 shown in Fig. 17. Image IMG112 is a modified version of image IMG102.

[0111] When generating a composite image based on the most recently captured image IMG103, a more precise composite image can be generated by shifting and deforming the previously captured images IMG101 and IMG102 based on the rotation angle of the camera 20. This further improves the visibility of the imaged subject.

[0112] In the above embodiment, an example has been described in which the image IFP1 of the left front pillar 4FP1 is seemingly removed from the image displayed on the display device 50. The objects that can be removed from the image are not limited to structures that make up the vehicle interior 4, such as pillars and side mirrors. Part or all of the work implement 2 may be removed from the image. The boom cylinder 10, arm cylinder 11, and bucket cylinder 12 may be removed, and only the boom 6, arm 7, and bucket 8 that make up the main body of the work implement 2 may be displayed on the display device 50. Only the bucket 8, which is a work tool for excavation and loading that is disposed at the tip of the work implement 2, may be displayed on the display device 50, or only the cutting edge of the bucket 8 may be displayed on the display device 50.

[0113] In the embodiment, an example has been described in which an image of an object is captured while the revolving unit 3 is rotating, but the present invention is not limited to image processing of captured images captured while the revolving unit 3 is rotating. When the revolving unit 3 is not rotating relative to the running unit 5 and the work implement 2 is operating relative to the revolving unit 3, it is also possible to prevent the work implement 2 from being displayed on the display device 50. In this case, based on multiple captured images, the times at which the multiple captured images were captured, and the movement speed of the work implement 2, it is possible to remove part or all of the work implement 2 from the image displayed on the display device 50. The movement speed of the work implement 2 can be obtained from detection values ​​of stroke sensors attached to each hydraulic cylinder that drives the work implement 2, detection values ​​of rotation angle sensors attached to each pin that serves as the rotation axis of each element of the work implement 2, image analysis of captured images of the work implement 2, etc.

[0114] In the embodiment, a hydraulic excavator 100 is given as an example of a work machine, but the present invention is also applicable to other types of work machines such as a loading shovel, a mechanical rope shovel, and a bucket crane.

[0115] In the embodiment, an example has been described in which the hydraulic excavator 100 is equipped with a swing motor that is the hydraulic actuator 34, and the swing body 3 swings when hydraulic oil is supplied to the swing motor. However, the excavator is not limited to this example, and may be equipped with an electric motor. The excavator may be a hybrid excavator that is equipped with an engine that is a drive source for the operation and travel of the work implement 2, and an electric motor that drives the swing body 3 with electric energy. The excavator may be an electric excavator in which the drive sources for the rotation of the swing body 3, the travel of the traveling body 5, and the operation of the work implement 2 are all electric motors, and the electric motors are driven by electric energy stored in a battery.

[0116] Although the embodiments have been described above, configurations that can be combined with each other in each embodiment may be appropriately combined. Furthermore, the embodiments disclosed herein are illustrative in all respects and should not be considered limiting. The scope of the present invention is defined by the claims rather than the above description, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0117] 1 main body, 2 work equipment, 3 rotating body, 4 vehicle cabin, 4FP1 left front pillar, 4FP2 right front pillar, 4S space, 6 boom, 7 arm, 8 bucket, 10 boom cylinder, 11 arm cylinder, 12 bucket cylinder, 20 camera, 20A right front camera, 20B right side camera, 20C rear camera, 20D left side camera, 20E front camera, 22, 72 communication device, 26 vehicle body controller, 27 turning angle sensor, 40 remote control device, 45 operation seat, 50 display device, 60 remote controller, 67 image output unit, 69 control signal generation unit, 100 hydraulic excavator, 200 remote control system, 260, 600 image processing unit, 261 image acquisition unit, 262 movement information acquisition unit, 263, 603 area acquisition unit, 264, 604 Viewpoint conversion unit, 265,605 image synthesis unit, 266 complementary image generation unit, 267 rotation angle acquisition unit, 270,610 recording unit, 278,618 past image data, 400 network, 601 direction acquisition unit, 611 camera direction data, 612 blind spot area data, RD rotation direction, RX rotation axis.

Claims

1. An image generation system for generating an image to be displayed on a remote control display device for remotely operating a work machine, The work machine includes a rotating body that rotates around a rotation axis, and an imaging device that is mounted on the rotating body and captures an image of an object that is present at a work site, The image generation system includes: the display device; an image recording unit that records a captured image captured by the imaging device and information indicating the orientation of the imaging device when the captured image was captured; an image processing computer that processes the captured image, the image processing computer acquires a latest image, which is the most recent captured image, and generates a converted image by converting the previous captured image recorded in the image recording unit to match the orientation of the imaging device when the imaging device captured the latest image, based on information indicating the orientation of the imaging device when the latest image was captured and information indicating the orientation of the imaging device when the previous captured image was captured, and displays on the display device a complemented image in which a blind spot area in the latest image, where the imaging target is blocked by an obstruction, is complemented with the converted image.

2. The image generation system according to claim 1 , wherein the image processing computer generates a composite image by superimposing the converted image on the blind spot area, and displays the composite image on the display device.

3. The image generating system according to claim 1 , wherein the image processing computer causes the display device to display the latest image and the converted image in an area of ​​the display device that corresponds to the blind spot area.

4. The image generation system according to claim 1 , wherein the image processing computer generates the transformed image from a plurality of previous captured images.

5. The image generation system according to claim 1 , wherein the rotating body has a cabin, and the imaging device is housed in the cabin.

6. the imaging device captures an image of the imaging target while the rotating body is rotating, 6. The image generation system according to claim 5, wherein the image processing computer acquires a rotation angle of the rotating body from when the previous captured image was taken to when the latest image was taken, and generates the converted image by shifting the position of the previous captured image based on the rotation angle.

7. The image generation system according to claim 6 , wherein the image processing computer generates the transformed image by transforming the previously captured image based on the rotation angle.

8. An image generation system for generating an image to be displayed on a display device on a remote control side for remotely operating a work machine, comprising: the work machine has an imaging device that captures an image of an object to be captured that is present at a work site, The image generation system includes: the display device; an image recording unit that records an image captured by the imaging device; an image processing computer that processes the captured image, the image processing computer acquires a latest image that is the latest captured image, generates a converted image by converting the previous captured image recorded in the image recording unit in accordance with the orientation of the imaging device when the imaging device captured the latest image, and displays on the display device a complemented image in which a blind spot area in the latest image where the imaging target is blocked by an obstruction is complemented with the converted image; The work machine has a rotating body that rotates around a rotating shaft, the imaging device is mounted on the rotating body and captures an image of the imaging target while the rotating body is rotating; The image processing computer acquires the rotation angle of the rotating body from when the previous captured image was taken to when the latest image was taken, and generates the converted image by shifting and deforming the position of the previous captured image based on the rotation angle.

9. An image generation system for generating an image to be displayed on a remote control display device for remotely operating a work machine, comprising: The work machine has a rotating body that rotates around a rotation axis, and an imaging device that is mounted on the rotating body and captures an image of an object to be imaged that is present at a work site, and the orientation of the imaging device changes as the rotating body rotates, The image generation system includes: the display device; an image recording unit that records an image captured by the imaging device; an image processing computer that processes the captured image, The image processing computer acquires a latest image, which is the most recent captured image, generates a converted image by converting the previous captured image recorded in the image recording unit to match the orientation of the imaging device when the imaging device captured the latest image, compares the latest image with the previous captured image, determines that areas with little change are blind spot areas in the latest image where the imaging subject is blocked by an obstruction, and displays a complemented image in which the blind spot area is complemented with the converted image on the display device.

10. An image generation system for generating an image to be displayed on a remote control display device for remotely operating a work machine, comprising: The work machine has a rotating body that rotates around a rotation axis, and an imaging device that is mounted on the rotating body and captures an image of an object to be imaged that is present at a work site, and the orientation of the imaging device changes as the rotating body rotates, The image generation system includes: the display device; an image recording unit that records an image captured by the imaging device; an image processing computer that processes the captured image, the image processing computer acquires a latest image, which is the most recent captured image, generates a converted image by converting the previous captured image recorded in the image recording unit to match the orientation of the imaging device when the imaging device captured the latest image, compares the latest image with the converted image, and displays on the display device a complementary image obtained by extracting a portion of the converted image that overlaps a blind spot area in the latest image where the imaging target is blocked by an obstruction.

11. The image generation system described in Claim 10, wherein the image processing computer extracts the complementary image until the blind spot area can be completely complemented.

Images