Image generation device and remote operation support system for construction machine

Abstract

The invention provides an image generation device and a remote operation support system for construction machinery, which can provide an image of a construction site with a sense of scene by a remote operator of the construction machinery, thereby improving work efficiency. An image generation device generates an image that supports remote operation of a construction machine (100) provided with a work attachment device. An image generation device generates a composite image including a main image included in an image captured by an imaging device (C1) in a construction site (WS) where a construction machine (100) operates, and a complemented image that complements a region (SR2) outside an imaging range (SR1) of the main image.

Description

[0001] This application claims priority based on Japanese Patent Application No. 2023-179460 filed on October 18, 2023. The entire contents of the Japanese Patent Application are incorporated herein by reference. Technical Field

[0002] The present invention relates to an image generating device and a remote operation support system for a construction machine. Background Art

[0003] Conventionally, a remote operation system for a construction machine is known (for example, Patent Document 1).

[0004] Patent Document 1: Japanese Patent Application Publication No. 2021-114757

[0005] The remote operation system is configured to display an image of the construction site on a display device in a remote operation room disposed outside the construction machine.

[0006] However, in this configuration, it is impossible to provide an image with a sense of reality to an operator who remotely operates a construction machine in a remote operation room, and thus there is a risk of reduced work efficiency.

[0007] For example, in a 2006 report on improving the operability of unmanned construction machinery by the Hokuriku Regional Development Bureau of the Ministry of Land, Infrastructure, Transport and Tourism, insufficient visual information was pointed out as the main cause of reduced work efficiency in remote operation. Summary of the invention

[0008] Therefore, from the viewpoint of improving work efficiency, it is desired that a remote operator of a construction machine provide an image of the construction site with a sense of reality.

[0009] An image generating device involved in one embodiment of the present invention generates an image for supporting remote operation of a construction machine equipped with a working attachment, and the image generating device generates a composite image including a main image included in an image captured by a camera device at a construction site where the construction machine is working and a supplementary image that supplements the outside of the shooting range of the main image.

[0010] Another embodiment of the present invention involves a remote operation support system for construction machinery, which includes a construction machinery equipped with working attachments and a remote operation support device for supporting remote operation of the construction machinery, wherein the remote operation support device includes: the image generating device; a display device for displaying the composite image generated by the image generating device at a remote location away from the construction site; and a remote operation device for operating the construction machinery working at the construction site from the remote location.

[0011] Effects of the Invention

[0012] According to the above-described embodiment, a remote operator of a construction machine can provide an image of the construction site with a sense of reality, thereby improving construction efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a schematic diagram showing an example of a remote operation support system according to an embodiment of the present invention.

[0014] Figure 2 This is a schematic plan view showing an example of a construction site.

[0015] Figure 3 This is a diagram showing an example of a synthesized image generated by the image generating device.

[0016] Figure 4 This is a diagram for explaining an example of a learned model.

[0017] Figure 5 It is a side view of an excavator as an example of a construction machine.

[0018] Figure 6 It means that it is installed on Figure 5 A diagram showing an example of a structure of a drive system on an excavator.

[0019] Figure 7 It means that it is installed on Figure 5 A diagram showing an example of the configuration of an electric power system on an excavator.

[0020] Figure 8 This is a diagram showing the relationship between the coordinate system related to the remote operation room and the coordinate system related to the shovel.

[0021] Fig. 9 It is a diagram showing a structural example of a display device.

[0022] In the figure: 210-image generating device, 220-display device, 230-remote operating device, 250-communication device (receiving device), C1-camera device, DT-truck (object), G-synthetic image, G1-main image, G2-complementary image, LM-learned model, M-other construction machinery (object), P-cone (object), SR1-shooting range of the main image, SR2-left and right adjacent areas, SW-staff (object), SYS-remote operating support system, T1-communication device (transmitting device), WS-construction site. DETAILED DESCRIPTION

[0023] Hereinafter, a mode for carrying out the invention will be described with reference to the drawings.

[0024] Figure 1It is a schematic diagram showing an example of the remote operation support system SYS according to the embodiment of the present invention. Figure 2 It is a schematic plan view showing an example of the construction site WS. Figure 3 It means composed of Figure 1 FIG. 2 is a diagram showing an example of a synthetic image G generated by the image generating device 210 of the present embodiment of the remote operation supporting system SYS.

[0025] The details will be described later, but the image generation device 210 of the present embodiment is a device for generating images to support remote operation of a construction machine 100 equipped with a working attachment WA. The image generation device 210 generates a composite image G including a main image G1 included in an image captured by a camera device C1 at a construction site WS where the construction machine 100 is working, and a supplementary image G2 that supplements the outside of the shooting range SR1 of the main image G1. In addition, the "supplementary image" may also be referred to as an "extrapolated image." The image generation device 210 of the present embodiment and the remote operation support system SYS for construction machinery are described in detail below.

[0026] like Figure 1 As shown, the remote operation support system SYS of the present embodiment includes, for example: a construction machine 100 having a working attachment WA; and a remote operation support device 200 for supporting remote operation of the construction machine 100. The construction machine 100 included in the remote operation support system SYS may be one or more. Furthermore, when the remote operation support system SYS includes a plurality of construction machines 100, the plurality of construction machines 100 may all be of the same type, or at least some of them may be of different types.

[0027] Furthermore, the remote operation support system SYS may include multiple remote operation support devices 200. That is, multiple remote operation support devices 200 can implement processing related to the remote operation support system SYS in a distributed manner. For example, multiple remote operation support devices 200 can communicate with each other and perform processing on a portion of the construction machines 100 in charge of all the construction machines 100 included in the remote operation support system SYS.

[0028] The construction machine 100 is, for example, a forklift, a crane, or an excavator that works at the construction site WS. The construction machine 100 includes, for example, an imaging device C1, a controller 30, a detection device S, a driving device D, a communication device T1, and a working attachment WA.

[0029] The camera C1 is configured to capture the construction site WS around the construction machine 100. The camera C1 outputs the captured image of the construction site WS to the controller 30, for example. In addition, the camera C1 may not be mounted on the construction machine 100. The camera C1 may be mounted on a pole installed at the construction site WS or a drone flying over the construction site WS, for example.

[0030] The camera device C1 includes, for example, one or more cameras. Figure 2 As shown, the camera device C1 includes, for example, a front camera C1F, a rear camera C1B, a right camera C1R, and a left camera C1L. In addition, the camera device C1 may also include one or more stereo cameras and / or one or more monocular cameras.

[0031] The front camera C1F and the rear camera C1B are respectively disposed at the front and rear of the construction machine 100 to capture the front and rear of the construction machine 100. The right camera C1R and the left camera C1L are respectively disposed at the right and left sides of the construction machine 100 to capture the right and left sides of the construction machine 100. Figure 2 In FIG. 8 , the shooting range SR of each camera is indicated by a dotted line.

[0032] exist Figure 2 In the example shown, the shooting ranges SR of the front camera C1F and the rear camera C1B overlap with the shooting ranges SR of the right camera C1R and the left camera C1L, respectively. Thus, the camera device C1 is configured to be able to capture images of the construction site WS in a 360° range around the construction machine 100. In addition, the camera device C1 may also be an omnidirectional camera installed on the body of the construction machine 100 or in the cab.

[0033] The detection device S is configured to detect physical quantities related to the construction machine 100. The detection device S includes, for example, a velocity sensor, an acceleration sensor, an angular velocity sensor, an inertial measurement unit (IMU), an angle sensor, a temperature sensor, a pressure sensor, a rotation speed sensor, a fuel level sensor, a urea water level sensor, etc. In addition, the detection device S may include a receiver of a global navigation satellite system (GNSS) for detecting position information of the construction machine 100. The detection device S outputs the detection result of the physical quantity or position and posture information related to the construction machine 100 to the controller 30.

[0034] The drive device D is configured to drive the construction machine 100. The drive device D includes, for example, an engine, a transmission, a hydraulic device (a hydraulic tank, a hydraulic pump, a control valve, a hydraulic actuator, etc.), an electric motor, etc. The drive device D generates power for, for example, moving the construction machine 100, rotating the construction machine 100, and operating the attachments of the construction machine 100.

[0035] The communication device T1 can communicate with the remote operation support device 200 via a predetermined communication line NW. The communication device T1 outputs the control signal received from the remote operation support device 200 via the communication line NW to the controller 30. In addition, the communication device T1 also functions as a sending device that sends the image data of the main image G1 input from the controller 30 or various data including the position and posture information of the construction machine 100 to the remote operation support device 200 via the communication line NW.

[0036] The working attachment WA is, for example, a device used by the construction machine 100 to work at the construction site WS, and is controlled by the controller 30 according to the remote operation of the remote operator. In the case where the construction machine 100 is a forklift, the working attachment WA includes, for example, a fork, a backrest, a lifting chain, a mast, etc. In the case where the construction machine 100 is a crane, the working attachment WA includes, for example, a boom, a cantilever, a hook, a cable, etc. In the case where the construction machine 100 is an excavator, the working attachment WA is, for example, an excavation attachment including a boom, a dipper arm, a bucket, etc.

[0037] The communication line NW includes, for example, a wide area network (WAN). In the wide area network, for example, a mobile communication network with a base station as the terminal may be included. Furthermore, the wide area network may include, for example, a satellite communication network using a communication satellite above the construction machine 100. Furthermore, the wide area network may also include, for example, the Internet. Furthermore, the communication line NW may include, for example, a local area network (LAN) of a facility where the remote operation support device 200 is installed. The local area network may be a wireless line, a wired line, or a line including both. Furthermore, the communication line NW may also include, for example, a short-distance communication line based on a prescribed wireless communication method such as WiFi or Bluetooth (registered trademark).

[0038] The controller 30 is a computing device that performs various operations. In the present embodiment, the controller 30 is composed of a microcomputer including a CPU and a memory 30a. Then, various functions of the controller 30 are realized by the CPU executing a program stored in the memory 30a.

[0039] The controller 30 extracts a portion of the image captured by the camera device C1 at the construction site WS, for example, and generates a main image G1 to be sent to the remote operation support device 200. Figure 2 As shown in FIG. 1 , when the viewing angle θ of the camera constituting the imaging device C1 increases, the distortion of the peripheral portion of the image increases. Figure 2 and Figure 3As shown, the controller 30 extracts the main image G1 which is an image of the imaging range SR1 corresponding to a predetermined field angle θ1 centered on the optical axis of each camera in the imaging range SR of each camera constituting the imaging device C1.

[0040] The field angle θ1 of the shooting range SR1 of the main image G1 is, for example, a range of about 20° to 30° corresponding to the central field of view or effective field of view of a person, with the optical axis of the camera constituting the camera device C1 as the center. That is, when the field angle θ of the camera constituting the camera device C1 is larger than 20° to 30°, the controller 30 extracts the main image G1 as a part of the image from the entire image of the shooting range SR, and transmits it to the remote operation support device 200 via the communication device T1. As a result, compared with the case where the entire image of the shooting range SR is transmitted to the remote operation support device 200, image data can be reduced.

[0041] Furthermore, by extracting the field of view angle θ1 corresponding to the central field of view or effective field of view of a person from the overall image of the shooting range SR captured by the camera device C1 at the construction site WS as the main image G1 of the shooting range SR1, it is possible to send a less distorted image from the construction machine 100 to the remote operation support device 200. In addition, when the field of view angle θ of the camera device C1 is a relatively small angle of, for example, about 20° to 30°, the distortion of the overall image of the shooting range SR becomes smaller. At this time, the controller 30 can extract the overall image of the shooting range SR of the camera device C1 as the main image G1, and send it to the remote operation support device 200 via the communication device T1.

[0042] like Figure 1 As shown, the remote operation support device 200 includes an image generation device 210, a display device 220, and a remote operation device 230. The remote operation support device 200 includes, for example, a controller 240 and a communication device 250. The remote operation support device 200 can communicate with the construction machine 100 through the communication device 250 and a predetermined communication line NW.

[0043] The image generating device 210 and the controller 240 are, for example, computing devices that perform various operations. In the present embodiment, the image generating device 210 and the controller 240 are composed of a microcomputer including a CPU and a memory. In addition, various functions of the image generating device 210 and the controller 240 are realized by the CPU executing a program stored in the memory. The image generating device 210 and the controller 240 may be composed of different microcomputers or a common microcomputer.

[0044] As described above, the image generation device 210 generates an image to support remote operation of the construction machine 100 equipped with the work attachment WA. The image generation device 210 generates a composite image G including a main image G1 included in an image captured by the camera device C1 at the construction site WS where the construction machine 100 is working, and a supplementary image G2 that supplements the outside of the capture range SR1 of the main image G1. More specifically, for example, Figure 2 As shown, the supplementary image G2 is an image that supplements the left and right adjacent areas SR2 of the shooting range SR1 of the main image G1. In addition, the supplementary image G2 may be an image that supplements the upper and lower adjacent areas of the shooting range SR1 of the main image G1. The supplementary image G2 is, for example, not an image shot by the camera C1, but an image produced by the image generation device 210 and supplements the area outside the shooting range SR of the camera C1 or the shooting range SR1 of the main image G1.

[0045] The main image G1 is, for example, an image displayed in the central field of view or effective field of view of the remote operator. The supplementary image G2 is, for example, an image displayed in the peripheral field of view of the remote operator. The difference between the feature quantity of the main image G1 and the feature quantity of the supplementary image G2 is below a predetermined threshold value. Here, the predetermined threshold value is preferably set to a range in which the remote operator who observes the main image G1 of the composite image G and remotely operates the construction machine 100 cannot recognize the difference between the actual image of the area SR2 captured by the camera device C1 and the supplementary image G2. In addition, the actual image of the area SR2 may not be sent from the construction machine 100 to the remote operation support device 200.

[0046] The feature quantities of the main image G1 and the complementary image G2 include, for example, RGB values, chromaticity, brightness, saturation, etc. of each pixel or unit area of ​​each image. Moreover, the feature quantities of the main image G1 and the complementary image G2 can be derived, for example, by algorithms such as SIFT (Scale-Invariant Feature Transform) or HOG (Histograms of Oriented Gradients). Moreover, the feature quantities of the main image G1 and the complementary image G2 can be feature quantities that digitize the distribution of colors in the image, or can be local feature quantities that detect feature points with large changes in shades in the image by a feature point detection method, and convert the area around the feature points into feature vectors by pixel values ​​or differential values. The complementary image G2 includes, for example, a monochrome image of the same color as the color of the end portion of the complementary image G2 adjacent to the main image G1, an image of illustration tone, and a CG image. Moreover, the complementary image G2 can include, for example, an image in which the distortion of the image taken by the camera device C1 is corrected.

[0047] The image generation device 210, for example, includes a learned model LM that has been learned using images outside the shooting range SR1 of the main image G1 captured by the camera device C1 at the construction site WS and images of the main image G1 as training data. The image generation device 210, for example, inputs the main image G1 into the learned model LM to generate a synthetic image G. The image outside the shooting range SR1 of the main image G1 included in the training data may be an image captured by the camera device C1 in the past (one minute ago, etc.), or may be an image of another construction site similar to the construction site WS. In addition, the learned model LM may be generated by unsupervised learning.

[0048] Figure 4 This is a diagram illustrating an example of a learned model LM. As machine learning used in the generation of the learned model LM, for example, machine learning using a deep neural network (DNN), i.e., deep learning (Deep Learning) is applied. The learned model LM, for example, inputs the main image G1 to the input layer (0 layer) of the neural network, thereby outputting a synthetic image G from the output layer (n layer). Furthermore, for example, the image generating device 210 can use methods such as GAN (Generative Adversarial Networks), TF-GAN, DeepFill, Pconv, Buoundless, etc. to generate a synthetic image G including the main image G1 and the complementary image G2.

[0049] The image generation device 210 generates a training data set, for example, using a plurality of main images G1 stored in a nonvolatile storage device constituting the controller 240 during operation of the construction machine 100 and position and posture information of the construction machine 100 corresponding to each main image G1. The image generation device 210 performs machine learning using the generated training data set, for example, to generate a learned model LM.

[0050] Furthermore, the image generation device 210 generates a complementary image G2 including an image of the object based on information of the object recognized from the image captured by the camera device C1 at the construction site WS and position and posture information of the construction machine 100 working at the construction site WS. Figure 2 As shown, when the front direction of the construction machine 100 is set to 0°, the right direction is set to 90°, the rear direction is set to 180°, and the left direction is set to 270° (the angle setting method is arbitrary), it is assumed that there are workers SW, cones P, trucks DT and other construction machines M at various angles.

[0051] At this time, the image generating device 210 recognizes objects such as the worker SW, the cone P, the truck DT, and other construction machinery M from the image of the camera C1 by a known method such as pattern matching. Furthermore, the image generating device 210 stores the object information in a non-volatile storage device such as a ROM constituting the image generating device 210, by associating the object recognition result with the position and posture information of the construction machinery 100 when the object was photographed by the camera C1. Alternatively, the object information may be stored in a volatile storage device such as a RAM constituting the image generating device 210.

[0052] Furthermore, the image generation device 210 compares the current position and posture information of the construction machine 100 with the object information stored in the non-volatile storage device, and if there is an object in the area SR2 corresponding to the complementary image G2, generates an image of the object and displays it superimposed on the complementary image G2. More specifically, Figure 2 As shown, when the cone P exists in the region SR2 where the complement image G2 is generated, Figure 3 As shown, the image generating device 210 generates a complement image G2 including an image of the cone P.

[0053] Here, typically, the “image of the cone P” is an image that has nothing to do with the actual image of the cone P captured by the camera C1 (that is, an image that the image generation device 210 generates without reference to the actual image of the cone P). However, the “image of the cone P” may be an image that the image generation device 210 generates based on the actual image of the cone P captured by the camera C1 in the past. Furthermore, a keyword representing an object recognized from the image of the camera C1 may be input into the image generation device 210, thereby generating a complementary image G2 including an image of the object.

[0054] Furthermore, for example, a part of the construction machine 100, such as the working attachment WA, specifically, the forks of a forklift, the boom of a crane, or the attachments (boom, arm, end attachment) of an excavator, sometimes exists in the area SR2 of the construction site WS corresponding to the supplementary image G2. In this case, the image generation device 210 can generate a supplementary image G2 including an image of a part of the construction machine 100 based on, for example, the shape, size, and posture of the construction machine 100 working at the construction site WS, and the positional relationship between the construction machine 100 and the camera device C1. At this time, the image generation device 210 can use, for example, CAD data such as a 3D model of the construction machine 100 pre-stored in a non-volatile storage device.

[0055] The display device 220 includes a display such as a liquid crystal display, an organic EL display, a head mounted display, etc., and displays the synthetic image G input from the image generation device 210. The display device 220 provides the synthetic image G of the construction site WS to the remote operator operating the construction machine 100 via the remote operation device 230 provided at a remote location away from the construction site WS.

[0056] The remote operating device 230 is, for example, a device that simulates the driver's seat or the operating seat of the construction machine 100, and is provided with the same operating device or output device as the driver's seat or the operating seat of the construction machine 100. More specifically, the operating device of the remote operating device 230 includes, for example, an operating lever, an operating pedal, an operating control panel, an operating button, a touch panel, etc. In addition, the output device of the remote operating device 230 includes, for example, a monitor, a speaker, a buzzer, etc. The operating device of the remote operating device 230 includes, for example, a sensor that detects the operation of the remote operator, and outputs a control signal corresponding to the operation of the remote operator to the controller 240.

[0057] The controller 240 outputs, for example, a control signal input to the remote operation device 230 to the communication device 250. Also, for example, the main image G1 transmitted from the construction machine 100 and received by the communication device 250 is input from the communication device 250 to the controller 240. The controller 240 outputs the main image G1 input from the communication device 250 to the image generation device 210.

[0058] The communication device 250 can communicate with the construction machine 100 via a predetermined communication line NW. For example, the communication device 250 transmits a control signal input from the controller 240 to the construction machine 100 via the communication line NW. In addition, the communication device 250 can also function as a receiving device that receives various data including the main image G1 and the position and posture information of the construction machine 100 transmitted from the construction machine 100 at the construction site WS via wireless communication at a remote location away from the construction site WS, and outputs the data to the image generation device 210 via the controller 240.

[0059] Next, the functions of the image generating device 210 and the remote operation support system SYS for a construction machine according to the present embodiment will be described.

[0060] As described above, the image generation device 210 of the present embodiment is a device for generating images to support remote operation of a construction machine 100 equipped with a working attachment WA. The image generation device 210 generates a composite image G including a main image G1 included in an image captured by the camera device C1 at a construction site WS where the construction machine 100 is working, and a supplementary image G2 that supplements the outside of the shooting range SR1 of the main image G1.

[0061] With this configuration, it is possible to extract a portion with less distortion centered on the optical axis of the camera C1 from the image captured by the camera C1 at the construction site WS as the main image G1 and generate a synthetic image G. On the other hand, the image of the area SR2 outside the shooting range SR1 of the main image G1 in the image captured by the camera C1 at the construction site WS tends to be more distorted than the main image G1. Therefore, the image generating device 210 generates a synthetic image G including a supplementary image G2 that supplements the area SR2 outside the shooting range SR1 of the main image G1, thereby avoiding a decrease in the sense of presence due to the distortion of the image of the above-mentioned area SR2. In addition, by including an image of the working attachment WA in at least one of the main image G1 and the supplementary image G2, the sense of presence of the synthetic image G can be further improved, thereby improving the operability of the remote operation of the construction machine 100. Therefore, according to the image generating device 210 of this embodiment, it is possible to provide a remote operator who operates the construction machine 100 at a remote location away from the construction site WS with an image of the construction site WS with a better sense of presence, thereby improving work efficiency.

[0062] Furthermore, in the image generating device 210 of the present embodiment, the complement image G2 is an image obtained by complementing a region SR2 adjacent to the left and right of the imaging range SR1 of the main image G1.

[0063] With this structure, a synthetic image G including a supplementary image G2 that supplements the adjacent area SR2 of the shooting range SR1 of the main image G1 can be generated in the rotation direction of the construction machine 100 such as a forklift, a crane or an excavator. As a result, when the construction machine 100 is rotating, a synthetic image G with a high sense of presence that is formed by supplementing the image of the rotation direction of the main image G1 with the supplementary image G2 can be provided to the remote operator.

[0064] In addition, the image generation device 210 of the present embodiment includes a learned model LM that has been learned using images outside the shooting range SR1 of the main image G1 shot by the camera device C1 at the construction site WS and images of the main image G1 as training data. In addition, the image generation device 210 inputs the main image G1 into the learned model LM to generate a synthetic image G.

[0065] With this structure, the difference between the feature quantity of the less distorted image of the region SR2 captured from the front by the camera C1 and the feature quantity of the supplementary image G2 can be minimized, and the difference between the feature quantity of the main image G1 and the feature quantity of the supplementary image G2 can be minimized. As a result, the difference between the feature quantities of the two images can be reduced to the extent that a remote operator who observes the main image G1 of the composite image G and remotely operates the construction machine 100 cannot recognize the difference between the actual image of the region SR2 captured from the front by the camera C1 and the supplementary image G2, thereby improving the sense of presence of the composite image G.

[0066] The image generation device 210 of this embodiment generates a complement image G2 including an image of an object based on information of an object recognized from an image captured by the camera C1 at the construction site WS and position and posture information of the construction machine 100 operating at the construction site WS.

[0067] This structure enables the images of objects such as workers SW, cones P, trucks DT, and other construction machines M that may exist around the construction machine 100 in the construction site WS to be displayed in the supplementary image G2, thereby providing a more realistic composite image G. In addition, from the perspective of improving safety, object detection around the construction machine 100 can also be performed using sensors other than the camera device C1.

[0068] Furthermore, the image generating device 210 of this embodiment generates a complement image G2 including an image of the construction machine 100 based on the shape, size, and posture of the construction machine 100 working at the construction site WS and the positional relationship between the construction machine 100 and the camera C1.

[0069] With this structure, for example, when a part of the construction machine 100 is present in the area SR2 of the construction site WS corresponding to the supplementary image G2, such as the forks of a forklift, the boom of a crane, or the attachment of an excavator, the part can be displayed in the supplementary image G2. Thus, a more realistic synthetic image G can be provided to a remote operator who operates the construction machine 100 at a remote location away from the construction site WS.

[0070] Furthermore, in the image generating device 210 of the present embodiment, the difference between the feature amount of the main image G1 and the feature amount of the complement image G2 is equal to or smaller than a predetermined threshold value.

[0071] Through this structure, the difference between the feature quantities of the two images can be reduced to a level where a remote operator who observes the main image G1 of the composite image G and remotely operates the construction machine 100 cannot recognize the difference between the actual image of the area SR2 taken from the front by the camera device C1 and the supplementary image G2, thereby improving the sense of presence of the composite image G.

[0072] Furthermore, the remote operation support system SYS for construction machinery of the present embodiment includes: a construction machinery 100 having a working attachment WA and a remote operation support device 200 for supporting remote operation of the construction machinery 100. The remote operation support device 200 includes: the above-mentioned image generation device 210; a display device 220 for displaying a synthetic image G generated by the image generation device 210 at a remote location away from the construction site WS; and a remote operation device 230 for operating the construction machinery 100 working at the construction site WS from a remote location.

[0073] Through this structure, a synthetic image G of the construction site WS with a better sense of presence can be displayed on the display device 220 at a remote location away from the construction site WS. In addition, at least one of the main image G1 and the supplementary image G2 included in the synthetic image G can include an image of the working attachment WA of the construction machine 100. Thus, by observing the synthetic image G displayed on the display device 220 and operating the remote operation device 230 set at the remote location, the work efficiency of the remote operator who remotely operates the construction machine 100 equipped with the working attachment WA at the construction site WS can be improved. In addition, when the display device 220 is a head-mounted display, it is advantageous from the perspective of improving the resolution in the center field of view or effective field of view of the operator OP and preventing the operator OP from dizziness based on the synthetic image G.

[0074] Furthermore, the construction machine 100 constituting the remote operation support system SYS for construction machines of the present embodiment is provided with a communication device T1 as a transmitting device for transmitting the main image G1 from the construction site WS via wireless communication. Furthermore, the remote operation support device 200 constituting the remote operation support system SYS is further provided with a communication device 250 as a receiving device for receiving the main image G1 transmitted from the transmitting device of the construction machine 100 at a remote location and outputting it to the image generating device 210 arranged at the remote location.

[0075] With this configuration, when the shooting range SR1 of the main image G1 is smaller than the shooting range SR of the camera C1, the data amount of the main image G1 can be reduced to less than the data amount of the entire image of the shooting range SR of the camera C1. As a result, when the main image G1 is transmitted from the transmission device at the construction site WS to the reception device at the remote location, the data amount can be reduced compared to the case of transmitting the entire image of the shooting range SR of the camera C1, thereby saving the communication band.

[0076] As described above, according to the image generation device 210 and the remote operation support system SYS of a construction machine of the present embodiment, the remote operator of the construction machine 100 can be provided with a synthetic image G of the construction site WS having a realistic feeling, thereby improving work efficiency.

[0077] Below, reference Figures 5 to 9 Next, an example in which the image generating device 210 and the remote operation support system SYS for a construction machine according to the present embodiment are applied to a shovel which is an example of the construction machine 100 will be described.

[0078] Figure 5 It is a side view of an excavator 100S as an example of a construction machine 100. On the lower traveling body 1 of the excavator 100S, an upper revolving body 3 is rotatably mounted via a revolving mechanism 2. A boom 4 is mounted on the upper revolving body 3. An arm 5 is mounted on the front end of the boom 4, and a bucket 6 as an end attachment is mounted on the front end of the arm 5.

[0079] The boom 4 , the arm 5 , and the bucket 6 constitute an excavation attachment as an example of a working attachment WA. The boom 4 is driven by a boom cylinder 7 , the arm 5 is driven by an arm cylinder 8 , and the bucket 6 is driven by a bucket cylinder 9 .

[0080] A boom angle sensor S1 is mounted on the boom 4 , an arm angle sensor S2 is mounted on the arm 5 , and a bucket angle sensor S3 is mounted on the bucket link. A swing angular velocity sensor S4 is mounted on the upper swing body 3 .

[0081] The boom angle sensor S1 is one of the posture detection sensors, and is configured to detect the rotation angle of the boom 4. In the present embodiment, the boom angle sensor S1 is a stroke sensor that detects the stroke amount of the boom cylinder 7, and derives the rotation angle of the boom 4 around the boom foot pin connecting the upper slewing body 3 and the boom 4 based on the stroke amount of the boom cylinder 7.

[0082] The boom angle sensor S2 is one of the posture detection sensors, which is configured to detect the rotation angle of the boom 5. In the present embodiment, the boom angle sensor S2 is a stroke sensor that detects the stroke amount of the boom cylinder 8, and derives the rotation angle of the boom 5 around the connecting pin connecting the boom 4 and the boom 5 based on the stroke amount of the boom cylinder 8.

[0083] The bucket angle sensor S3 is one of the posture detection sensors, and is configured to detect the rotation angle of the bucket 6. In the present embodiment, the bucket angle sensor S3 is a stroke sensor that detects the stroke amount of the bucket cylinder 9, and derives the rotation angle of the bucket 6 around the connecting pin connecting the boom 5 and the bucket 6 based on the stroke amount of the bucket cylinder 9.

[0084] The rotation angular velocity sensor S4 is configured to detect the rotation angular velocity of the upper rotating body 3. In the present embodiment, the rotation angular velocity sensor S4 is a gyro sensor. The rotation angular velocity sensor S4 may also be configured to calculate the rotation angle based on the rotation angular velocity. The rotation angular velocity sensor S4 may also be configured by other sensors such as a rotary encoder.

[0085] The upper rotating body 3 is equipped with a cab 10 as a cab, an engine 11, a positioning device 18, a sound collecting device A1, a camera device C1, a communication device T1, etc. In addition, a controller 30 is mounted in the cab 10. In addition, a driver's seat and an operating device, etc. are provided in the cab 10. However, the excavator 100S may be an unmanned excavator in which the cab 10 is omitted.

[0086] The engine 11 is a driving source of the shovel 100S. In the present embodiment, the engine 11 is a diesel engine. The output shaft of the engine 11 is connected to the main pump 14 and the pilot pump 15 (see Figure 6 )’s respective input shafts are connected.

[0087] The positioning device 18 is configured to measure the position of the shovel 100S. In the present embodiment, the positioning device 18 is a GNSS compass, and is configured to be able to measure the position and orientation of the upper swing body 3 .

[0088] The sound collecting device A1 is configured to collect sounds generated around the shovel 100S. In the present embodiment, the sound collecting device A1 is a microphone attached to the upper swing body 3 .

[0089] The camera device C1 is configured to capture the surroundings of the shovel 100S. In the present embodiment, the camera device C1 includes a rear camera C1B installed at the rear end of the upper surface of the upper rotating body 3, a front camera C1F installed at the front end of the upper surface of the cab 10, a left camera C1L installed at the left end of the upper surface of the upper rotating body 3, and a right camera C1R installed at the right end of the upper surface of the upper rotating body 3. The camera device C1 may be an omnidirectional camera installed at a predetermined position in the cab 10. The predetermined position is, for example, a position corresponding to the eye position of an operator sitting on a driver's seat installed in the cab 10.

[0090] The communication device T1 is configured to control communication with devices outside the shovel 100S. In the present embodiment, the communication device T1 is configured to control wireless communication between the communication device T1 and devices outside the shovel 100S via a wireless communication network. As described above, the controller 30 is a calculation device that performs various calculations.

[0091] Figure 6 It means that it is installed on Figure 5This is a block diagram of a configuration example of a drive system of an excavator 100S. Figure 6 In the figure, the mechanical power transmission line is represented by a double line, the working oil pipeline route is represented by a thick solid line, the pilot pipeline route is represented by a dotted line, and the electric control line is represented by a dotted line.

[0092] The drive system of the shovel 100S is composed of the engine 11 , the regulator 13 , the main pump 14 , the pilot pump 15 , the control valve unit 17 , the controller 30 , the solenoid valve unit 45 , etc. The engine 11 is driven and controlled by the engine control unit 74 .

[0093] The main pump 14 supplies hydraulic oil to the control valve unit 17 via the hydraulic oil line 16. In the present embodiment, the main pump 14 is a swash plate type variable displacement hydraulic pump.

[0094] The regulator 13 is configured to control the discharge volume of the main pump 14. In the present embodiment, the regulator 13 is configured to adjust the swash plate deflection angle of the main pump 14 according to the discharge pressure of the main pump 14 or a control signal from the controller 30. The discharge volume (displacement) of the main pump 14 per rotation is controlled by the regulator 13.

[0095] The pilot pump 15 is configured to supply hydraulic oil to various hydraulic control devices via the pilot line 25. In the present embodiment, the pilot pump 15 is a fixed capacity hydraulic pump. However, the pilot pump 15 may be omitted. In this case, the function of the pilot pump 15 may also be achieved by the main pump 14. That is, in addition to the function of supplying hydraulic oil to the control valve unit 17, the main pump 14 may also have the function of supplying hydraulic oil to the solenoid valve unit 45 and the like via a throttle and the like.

[0096] The control valve unit 17 is configured to selectively supply the working oil received from the main pump 14 to one or more hydraulic actuators. In the present embodiment, the control valve unit 17 includes a plurality of control valves corresponding to the plurality of hydraulic actuators. Furthermore, the control valve unit 17 is configured to selectively supply the working oil discharged from the main pump 14 to one or more hydraulic actuators. The hydraulic actuators include, for example, a boom cylinder 7, an arm cylinder 8, a bucket cylinder 9, a left travel hydraulic motor 1L, a right travel hydraulic motor 1R, and a swing hydraulic motor 2A.

[0097] The controller 30 is configured to control the electromagnetic valve unit 45 according to the operation signal received through the communication device T1. In this embodiment, the operation signal is transmitted from a remote operation room located at a remote location away from the construction site WS. The operation signal may be generated by an operation device provided in the cab 10.

[0098] The solenoid valve unit 45 includes a plurality of solenoid valves disposed in respective pilot lines 25 that connect the pilot pump 15 and the pilot ports of the respective control valves in the control valve unit 17 .

[0099] Thus, the controller 30 realizes the raising and lowering of the boom 4, the opening and closing of the arm 5, the opening and closing of the bucket 6, the rotation of the upper rotating body 3, and the traveling of the lower traveling body 1 according to the operation signal from the outside such as the remote operation room. Figure 6 The drive system shown is composed of Figure 1 A drive device D in a construction machine 100 is shown.

[0100] Figure 7 It means that it is installed on Figure 5 A block diagram of a configuration example of an electrical system of an excavator 100S. Figure 6 As shown, the engine 11 is connected to the engine control unit 74. Various data indicating the state of the engine 11 are transmitted from the engine control unit 74 to the controller 30. The controller 30 is configured to be able to store various data indicating the state of the engine 11 in the memory 30a.

[0101] The battery 70 is configured to supply electric power to various electric loads mounted on the shovel 100S. The AC generator 11a (generator), the starter 11b, the controller 30, the electrical components 72, etc. are configured to operate by the electric power stored in the battery 70. The starter 11b is configured to be driven by the electric power stored in the battery 70 and start the engine 11. In addition, the battery 70 is configured to be charged by the electric power generated by the AC generator 11a.

[0102] The water temperature sensor 11c transmits data on the temperature of the engine cooling water to the controller 30. The regulator 13 transmits data on the swash plate deflection angle to the controller 30. The discharge pressure sensor 14b transmits data on the discharge pressure of the main pump 14 to the controller 30. The positioning device 18 transmits data on the position of the shovel 100S to the controller 30.

[0103] An oil temperature sensor 14c is provided in a pipe 14-1 between a hydraulic oil tank storing hydraulic oil sucked by the main pump 14 and the main pump 14. The oil temperature sensor 14c transmits data on the temperature of the hydraulic oil flowing through the pipe 14-1 to the controller 30.

[0104] The urea water remaining amount sensor 21a provided in the urea water tank 21 transmits data on the urea water remaining amount to the controller 30. The fuel remaining amount sensor 22a provided in the fuel tank 22 transmits data on the fuel remaining amount to the controller 30.

[0105] In the shovel 100S, a boom angle sensor S1, an arm angle sensor S2, a bucket angle sensor S3, a rotation angular velocity sensor S4, a water temperature sensor 11c, a discharge pressure sensor 14b, an oil temperature sensor 14c, a urea water remaining amount sensor 21a, and a fuel remaining amount sensor 22a constitute Figure 1 The detection device S in the construction machinery 100.

[0106] The communication device T1 is configured to transmit and receive information with the communication device 250 constituting the remote operation support device 200 installed in the remote operation room RC via wireless communication. In this embodiment, the communication device T1 and the communication device 250 are configured to transmit and receive information via a fifth-generation mobile communication line (5G line), an LTE line, or a satellite line.

[0107] The remote operation room RC is provided with a sound output device A2, an indoor camera device C2, a remote operation support device 200, etc. The remote operation support device 200 includes an image generation device 210, a display device 220, a remote operation device 230, a controller 240, a communication device 250, etc. In addition, the remote operation room RC is provided with a driver's seat DS where an operator OP who remotely operates the shovel 100S sits. As described above, the image generation device 210 and the controller 240 are computing devices that perform various computations.

[0108] The sound output device A2 is configured to output sound. In the present embodiment, the sound output device A2 is a speaker configured to reproduce the sound collected by the sound collecting device A1 mounted on the shovel 100S.

[0109] The indoor camera device C2 is configured to capture images in the remote operation room RC. In the present embodiment, the indoor camera device C2 is a camera installed inside the remote operation room RC, and is configured to capture images of the operator OP sitting in the driver's seat DS.

[0110] The communication device 250 is configured to control wireless communication with the communication device T1 mounted on the shovel 100S.

[0111] In the present embodiment, the driver's seat DS has the same structure as the driver's seat provided in the cab of a common excavator. Specifically, a left control console box is arranged on the left side of the driver's seat DS, and a right control console box is arranged on the right side of the driver's seat DS. In addition, a left operating lever is arranged at the front end of the upper surface of the left control console box, and a right operating lever is arranged at the front end of the upper surface of the right control console box. In addition, a travel lever and a travel pedal are arranged in front of the driver's seat DS. In addition, an engine speed adjustment control panel 75 is arranged in the center of the upper surface of the right control console box. The left operating lever, the right operating lever, the travel lever, the travel pedal and the engine speed adjustment control panel 75 respectively constitute the remote control device 230.

[0112] The engine speed adjustment control dial 75 is a control dial for adjusting the speed of the engine 11 , and is configured to be able to switch the engine speed in four stages, for example.

[0113] Specifically, the engine speed adjustment control dial 75 is configured to be able to switch the engine speed among four stages: the SP mode, the H mode, the A mode, and the idle mode. The engine speed adjustment control dial 75 transmits data related to the setting of the engine speed to the controller 30 .

[0114] The SP mode is a speed mode selected when the operator OP wants to prioritize the workload, and uses the highest engine speed. The H mode is a speed mode selected when the operator OP wants to balance the workload and the fuel consumption rate, and uses the second highest engine speed. The A mode is a speed mode selected when the operator OP wants to prioritize the fuel consumption rate while operating the excavator with low noise, and uses the third highest engine speed. The idle mode is a speed mode selected when the operator OP wants to set the engine to an idle state, and uses the lowest engine speed. In addition, the engine 11 is speed-controlled at a constant engine speed of the speed mode selected via the engine speed adjustment control panel 75.

[0115] The remote operating device 230 is provided with an operation sensor 29 for detecting the operation content of the remote operating device 230. The operation sensor 29 is, for example, a tilt sensor for detecting the tilt angle of the operating lever, or an angle sensor for detecting the swing angle of the operating lever around the swing axis. The operation sensor 29 may be composed of other sensors such as a pressure sensor, a current sensor, a voltage sensor, or a distance sensor. The operation sensor 29 outputs information related to the detected operation content of the operating device 26 to the controller 240. The controller 240 generates an operation signal based on the received information, and sends the generated operation signal to the excavator 100S. The operation sensor 29 may be configured to generate an operation signal. At this time, the operation sensor 29 may output the operation signal to the communication device 250 without passing through the controller 240.

[0116] The display device 220 is configured to display information related to the surrounding conditions of the shovel 100S. In the present embodiment, the display device 220 is a multi-display composed of 9 monitors in 3 vertical levels and 3 horizontal columns, and is configured to display the conditions of the space in front of, to the left of, and to the right of the shovel 100S. Each monitor is a liquid crystal monitor or an organic EL monitor. However, the display device 220 may be composed of one or more curved monitors, or may be composed of a projector.

[0117] The display device 220 may be a display device that the operator OP can wear. For example, the display device 220 is a head-mounted display, which may be configured to be able to send and receive information between the controller 240 through wireless communication. The head-mounted display may be connected to the controller 240 by wire. The head-mounted display may be a transmissive head-mounted display or a non-transmissive head-mounted display. The head-mounted display may be a monocular head-mounted display or a binocular head-mounted display.

[0118] The display device 220 is configured to display an image that allows the operator OP in the remote operation room RC to visually recognize the surroundings of the shovel 100S. That is, the display device 220 displays an image so that the operator can check the conditions around the shovel 100S as if he is in the cab 10 of the shovel 100S even if he is in the remote operation room RC.

[0119] As described above, the image generation device 210 is a device that generates an image to support remote operation of the shovel 100S as the construction machine 100 equipped with the excavation attachment as the work attachment WA. The image generation device 210 generates a composite image G including a main image G1 included in an image captured by the camera device C1 at the construction site WS where the shovel 100S is working and a supplementary image G2 that supplements the outside of the capture range SR1 of the main image G1. The display device 220 displays an image including at least a portion of the composite image G generated by the image generation device 210.

[0120] Next, refer to Figure 8 The relationship between the first coordinate system having the reference point R1 in the remote control room RC as the origin and the second coordinate system having the reference point R2 in the shovel 100S as the origin will be described. In the following description, the first coordinate system is referred to as the control room coordinate system, and the second coordinate system is referred to as the shovel coordinate system. Figure 8 This is a diagram showing the relationship between the operator's cabin coordinate system and the shovel coordinate system.

[0121] The operation room coordinate system is a three-dimensional UVW orthogonal coordinate system with the reference point R1 in the remote operation room RC as the origin, and has a U axis extending parallel to the front-to-back direction of the driver's seat DS, a V axis extending parallel to the left-to-right direction of the driver's seat DS, and a W axis orthogonal to the U axis and the V axis.

[0122] The shovel coordinate system is a three-dimensional XYZ orthogonal coordinate system with the reference point R2 on the upper rotating body 3 as the origin, and has an X-axis extending parallel to the front-to-back direction of the upper rotating body 3, a Y-axis extending parallel to the left-to-right direction of the upper rotating body 3, and a Z-axis orthogonal to the X-axis and the Y-axis. Figure 8 In the example, the reference point R2 is a point on the rotation axis, the XY plane is a horizontal plane, and the Z axis is a vertical axis. Figure 8 In the example, the ground contact surface as the virtual plane where the shovel 100S is located is a horizontal plane. Then, the X axis corresponds to the U axis of the operator's room coordinate system, the Y axis corresponds to the V axis of the operator's room coordinate system, and the Z axis corresponds to the W axis of the operator's room coordinate system.

[0123] In the present embodiment, each three-dimensional coordinate in the operator's room coordinate system is pre-associated with one of the three-dimensional coordinates in the shovel coordinate system. Therefore, if the three-dimensional coordinates of the operator's viewpoint E1, which is the eye position of the operator OP in the remote operator's room RC, are determined, the three-dimensional coordinates of the virtual operator's viewpoint E1', which is the eye position of the virtual operator in the shovel 100S, are uniquely determined. In addition, the eye position of the operator OP is, for example, the middle point between the left eye position and the right eye position of the operator OP. However, the eye position of the operator OP may be a preset position. That is, the operator's viewpoint E1 and the virtual operator's viewpoint E1' may be fixed points.

[0124] Fig. 9 2 is a diagram showing a configuration example of the display device 220. Fig. 9 As shown, the display device 220 is a multi-screen display consisting of nine monitors in three vertical rows and three horizontal columns. Specifically, the display device 220 includes a central monitor D1a, an upper monitor D1b, a lower monitor D1c, a left monitor D1d, a right monitor D1e, an upper left monitor D1f, an upper right monitor D1g, a lower left monitor D1h, and a lower right monitor D1i.

[0125] The display device 220 displays the user information on at least a portion of the central monitor D1a, the upper monitor D1b, and the lower monitor D1c corresponding to the central field of view of the operator OP in the remote operation room RC. Figure 2 and Figure 3 The display device 220 displays the main image G1 for explanation on at least a portion of the left monitor D1d, the right monitor D1e, the upper left monitor D1f, the upper right monitor D1g, the lower left monitor D1h, and the lower right monitor D1i corresponding to the peripheral field of view of the operator OP in the remote operation room RC. Figure 2 and Figure 3 The following is a description of the complementary image G2.

[0126] With this structure, from the image captured at the construction site WS by the camera C1 installed in the shovel 100S, a portion with less distortion centered on the optical axis of the camera C1 can be displayed as the main image G1 in the central field of view of the operator OP in the remote operation room RC. In addition, a supplementary image G2 that supplements the area SR2 outside the shooting range SR1 of the main image G1 can be displayed in the peripheral field of view of the operator OP in the remote operation room RC. In addition, at least one of the main image G1 and the supplementary image G2 can include an image of the excavation attachment of the shovel 100S. As a result, an image of the construction site WS with a more realistic feeling can be provided to the operator OP who operates the shovel 100S equipped with the excavation attachment at a remote location away from the construction site WS, thereby improving the working efficiency of the shovel 100S.

[0127] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the above embodiments. Various modifications or substitutions can be applied to the above embodiments without departing from the scope of the present invention. Furthermore, the features described separately can be combined as long as there is no technical contradiction.

[0128] For example, in the above-mentioned embodiment, an example is described in which the image generating device is set at a remote location away from the construction site. However, the image generating device may also be mounted on a construction machine. In this case, the synthetic image generated by the image generating device can be sent from the construction machine at the construction site, and the synthetic image can be received at the remote location where the remote operating device is set and displayed on the display device.

[0129] Furthermore, the image generation device may generate a synthetic image without using machine learning. For example, the image generation device may extract a past image of the camera device stored in association with the position and posture information of the construction machine according to the current position and posture information of the construction machine, and display it as a supplementary image.

Claims

1. An image generating device for generating an image for supporting remote operation of a construction machine equipped with a working attachment, wherein: The image generation device generates a composite image including a main image included in an image captured by a camera device at a construction site where the construction machine is working and a supplementary image that supplements an area outside a capture range of the main image.

2. The image generating device according to claim 1, wherein: The complementary image is an image obtained by complementing the areas adjacent to the left and right of the shooting range of the main image.

3. The image generating device according to claim 1, comprising: The learning model is learned by using images outside the shooting range of the main image shot by the camera device at the construction site and images of the main image as training data, The main image is input into the learned model to generate the synthesized image.

4. The image generating device according to claim 1, wherein: The complement image including the image of the object is generated based on information of the object recognized from the image captured by the camera device at the construction site and position and posture information of a construction machine working at the construction site.

5. The image generating device according to claim 1, wherein: The complement image including the image of the construction machine is generated based on the shape, size, and posture of the construction machine working at the construction site and the positional relationship between the construction machine and the imaging device.

6. The image generating device according to claim 1, wherein: A difference between a feature amount of the main image and a feature amount of the complement image is equal to or smaller than a predetermined threshold.

7. A remote operation support system for a construction machine, comprising a construction machine having a working attachment and a remote operation support device for supporting remote operation of the construction machine, wherein: The remote operation support device comprises: The image generating device according to any one of claims 1 to 6; a display device for displaying the composite image generated by the image generating device at a remote location away from the construction site; and A remote operation device operates the construction machine working at the construction site from the remote location.

8. The remote operation support system for construction machinery according to claim 7, wherein: The construction machine includes a transmission device for transmitting the main image from the construction site by wireless communication. The remote operation support device further includes a receiving device that receives the main image transmitted from the transmitting device at the remote location and outputs the main image to the image generating device disposed at the remote location.

Images