Image generating device and remote operation support system of work machine

Abstract

To provide an image of a work site with presence by a remote operator of a work machine, and improve work efficiency.SOLUTION: An image generating device generates an image supporting remote operation of a work machine 100 provided on a work attachment. The image generating device generates a composite image including a main image included in an image photographed by an imaging device C1 at a work site WS where the work machine 100 is worked, and a compensation image for compensating a region SR2 outside an imaging range SR1 of the main image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to an image generating device and a remote operation support system for a work machine. [Background technology]

[0002] 2. Description of the Related Art Conventionally, a remote control system for a work machine is known (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2021-114757 A Summary of the Invention [Problem to be solved by the invention]

[0004] The above-mentioned remote control system is configured to display an image of the work site on a display device arranged in a remote control room outside the work machine.

[0005] However, with this configuration, it is not possible to provide a realistic image to the operator remotely operating the work machine in the remote control room, and there is a risk that work efficiency will decrease.

[0006] For example, the 2006 report by the Hokuriku Regional Development Bureau of the Ministry of Land, Infrastructure, Transport and Tourism on improving the operability of unmanned construction machinery points out that the biggest factor in reducing work efficiency when remotely operating machinery is a lack of visual information.

[0007] Therefore, from the perspective of improving work efficiency, it is desirable to provide a remote operator of a work machine with a more realistic image of the work site. [Means for solving the problem]

[0008] An image generating device according to one embodiment of the present disclosure is an image generating device that generates images to assist in remote operation of a work machine equipped with a work attachment, and generates a composite image that includes a main image contained in an image captured by an imaging device at a work site where the work machine is working, and a complementary image that complements the area outside the imaging range of the main image.

[0009] A work machine remote operation support system according to another embodiment of the present disclosure is a work machine remote operation support system including a work machine equipped with a work attachment and a remote operation support device that supports remote operation of the work machine, wherein the remote operation support device includes the image generation device, a display device that displays the composite image generated by the image generation device at a remote location away from the work site, and a remote operation device that operates the work machine working at the work site from the remote location. Effect of the Invention

[0010] According to the above-described embodiment, a more realistic image of the work site can be provided to the remote operator of the work machine, thereby improving work efficiency. [Brief description of the drawings]

[0011] [Figure 1] 1 is a schematic diagram illustrating an example of a remote operation support system according to an embodiment of the present disclosure. [Diagram 2] FIG. 1 is a schematic plan view showing an example of a work site. [Diagram 3] FIG. 2 is a diagram illustrating an example of a composite image generated by an image generating device. [Figure 4] FIG. 1 is a diagram illustrating an example of a trained model. [Diagram 5] FIG. 1 is a side view of a shovel as an example of a work machine. [Figure 6] 6 is a diagram showing a configuration example of a drive system mounted on the excavator shown in FIG. 5. [Figure 7] FIG. 6 is a diagram showing a configuration example of an electrical system mounted on the excavator shown in FIG. 5. [Figure 8]FIG. 2 is a diagram showing the relationship between a coordinate system relating to a remote control room and a coordinate system relating to a shovel. [Figure 9] FIG. 1 is a diagram illustrating a configuration example of a display device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the invention will be described with reference to the drawings.

[0013] Fig. 1 is a schematic diagram showing an example of a remote operation support system SYS according to an embodiment of the present disclosure. Fig. 2 is a schematic plan view showing an example of a work site WS. Fig. 3 is a diagram showing an example of a composite image G generated by an image generating device 210 of this embodiment constituting the remote operation support system SYS shown in Fig. 1.

[0014] Although details will be described later, the image generation device 210 of this embodiment is a device that generates images to support remote operation of a work machine 100 equipped with a work attachment WA. The image generation device 210 generates a composite image G that includes a main image G1 included in an image captured by an imaging device C1 at a work site WS where the work machine 100 is working, and a complementary image G2 that complements the outside of an imaging range SR1 of the main image G1. Note that the "complementary image" can also be referred to as an "extrapolated image." The image generation device 210 and the work machine remote operation support system SYS of this embodiment will be described in detail below.

[0015] As shown in Fig. 1, the remote operation support system SYS of this embodiment includes, for example, a work machine 100 equipped with a work attachment WA, and a remote operation support device 200 that supports remote operation of the work machine 100. The work machine 100 included in the remote operation support system SYS may be one unit, or multiple units. Furthermore, when the remote operation support system SYS includes multiple work machines 100, the multiple work machines 100 may all be of the same type, or at least some of them may be of different types.

[0016] Furthermore, the remote operation support system SYS may include a plurality of remote operation support devices 200. That is, the plurality of remote operation support devices 200 may execute processing related to the remote operation support system SYS in a distributed manner. For example, each of the plurality of remote operation support devices 200 may communicate with a portion of the work machines 100 that it is responsible for among all the work machines 100 included in the remote operation support system SYS, and execute processing targeted at the portion of the work machines 100.

[0017] The work machine 100 is a machine that performs work at a work site WS, such as a forklift, a crane, or a shovel. The work machine 100 includes, for example, an imaging device C1, a controller 30, a detection device S, a drive device D, a communication device T1, and a work attachment WA.

[0018] The imaging device C1 is configured to capture an image of the work site WS around the work machine 100. The imaging device C1 outputs, for example, an image of the work site WS that has been captured to the controller 30. Note that the imaging device C1 does not have to be mounted on the work machine 100. The imaging device C1 may be attached, for example, to a pole installed at the work site WS or to a drone flying above the work site WS.

[0019] The imaging device C1 includes, for example, one or more cameras. More specifically, the imaging device C1 includes, for example, a front camera C1F, a rear camera C1B, a right camera C1R, and a left camera C1L as shown in Fig. 2. The imaging device C1 may include one or more stereo cameras and / or one or more monocular cameras.

[0020] The front camera C1F and the rear camera C1B are respectively installed on the front and rear of the body of the work machine 100 and photograph the areas in front of and behind the work machine 100. The right camera C1R and the left camera C1L are respectively installed on the right and left sides of the body of the work machine 100 and photograph the right and left sides of the work machine 100. In Fig. 2, the imaging range SR of each camera is indicated by a dashed line.

[0021] 2, the imaging ranges SR of the front camera C1F and rear camera C1B overlap with the imaging ranges SR of the right camera C1R and left camera C1L, respectively. As a result, the imaging device C1 is configured to be able to capture images of the work site WS in a 360° range around the work machine 100. Note that the imaging device C1 may be a spherical camera installed on the body or inside the cab of the work machine 100.

[0022] The detection device S is configured to detect physical quantities related to the work machine 100. The detection device S includes, for example, a speed sensor, an acceleration sensor, an angular velocity sensor, an inertial measurement unit (IMU), an angle sensor, a temperature sensor, a pressure sensor, a rotation speed sensor, a fuel level sensor, a urea water level sensor, and the like. The detection device S may also include a global navigation satellite system (GNSS) receiver that detects position information of the work machine 100. The detection device S outputs the detection results of the physical quantities and position and attitude information related to the work machine 100 to the controller 30.

[0023] The drive device D is configured to drive the work machine 100. The drive device D includes, for example, an engine, a transmission, a hydraulic device (hydraulic tank, hydraulic pump, control valve, hydraulic actuator, etc.), an electric motor, etc. The drive device D generates power for, for example, making the work machine 100 travel, turning the work machine 100, and operating an attachment of the work machine 100.

[0024] The communication device T1 can communicate with the remote operation support device 200 via a predetermined communication line NW. The communication device T1 outputs a control signal received from the remote operation support device 200 via the communication line NW to the controller 30. The communication device T1 also functions as a transmitting device that transmits various data including image data of the main image G1 input from the controller 30 and position and orientation information of the work machine 100 to the remote operation support device 200 via the communication line NW.

[0025] The work attachment WA is, for example, a device that allows the work machine 100 to perform work at a work site WS, and is controlled by the controller 30 based on remote operation by a remote operator. If the work machine 100 is a forklift, the work attachment WA includes, for example, a fork, a backrest, a lift chain, a mast, etc. If the work machine 100 is a crane, the work attachment WA includes, for example, a boom, a jib, a hook, a guy line, etc. If the work machine 100 is a shovel, the work attachment WA is, for example, an excavation attachment including a boom, an arm, a bucket, etc.

[0026] The communication line NW includes, for example, a wide area network (WAN). The wide area network may include, for example, a mobile communication network with a base station as a terminal. The wide area network may also include, for example, a satellite communication network that uses a communication satellite above the work machine 100. The wide area network may also include, for example, an Internet network. The communication line NW may also include, for example, a local network (LAN) of a facility in which the remote operation support device 200 is installed. The local network may be a wireless line, a wired line, or a line including both. The communication line NW may also include, for example, a short-distance communication line based on a predetermined wireless communication method such as WiFi or Bluetooth (registered trademark).

[0027] The controller 30 is a calculation device that executes various calculations. In this embodiment, the controller 30 is configured with a microcomputer including a CPU and a memory 30a. Various functions of the controller 30 are realized by the CPU executing a program stored in the memory 30a.

[0028] The controller 30 extracts, for example, a part of an image captured by the imaging device C1 at the work site WS, and generates a main image G1 to be transmitted to the remote operation support device 200. As shown in Fig. 2, when the angle of view θ of the camera constituting the imaging device C1 becomes wider, the distortion of the peripheral part of the image becomes larger. Therefore, the controller 30 extracts, for example, as shown in Fig. 2 and Fig. 3, a main image G1 which is an image of an imaging range SR1 corresponding to a predetermined angle of view θ1 centered on the optical axis of each camera from among the imaging ranges SR of each camera constituting the imaging device C1.

[0029] The angle of view θ1 of the imaging range SR1 of the main image G1 is, for example, in the range of about 20° to 30°, which corresponds to the central or effective visual field of a human being, with the optical axis of the camera constituting the imaging device C1 as the center. That is, when the angle of view θ of the camera constituting the imaging device C1 is wider than 20° to 30°, the controller 30 extracts the main image G1, which is a part of the entire image of the imaging range SR, and transmits it to the remote operation support device 200 via the communication device T1. This makes it possible to reduce image data compared to the case where the entire image of the imaging range SR is transmitted to the remote operation support device 200.

[0030] Furthermore, by extracting a main image G1 having an imaging range SR1 with a viewing angle θ1 corresponding to a human's central visual field or effective visual field from the entire image of the imaging range SR captured by the imaging device C1 at the work site WS, an image with less distortion can be transmitted from the work machine 100 to the remote operation support device 200. Note that when the viewing angle θ of the imaging device C1 is a relatively narrow angle, for example, about 20° to 30°, the image with less distortion in the entire imaging range SR is. In this case, the controller 30 may extract the entire image of the imaging range SR of the imaging device C1 as the main image G1 and transmit it to the remote operation support device 200 via the communication device T1.

[0031] 1, the remote operation support device 200 includes an image generating device 210, a display device 220, and a remote operation device 230. The remote operation support device 200 also includes, for example, a controller 240 and a communication device 250. The remote operation support device 200 can communicate with the work machine 100 via a predetermined communication line NW through the communication device 250.

[0032] The image generating device 210 and the controller 240 are, for example, arithmetic devices that execute various calculations. In this embodiment, the image generating device 210 and the controller 240 are configured with a microcomputer including a CPU and a memory. The various functions of the image generating device 210 and the controller 240 are realized by the CPU executing a program stored in the memory. The image generating device 210 and the controller 240 may be configured with separate microcomputers, or may be configured with a common microcomputer.

[0033] As described above, the image generating device 210 generates an image that supports remote operation of the work machine 100 equipped with the work attachment WA. The image generating device 210 generates a composite image G including a main image G1 included in an image captured by the imaging device C1 at the work site WS where the work machine 100 works, and a complementary image G2 that complements the outside of the imaging range SR1 of the main image G1. More specifically, the complementary image G2 is, for example, an image that complements an area SR2 adjacent to the left and right of the imaging range SR1 of the main image G1, as shown in FIG. 2. The complementary image G2 may also be an image that complements an area adjacent to the top and bottom of the imaging range SR1 of the main image G1. The complementary image G2 is, for example, not an image captured by the imaging device C1, but an image created by the image generating device 210, and that complements an area outside the imaging range SR of the imaging device C1 or the imaging range SR1 of the main image G1.

[0034] The main image G1 is, for example, an image to be displayed in the central vision or effective vision of the remote operator. The complementary image G2 is, for example, an image to be displayed in the peripheral vision of the remote operator. The difference between the feature amount of the main image G1 and the feature amount of the complementary image G2 is equal to or less than a predetermined threshold. Here, it is desirable to set the predetermined threshold in a range in which, for example, a remote operator who remotely operates the work machine 100 while looking at the main image G1 of the composite image G cannot recognize the difference between the actual image of the area SR2 captured by the imaging device C1 and the complementary image G2. Note that the actual image of the area SR2 does not need to be transmitted from the work machine 100 to the remote operation support device 200.

[0035] The feature values ​​of the main image G1 and the complementary image G2 include, for example, RGB values, chromaticity, brightness, saturation, etc., for each pixel or unit area of ​​each image. The feature values ​​of the main image G1 and the complementary image G2 may be derived by algorithms such as SIFT (Scale-Invariant Feature Transform) and HOG (Histograms of Oriented Gradients). The feature values ​​of the main image G1 and the complementary image G2 may be quantified color distributions in the images, or may be local feature values ​​obtained by detecting feature points in the images with large changes in shading using a feature point detection method and converting the areas around the feature points into feature vectors using pixel values ​​or differential values. The complementary image G2 includes, for example, a single-color image with the same color as the color of the end portion of the main image G1 adjacent to the complementary image G2, an illustration-like image, and a CG image. The complementary image G2 may include, for example, an image obtained by correcting distortion of an image captured by the imaging device C1.

[0036] The image generating device 210 is equipped with a trained model LM that has been trained using, as teacher data, an image outside the imaging range SR1 of the main image G1 captured by the imaging device C1 at the work site WS and an image including the main image G1. The image generating device 210, for example, inputs the main image G1 into the trained model LM to generate a composite image G. The image outside the imaging range SR1 of the main image G1 included in the teacher data may be an image captured by the imaging device C1 in the past (such as one minute ago), or may be an image of another work site similar to the work site WS. The trained model LM may be generated by unsupervised learning.

[0037] FIG. 4 is a diagram for explaining an example of the trained model LM. Machine learning used to generate the trained model LM is, for example, machine learning using a deep neural network (DNN), which applies deep learning. The trained model LM outputs a composite image G from an output layer (n layer) by inputting a main image G1 to an input layer (0 layer) of a neural network, for example. The image generating device 210 may generate a composite image G including a main image G1 and a complementary image G2 using a method such as GAN (Generative Adversarial Networks), TF-GAN, DeepFill, Pconv, or Buoundless.

[0038] The image generating device 210 generates a teacher data set using, for example, a plurality of main images G1 stored in a non-volatile storage device constituting the controller 240 while the work machine 100 is in operation, and position and orientation information of the work machine 100 corresponding to each main image G1. The image generating device 210 generates a trained model LM by, for example, performing machine learning using the generated teacher data set.

[0039] Furthermore, the image generating device 210 generates a complementary image G2 including an image of an object based on, for example, information about the object recognized from an image captured by the imaging device C1 at the work site WS and the position and orientation information of the work machine 100 working at the work site WS. More specifically, for example, as shown in Fig. 2, it is assumed that the forward direction of the work machine 100 is 0°, the right direction is 90°, the backward direction is 180°, and the left direction is 270° (the angle setting method is arbitrary), and that a worker SW, a cone P, a truck T, and another work machine M are present at each angle.

[0040] In this case, the image generating device 210 recognizes objects such as the worker SW, the cone P, the truck DT, and other work machines M from the image captured by the imaging device C1 using a known method such as pattern matching. Furthermore, the image generating device 210 stores object information, which associates the object recognition result with the position and orientation information of the work machine 100 when the object was photographed by the imaging device C1, in a non-volatile storage device such as a ROM constituting the image generating device 210. Note that the object information may be stored in a volatile storage device such as a RAM constituting the image generating device 210.

[0041] Then, the image generating device 210 compares the current position and orientation information of the work machine 100 with the object information stored in the non-volatile storage device, and if an object exists in the area SR2 corresponding to the complementary image G2, it generates an image of the object and displays it superimposed on the complementary image G2. More specifically, as shown in Fig. 2, if a cone P exists in the area SR2 where the complementary image G2 is generated, the image generating device 210 generates a complementary image G2 including an image of the cone P, as shown in Fig. 3.

[0042] Here, the "image of the cone P" is typically an image unrelated to the image of the actual cone P captured by the imaging device C1 (an image generated by the image generating device 210 without based on the image of the actual cone P). However, the "image of the cone P" may also be an image generated by the image generating device 210 based on an image of the actual cone P captured in the past by the imaging device C1. In addition, a keyword representing an object recognized from the image of the imaging device C1 may be input to the image generating device 210 to generate a complementary image G2 including an image of the object.

[0043] Also, there are cases where a part of the work machine 100 is present in the area SR2 of the work site WS corresponding to the complementary image G2, such as a work attachment WA, specifically, the forks of a forklift, the boom of a crane, or an attachment (boom, arm, end attachment) of a shovel. In such cases, the image generating device 210 may generate a complementary image G2 including an image of a part of the work machine 100, based on, for example, the shape, dimensions, and attitude of the work machine 100 working at the work site WS, and the positional relationship between the work machine 100 and the imaging device C1. In this case, the image generating device 210 may use CAD data, such as a 3D model of the work machine 100, stored in advance in a non-volatile storage device.

[0044] The display device 220 includes a display such as a liquid crystal display, an organic EL display, or a head mounted display, and displays the composite image G input from the image generating device 210. The display device 220 provides the composite image G of the work site WS to a remote operator who operates the work machine 100 via a remote operation device 230 installed in a remote location away from the work site WS.

[0045] The remote control device 230 is, for example, a device simulating the driver's seat or the operator's seat of the work machine 100, and is equipped with operation devices and output devices similar to those of the driver's seat or the operator's seat of the work machine 100. More specifically, the operation devices of the remote control device 230 include, for example, an operation lever, an operation pedal, an operation dial, an operation button, a touch panel, etc. Furthermore, the output devices of the remote control device 230 include, for example, a monitor, a speaker, a buzzer, etc. The operation device of the remote control device 230 includes, for example, a sensor that detects the operation of the remote operator, and outputs a control signal according to the operation of the remote operator to the controller 240.

[0046] The controller 240 outputs, for example, a control signal input to the remote operation device 230 to the communication device 250. In addition, for example, a main image G1 transmitted from the work machine 100 and received by the communication device 250 is input to the controller 240 from the communication device 250. The controller 240 outputs the main image G1 input from the communication device 250 to the image generation device 210.

[0047] The communication device 250 can communicate with the work machine 100 via a specified communication line NW. The communication device 250 transmits, for example, a control signal input from the controller 240 to the work machine 100 via the communication line NW. The communication device 250 also functions as a receiving device that receives various data including the main image G1 and position and orientation information of the work machine 100 transmitted by wireless communication from the work machine 100 at the work site WS at a remote location away from the work site WS, and outputs the data to the image generating device 210 via the controller 240.

[0048] The operation of the image generating device 210 and the work machine remote operation support system SYS of this embodiment will be described below.

[0049] As described above, the image generating device 210 of this embodiment is a device that generates images to assist in remote operation of a work machine 100 equipped with a work attachment WA. The image generating device 210 generates a composite image G that includes a main image G1 included in an image captured by the imaging device C1 at the work site WS where the work machine 100 is working, and a complementary image G2 that complements the outside of the imaging range SR1 of the main image G1.

[0050] With this configuration, a portion with less distortion centered on the optical axis of the imaging device C1 can be extracted as the main image G1 from the image captured by the imaging device C1 at the work site WS to generate a composite image G. On the other hand, among the images captured by the imaging device C1 at the work site WS, the image of the area SR2 outside the imaging range SR1 of the main image G1 tends to be more distorted than the main image G1. Therefore, by generating a composite image G including a complementary image G2 that complements the area SR2 outside the imaging range SR1 of the main image G1 by the image generating device 210, it is possible to avoid a decrease in realism due to the distortion of the image of the area SR2 described above. Furthermore, by including an image of the work attachment WA in at least one of the main image G1 and the complementary image G2, the realism of the composite image G can be further improved, and the operability of the remote operation of the work machine 100 can be improved. Therefore, according to the image generating device 210 of this embodiment, a more realistic image of the work site WS can be provided to a remote operator who operates the work machine 100 at a remote location away from the work site WS, and work efficiency can be improved.

[0051] In the image generating device 210 of this embodiment, the complement image G2 is an image that complements the regions SR2 adjacent to the left and right of the imaging range SR1 of the main image G1.

[0052] This configuration makes it possible to generate a composite image G including a complementary image G2 that complements an area SR2 adjacent to an imaging range SR1 of a main image G1 in the turning direction of a work machine 100 such as a forklift, a crane, or a shovel. This makes it possible to provide a remote operator with a highly realistic composite image G in which an image in the turning direction of the main image G1 is complemented with the complementary image G2 when the work machine 100 turns.

[0053] The image generating device 210 of this embodiment is equipped with a trained model LM trained using images outside the imaging range SR1 of the main image G1 captured by the imaging device C1 at the work site WS and images including the main image G1 as training data. The image generating device 210 inputs the main image G1 to the trained model LM to generate a composite image G.

[0054] This configuration makes it possible to reduce as much as possible the difference between the feature amount of an image with little distortion captured from the front of the area SR2 by the imaging device C1 and the feature amount of the complementary image G2, and to reduce as much as possible the difference between the feature amount of the main image G1 and the feature amount of the complementary image G2. As a result, the difference in the feature amounts of both images can be reduced to an extent that a remote operator who remotely operates the work machine 100 while looking at the main image G1 of the composite image G cannot recognize the difference between the actual image of the area SR2 captured from the front by the imaging device C1 and the complementary image G2, thereby improving the sense of realism of the composite image G.

[0055] In addition, the image generating device 210 of this embodiment generates a complementary image G2 including an image of the object based on information of the object recognized from an image captured by the imaging device C1 at the work site WS and position and orientation information of the work machine 100 performing work at the work site WS.

[0056] With this configuration, images of objects such as the worker SW, cones P, trucks DT, and other work machines M that may be present around the work machine 100 at the work site WS can be displayed in the complementary image G2, providing a more realistic composite image G. From the viewpoint of improving safety, detection of objects around the work machine 100 can also be performed by using a sensor other than the imaging device C1 in combination.

[0057] In addition, the image generation device 210 of this embodiment generates a complementary image G2 including an image of the work machine 100 based on the shape, dimensions, and posture of the work machine 100 working at the work site WS, as well as the positional relationship between the work machine 100 and the imaging device C1.

[0058] With this configuration, when a part of the work machine 100, such as the forks of a forklift, the boom of a crane, or an attachment of a shovel, is present in the area SR2 of the work site WS corresponding to the complementary image G2, that part can be displayed in the complementary image G2. This makes it possible to provide a more realistic composite image G to a remote operator who operates the work machine 100 at a remote location away from the work site WS.

[0059] In the image generating device 210 of this embodiment, the difference between the feature amount of the main image G1 and the feature amount of the complement image G2 is equal to or less than a predetermined threshold value.

[0060] With this configuration, the difference in the features of the two images is reduced to the extent that a remote operator who remotely operates the work machine 100 while looking at the main image G1 of the composite image G cannot recognize the difference between the actual image of the area SR2 captured from the front by the imaging device C1 and the complementary image G2, thereby improving the sense of realism of the composite image G.

[0061] Moreover, the work machine remote operation support system SYS of this embodiment includes a work machine 100 equipped with a work attachment WA, and a remote operation support device 200 that supports the remote operation of the work machine 100. The remote operation support device 200 includes the above-mentioned image generation device 210, a display device 220 that displays a composite image G generated by the image generation device 210 at a remote location away from the work site WS, and a remote operation device 230 that remotely operates the work machine 100 working at the work site WS.

[0062] With this configuration, a more realistic composite image G of the work site WS can be displayed on the display device 220 at a remote location away from the work site WS. Furthermore, at least one of the main image G1 and the complementary image G2 included in the composite image G can include an image of the work attachment WA of the work machine 100. This can improve the work efficiency of a remote operator who operates a remote control device 230 installed at a remote location to remotely operate a work machine 100 at the work site WS equipped with a work attachment WA while viewing the composite image G displayed on the display device 220. Furthermore, when the display device 220 is a head-mounted display, this is advantageous in terms of improving the resolution in the central visual field or effective visual field of the operator OP and preventing the operator OP from getting sick from the composite image G.

[0063] The work machine 100 constituting the work machine remote operation support system SYS of this embodiment is equipped with a communication device T1 as a transmitting device that transmits the main image G1 from the work site WS by wireless communication. The remote operation support device 200 constituting the remote operation support system SYS is further equipped with a communication device 250 as a receiving device that receives the main image G1 transmitted from the transmitting device of the work machine 100 at a remote location and outputs it to an image generating device 210 arranged at the remote location.

[0064] With this configuration, when the imaging range SR1 of the main image G1 is narrower than the imaging range SR of the imaging device C1, the amount of data of the main image G1 can be reduced to less than the amount of data of the entire image of the imaging range SR of the imaging device C1. As a result, when transmitting the main image G1 from the transmitting device at the work site WS to the receiving device at a remote location, the amount of data can be reduced compared to the case of transmitting the entire image of the imaging range SR of the imaging device C1, and communication bandwidth can be saved.

[0065] As described above, according to the image generation device 210 and the work machine remote operation support system SYS of this embodiment, a more realistic composite image G of the work site WS can be provided to the remote operator of the work machine 100, thereby improving work efficiency.

[0066] Hereinafter, with reference to Figs. 5 to 9, an example in which the image generating device 210 and the work machine remote operation support system SYS of this embodiment are applied to a shovel, which is an example of a work machine 100, will be described.

[0067] 5 is a side view of an excavator 100S as an example of a work machine 100. An upper rotating body 3 is rotatably mounted on a lower traveling body 1 of the excavator 100S via a rotating mechanism 2. A boom 4 is attached to the upper rotating body 3. An arm 5 is attached to the tip of the boom 4, and a bucket 6 is attached to the tip of the arm 5 as an end attachment.

[0068] The boom 4, the arm 5, and the bucket 6 constitute an excavation attachment, which is an example of a work attachment WA. The boom 4 is driven by a boom cylinder 7, the arm 5 is driven by an arm cylinder 8, and the bucket 6 is driven by a bucket cylinder 9.

[0069] A boom angle sensor S1 is attached to the boom 4, an arm angle sensor S2 is attached to the arm 5, and a bucket angle sensor S3 is attached to the bucket link. A swing angular velocity sensor S4 is attached to the upper swing body 3.

[0070] The boom angle sensor S1 is one of the posture detection sensors, and is configured to detect the rotation angle of the boom 4. In this embodiment, the boom angle sensor S1 is a stroke sensor that detects the stroke amount of the boom cylinder 7, and derives the rotation angle of the boom 4 around a boom foot pin that connects the upper rotating body 3 and the boom 4 based on the stroke amount of the boom cylinder 7.

[0071] The arm angle sensor S2 is one of the posture detection sensors, and is configured to detect the rotation angle of the arm 5. In this embodiment, the arm angle sensor S2 is a stroke sensor that detects the stroke amount of the arm cylinder 8, and derives the rotation angle of the arm 5 around the connecting pin that connects the boom 4 and the arm 5 based on the stroke amount of the arm cylinder 8.

[0072] The bucket angle sensor S3 is one of the posture detection sensors, and is configured to detect the rotation angle of the bucket 6. In this embodiment, the bucket angle sensor S3 is a stroke sensor that detects the stroke amount of the bucket cylinder 9, and derives the rotation angle of the bucket 6 around a connecting pin that connects the arm 5 and the bucket 6 based on the stroke amount of the bucket cylinder 9.

[0073] The rotation angular velocity sensor S4 is configured to detect the rotation angular velocity of the upper rotating body 3. In this embodiment, the rotation angular velocity sensor S4 is a gyro sensor. The rotation angular velocity sensor S4 may be configured to calculate a rotation angle based on the rotation angular velocity. The rotation angular velocity sensor S4 may be configured with another sensor such as a rotary encoder.

[0074] The upper rotating body 3 is equipped with a cabin 10 serving as a driver's room, an engine 11, a positioning device 18, a sound collecting device A1, an imaging device C1, a communication device T1, and the like. A controller 30 is also installed inside the cabin 10. A driver's seat, an operating device, and the like are also installed inside the cabin 10. However, the excavator 100S may be an unmanned excavator in which the cabin 10 is omitted.

[0075] The engine 11 is a drive source of the excavator 100S. In this embodiment, the engine 11 is a diesel engine. An output shaft of the engine 11 is connected to input shafts of a main pump 14 and a pilot pump 15 (see FIG. 6).

[0076] The positioning device 18 is configured to measure the position of the excavator 100S. In this embodiment, the positioning device 18 is a GNSS compass, and is configured to be able to measure the position and orientation of the upper rotating body 3.

[0077] The sound collector A1 is configured to collect sounds generated around the excavator 100S. In this embodiment, the sound collector A1 is a microphone attached to the upper rotating body 3.

[0078] The imaging device C1 is configured to capture an image of the periphery of the excavator 100S. In this embodiment, the imaging device C1 includes a rear camera C1B attached to the rear end of the upper surface of the upper rotating body 3, a front camera C1F attached to the front end of the upper surface of the cabin 10, a left camera C1L attached to the left end of the upper surface of the upper rotating body 3, and a right camera C1R attached to the right end of the upper surface of the upper rotating body 3. The imaging device C1 may be a celestial sphere camera installed at a predetermined position in the cabin 10. The predetermined position is, for example, a position corresponding to the eye position of an operator seated in a driver's seat installed in the cabin 10.

[0079] The communication device T1 is configured to control communication with a device external to the shovel 100S. In this embodiment, the communication device T1 is configured to control wireless communication between the communication device T1 and a device external to the shovel 100S via a wireless communication network. The controller 30 is a calculation device that executes various calculations as described above.

[0080] Fig. 6 is a block diagram showing a configuration example of a drive system mounted on the excavator 100S of Fig. 5. In Fig. 6, mechanical power transmission lines are indicated by double lines, hydraulic oil lines by thick solid lines, pilot lines by dashed lines, and electrical control lines by dotted lines.

[0081] The drive system of the excavator 100S is made up of an engine 11, a regulator 13, a main pump 14, a pilot pump 15, a control valve unit 17, a controller 30, and a solenoid valve unit 45. The engine 11 is controlled by an engine control unit 74.

[0082] The main pump 14 supplies hydraulic oil to a control valve unit 17 via a hydraulic oil line 16. In this embodiment, the main pump 14 is a swash plate type variable displacement hydraulic pump.

[0083] The regulator 13 is configured to control the discharge amount of the main pump 14. In this embodiment, the regulator 13 is configured to adjust the swash plate tilt angle of the main pump 14 in response to the discharge pressure of the main pump 14 or a control signal from the controller 30. The discharge amount (displacement volume) of the main pump 14 per rotation is controlled by the regulator 13.

[0084] The pilot pump 15 is configured to supply hydraulic oil to various hydraulic control devices via a pilot line 25. In this embodiment, the pilot pump 15 is a fixed displacement hydraulic pump. However, the pilot pump 15 may be omitted. In this case, the function of the pilot pump 15 may be realized by the main pump 14. That is, the main pump 14 may have a function of supplying hydraulic oil to the solenoid valve unit 45 and the like via a throttle or the like, in addition to the function of supplying hydraulic oil to the control valve unit 17.

[0085] The control valve unit 17 is configured to selectively supply hydraulic oil received from the main pump 14 to one or more hydraulic actuators. In this embodiment, the control valve unit 17 includes a plurality of control valves corresponding to the plurality of hydraulic actuators. The control valve unit 17 is configured to selectively supply hydraulic oil discharged from the main pump 14 to one or more hydraulic actuators. The hydraulic actuators include, for example, the boom cylinder 7, the arm cylinder 8, the bucket cylinder 9, the left traveling hydraulic motor 1L, the right traveling hydraulic motor 1R, and the swing hydraulic motor 2A.

[0086] The controller 30 is configured to control the solenoid valve unit 45 based on an operation signal received through the communication device T1. In this embodiment, the operation signal is transmitted from a remote control room located at a remote location away from the work site WS. The operation signal may be generated by an operation device provided in the cabin 10.

[0087] The solenoid valve unit 45 includes a plurality of solenoid valves arranged in each pilot line 25 that connects the pilot pump 15 and the pilot port of each control valve in the control valve unit 17 .

[0088] In this way, the controller 30 can realize operations such as raising and lowering the boom 4, opening and closing the arm 5, opening and closing the bucket 6, rotating the upper rotating body 3, and traveling the lower traveling body 1 in response to an operation signal from outside, such as a remote operation room. That is, in the excavator 100S, the drive system shown in Fig. 6 constitutes the drive device D in the work machine 100 shown in Fig. 1.

[0089] Fig. 7 is a block diagram showing an example of the configuration of an electrical system mounted on the excavator 100S of Fig. 5. As shown in Fig. 6, the engine 11 is connected to an engine control unit 74. Various data indicating the state of the engine 11 is transmitted from the engine control unit 74 to the controller 30. The controller 30 is configured to be able to accumulate various data indicating the state of the engine 11 in a memory 30a.

[0090] The battery 70 is configured to supply electric power to various electric loads mounted on the excavator 100S. The alternator 11a (generator), the starter 11b, the controller 30, the electric components 72, etc. are configured to operate with electric power stored in the battery 70. The starter 11b is configured to be driven by the electric power stored in the battery 70 and start the engine 11. The battery 70 is also configured to be charged with electric power generated by the alternator 11a.

[0091] The water temperature sensor 11c transmits data related to the temperature of the engine coolant to the controller 30. The regulator 13 transmits data related to the swash plate tilt angle to the controller 30. The discharge pressure sensor 14b transmits data related to the discharge pressure of the main pump 14 to the controller 30. The positioning device 18 transmits data related to the position of the excavator 100S to the controller 30.

[0092] An oil temperature sensor 14c is provided in a pipe 14-1 between the main pump 14 and a hydraulic oil tank in which the hydraulic oil to be sucked by the main pump 14 is stored. The oil temperature sensor 14c transmits data relating to the temperature of the hydraulic oil flowing through the pipe 14-1 to the controller 30.

[0093] A urea water remaining amount sensor 21a provided in the urea water tank 21 transmits data relating to the remaining amount of urea water to the controller 30. A fuel remaining amount sensor 22a provided in the fuel tank 22 transmits data to the controller 30 relating to the remaining amount of fuel.

[0094] In the shovel 100S, the boom angle sensor S1, the arm angle sensor S2, the bucket angle sensor S3, the swing angular velocity sensor S4, the water temperature sensor 11c, the discharge pressure sensor 14b, the oil temperature sensor 14c, the urea water remaining sensor 21a, and the fuel remaining sensor 22a constitute the detection device S in the work machine 100 of Figure 1.

[0095] The communication device T1 is configured to transmit and receive information via wireless communication between the communication device 250 constituting the remote operation support device 200 installed in the remote operation room RC. In this embodiment, the communication device T1 and the communication device 250 are configured to transmit and receive information via a fifth generation mobile communication line (5G line), an LTE line, a satellite line, or the like.

[0096] The remote control room RC is provided with a sound output device A2, an indoor imaging device C2, a remote operation support device 200, and the like. The remote operation support device 200 includes an image generation device 210, a display device 220, a remote control device 230, a controller 240, and a communication device 250, and the like. The remote control room RC is also provided with a driver's seat DS in which an operator OP who remotely operates the excavator 100S sits. As described above, the image generation device 210 and the controller 240 are arithmetic devices that execute various calculations.

[0097] The sound output device A2 is configured to output sound. In this embodiment, the sound output device A2 is a speaker and is configured to reproduce the sound collected by the sound collection device A1 attached to the excavator 100S.

[0098] The indoor imaging device C2 is configured to capture an image of the inside of the remote control room RC. In this embodiment, the indoor imaging device C2 is a camera installed inside the remote control room RC, and is configured to capture an image of an operator OP seated in the driver's seat DS.

[0099] The communication device 250 is configured to control wireless communication with the communication device T1 attached to the excavator 100S.

[0100] In this embodiment, the driver's seat DS has a structure similar to that of a driver's seat installed in the cabin of a normal excavator. Specifically, a left console box is disposed on the left side of the driver's seat DS, and a right console box is disposed on the right side of the driver's seat DS. A left operation lever is disposed on the front end of the upper surface of the left console box, and a right operation lever is disposed on the front end of the upper surface of the right console box. In addition, a travel lever and a travel pedal are disposed in front of the driver's seat DS. Furthermore, an engine speed adjustment dial 75 is disposed in the center of the upper surface of the right console box. The left operation lever, the right operation lever, the travel lever, the travel pedal, and the engine speed adjustment dial 75 each constitute a remote operation device 230.

[0101] The engine speed adjustment dial 75 is a dial for adjusting the speed of the engine 11, and is configured to be able to change the engine speed in four steps, for example.

[0102] Specifically, the engine speed adjustment dial 75 is configured to be able to switch the engine speed among four stages: SP mode, H mode, A mode, and idling mode. The engine speed adjustment dial 75 transmits data relating to the setting of the engine speed to the controller 30.

[0103] The SP mode is a rotation speed mode selected when the operator OP wants to prioritize the amount of work, and uses the highest engine speed. The H mode is a rotation speed mode selected when the operator OP wants to balance the amount of work and fuel efficiency, and uses the second highest engine speed. The A mode is a rotation speed mode selected when the operator OP wants to operate the excavator with low noise while prioritizing fuel efficiency, and uses the third highest engine speed. The idling mode is a rotation speed mode selected when the operator OP wants to put the engine into an idling state, and uses the lowest engine speed. The engine 11 is constantly controlled at the engine speed of the rotation speed mode selected via the engine speed adjustment dial 75.

[0104] The remote control device 230 is provided with an operation sensor 29 for detecting the operation content of the remote control device 230. The operation sensor 29 is, for example, an inclination sensor that detects the inclination angle of the operation lever, or an angle sensor that detects the swing angle of the operation lever around the swing axis. The operation sensor 29 may be composed of other sensors such as a pressure sensor, a current sensor, a voltage sensor, or a distance sensor. The operation sensor 29 outputs information on the detected operation content of the operation device 26 to the controller 240. The controller 240 generates an operation signal based on the received information and transmits the generated operation signal to the excavator 100S. The operation sensor 29 may be configured to generate an operation signal. In this case, the operation sensor 29 may output the operation signal to the communication device 250 without passing through the controller 240.

[0105] The display device 220 is configured to display information related to the situation around the shovel 100S. In this embodiment, the display device 220 is a multi-display made up of nine monitors in three vertical rows and three horizontal columns, and is configured to be able to display the state of the space in front, to the left, and to the right of the shovel 100S. Each monitor is a liquid crystal monitor, an organic EL monitor, or the like. However, the display device 220 may be made up of one or more curved monitors, or may be made up of a projector.

[0106] The display device 220 may be a display device that can be worn by the operator OP. For example, the display device 220 may be a head-mounted display configured to transmit and receive information to and from the controller 240 by wireless communication. The head-mounted display may be connected to the controller 240 by wire. The head-mounted display may be a transparent head-mounted display or a non-transparent head-mounted display. The head-mounted display may be a monocular head-mounted display or a binocular head-mounted display.

[0107] The display device 220 is configured to display an image that enables the operator OP in the remote control room RC to visually recognize the surroundings of the shovel 100S. In other words, the display device 220 displays an image so that the operator can confirm the surroundings of the shovel 100S as if he or she were in the cabin 10 of the shovel 100S, despite being in the remote control room RC.

[0108] As described above, the image generating device 210 is a device that generates an image to support remote operation of the shovel 100S as the work machine 100 equipped with an excavation attachment as the work attachment WA. The image generating device 210 generates a composite image G including a main image G1 included in an image captured by the imaging device C1 at the work site WS where the shovel 100S is working, and a complementary image G2 that complements the outside of the imaging range SR1 of the main image G1. The display device 220 displays an image that includes at least a part of the composite image G generated by the image generating device 210.

[0109] Next, the relationship between a first coordinate system having a reference point R1 in the remote control room RC as its origin and a second coordinate system having a reference point R2 in the excavator 100S as its origin will be described with reference to Fig. 8. In the following description, the first coordinate system will be referred to as the control room coordinate system, and the second coordinate system will be referred to as the excavator coordinate system. Fig. 8 is a diagram showing the relationship between the control room coordinate system and the excavator coordinate system.

[0110] The control room coordinate system is a three-dimensional UVW Cartesian coordinate system with its origin at reference point R1 in the remote control room RC, and has a U axis that extends parallel to the fore-and-aft direction of the driver's seat DS, a V axis that extends parallel to the left-and-right direction of the driver's seat DS, and a W axis that is perpendicular to the U and V axes.

[0111] The excavator coordinate system is a three-dimensional XYZ Cartesian coordinate system with the reference point R2 on the upper rotating body 3 as the origin, and has an X axis extending parallel to the front-rear direction of the upper rotating body 3, a Y axis extending parallel to the left-right direction of the upper rotating body 3, and a Z axis perpendicular to the X and Y axes. In the example of FIG. 8, the reference point R2 is a point on the rotation axis, the XY plane is a horizontal plane, and the Z axis is a vertical axis. That is, in the example of FIG. 8, the ground surface, which is a virtual plane on which the excavator 100S is located, is a horizontal plane. The X axis corresponds to the U axis of the operation room coordinate system, the Y axis corresponds to the V axis of the operation room coordinate system, and the Z axis corresponds to the W axis of the operation room coordinate system.

[0112] In this embodiment, each three-dimensional coordinate in the operation room coordinate system is previously associated with one of the three-dimensional coordinates in the shovel coordinate system. Therefore, once the three-dimensional coordinates of the operator viewpoint E1, which is the eye position of the operator OP in the remote operation room RC, are determined, the three-dimensional coordinates of the virtual operator viewpoint E1', which is the eye position of the virtual operator in the shovel 100S, are uniquely determined. Note that the eye position of the operator OP is, for example, the midpoint between the left eye position and the right eye position of the operator OP. However, the eye position of the operator OP may be a position set in advance. In other words, the operator viewpoint E1 and the virtual operator viewpoint E1' may be fixed points.

[0113] Fig. 9 is a diagram showing an example of the configuration of the display device 220. As shown in Fig. 9, the display device 220 is a multi-display consisting of nine monitors in three vertical rows and three horizontal columns. Specifically, the display device 220 includes a center monitor D1a, an upper monitor D1b, a lower monitor D1c, a left monitor D1d, a right monitor D1e, an upper left monitor D1f, an upper right monitor D1g, a lower left monitor D1h, and a lower right monitor D1i.

[0114] The display device 220 displays the main image G1 described with reference to Figures 2 and 3 on at least a portion of the central monitor D1a, upper monitor D1b, and lower monitor D1c, which correspond to the central visual field of the operator OP in the remote control room RC. The display device 220 also displays the complementary image G2 described with reference to Figures 2 and 3 on at least a portion of the left monitor D1d, right monitor D1e, upper left monitor D1f, upper right monitor D1g, lower left monitor D1h, and lower right monitor D1i, which correspond to the peripheral visual field of the operator OP in the remote control room RC.

[0115] With this configuration, from an image captured at the work site WS by the imaging device C1 attached to the shovel 100S, a portion with less distortion centered on the optical axis of the imaging device C1 can be displayed as a main image G1 in the central visual field of the operator OP in the remote control room RC. Also, a complementary image G2 that complements an area SR2 outside the imaging range SR1 of the main image G1 can be displayed in the peripheral visual field of the operator OP in the remote control room RC. Also, at least one of the main image G1 and the complementary image G2 can include an image of the excavation attachment of the shovel 100S. As a result, a more realistic image of the work site WS can be provided to the operator OP who operates the shovel 100S equipped with an excavation attachment at a remote location away from the work site WS, thereby improving the work efficiency of the shovel 100S.

[0116] The preferred embodiments of the present invention have been described above in detail. However, the present invention is not limited to the above-described embodiments. Various modifications or substitutions may be applied to the above-described embodiments without departing from the scope of the present invention. Furthermore, features described separately may be combined unless technical contradictions arise.

[0117] For example, in the above embodiment, an example has been described in which the image generation device is installed in a remote location away from the work site. However, the image generation device may be mounted on a work machine. In this case, a composite image generated by the image generation device can be transmitted from the work machine at the work site, received at a remote location where a remote control device is installed, and displayed on a display device.

[0118] The image generating device may generate a composite image without using machine learning. For example, the image generating device may extract a past image of an imaging device that has been stored in association with the position and orientation information of the work machine according to the current position and orientation information of the work machine, and display the extracted image as a complementary image. [Explanation of symbols]

[0119] 210... Image generating device 220...Display device 230 Remote control device 250 Communication device (receiving device) C1 Imaging device DT···Track (Object) G Composite image G1...Main image G2 Complementary image LM: trained model M: Other work machines (objects) P···Cone (Object) SR1: Main image capture range SR2: Adjacent areas on the left and right SW...Worker (object) SYS: Remote operation support system T1: Communication equipment (transmitter) WS···Worksite

Claims

1. An image generating device that generates an image to assist in remote operation of a work machine equipped with a work attachment, generating a composite image including a main image included in an image captured by an imaging device at a work site where the work machine is working, and a complementary image that complements an area outside an imaging range of the main image; Image generating device.

2. The complementary image is an image that complements adjacent regions on the left and right sides of an imaging range of the main image.

2. The image generating device of claim 1.

3. A trained model is provided in which an image outside the imaging range of the main image captured by the imaging device at the work site and an image including the main image are trained as teacher data, The main image is input to the trained model to generate the synthetic image.

2. The image generating device of claim 1.

4. generating the complementary image including an image of the object based on information of the object recognized from the image captured by the imaging device at the work site and position and orientation information of a work machine working at the work site; 2. The image generating device of claim 1.

5. generating the complementary image including an image of the work machine based on the shape, dimensions, and posture of a work machine working at the work site, and the positional relationship between the work machine and the imaging device; 2. The image generating device of claim 1.

6. a difference between the feature amount of the main image and the feature amount of the complement image is equal to or smaller than a predetermined threshold value; 2. The image generating device of claim 1.

7. A remote operation support system for a work machine including a work machine equipped with a work attachment and a remote operation support device that supports remote operation of the work machine, The remote operation support device includes: An image generating device according to any one of claims 1 to 6; a display device that displays the composite image generated by the image generating device at a remote location away from the work site; and a remote operation device that operates the work machine working at the work site from the remote location. A remote operation support system for work machines.

8. The work machine includes a transmission device that transmits the main image from the work site by wireless communication, the remote operation support device further includes a receiving device that receives the main image transmitted from the transmitting device at the remote location and outputs the main image to the image generating device disposed at the remote location. The remote operation support system for a work machine according to claim 7.

Images