Remote operation system for working machine and remote operation method for working machine

Abstract

To flexibly update or improve a periphery monitoring function of a work machine in remote control of the work machine.SOLUTION: In a work machine remote control system 100, a control device 60 comprises: an image data receiving unit 62 for receiving image data showing a peripheral image of a work machine 1; and an object detection unit 63 for detecting a person existing in the periphery of the work machine 1 based on the image data.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present disclosure relates to a remote operation system for a working machine and a method for remotely operating a working machine.

Background Art

[0002] In the technical field related to working machines, working machines equipped with a peripheral monitoring device as disclosed in Patent Document 1 are known. By providing information around the working machine to the operator of the working machine, a decrease in working efficiency is suppressed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Also, in the technical field related to working machines, a technology for remotely operating a working machine is known. When remotely operating a working machine, if a peripheral monitoring function of the working machine is provided in the working machine, it may be difficult to flexibly update or improve the peripheral monitoring function of the working machine.

[0005] An object of the present disclosure is to flexibly update or improve a peripheral monitoring function of a working machine in remote operation of the working machine.

Means for Solving the Problems

[0006] According to the present disclosure, there is provided a remote operation system for a working machine, including an image data receiving unit that receives image data showing a peripheral image of the working machine, and an object detection unit that detects a person existing around the working machine based on the image data, at a remote operation location of the working machine.

Effects of the Invention

[0007] According to the present disclosure, in the remote operation of a working machine, the update or improvement of the peripheral monitoring function of the working machine can be flexibly implemented.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Modes for Carrying Out the Invention

[0009] Hereinafter, embodiments according to the present disclosure will be described with reference to the drawings, but the present disclosure is not limited thereto. The components of the embodiments described below can be appropriately combined. Also, some components may not be used.

[0010] [Remote Operation System] FIG. 1 is a schematic diagram showing a remote operation system 100 of a working machine 1 according to an embodiment. The remote operation system 100 remotely operates the working machine 1 existing at a work site. At least a part of the remote operation system 100 is arranged in a remote operation room 200 at a remote operation location. The remote operation system 100 includes a remote operation device 40, a display device 50, and a control device 60.

[0011] The remote operation device 40 is arranged in the remote operation room 200 outside the working machine 1. The remote operation device 40 is operated by an operator in the remote operation room 200. The operator can operate the remote operation device 40 while sitting on the operator's seat 45.

[0012] The display device 50 is arranged in the remote operation room 200 outside the working machine 1. The display device 50 displays an image of the work site. The image of the work site includes a surrounding image which is an image around the working machine 1. The surrounding image of the working machine 1 includes at least an image of the construction target of the working machine 1. In the embodiment, the image of the construction target of the working machine 1 includes an image of the excavation target of the working machine 4.

[0013] The operator operates the remote operation device 40 while looking at the image of the work site displayed on the display device 50. The working machine 1 is remotely operated by the remote operation device 40.

[0014] The control device 60 is arranged in the remote operation room 200 outside the working machine 1. The control device 60 includes a computer system.

[0015] The working machine 1 includes a control device 300. The control device 300 includes a computer system.

[0016] The control device 60 and the control device 300 communicate with each other via a communication system 400. Examples of the communication system 400 include the Internet, a local area network (LAN), a mobile phone communication network, and a satellite communication network.

[0017] [Working Machine] FIG. 2 is a perspective view showing a working machine 1 according to an embodiment. In the embodiment, the working machine 1 is assumed to be a hydraulic excavator. The working machine 1 operates at a work site.

[0018] As shown in FIG. 2, the working machine 1 includes a traveling body 2, a revolving body 3 supported by the traveling body 2, a working device 4 supported by the revolving body 3, a hydraulic cylinder 5 for driving the working device 4, and an imaging device 30.

[0019] The traveling body 2 is capable of traveling while supporting the revolving body 3. The traveling body 2 has a pair of crawlers. The traveling body 2 travels by rotation of the crawlers. The revolving body 3 is capable of revolving about a revolving axis RX while being supported by the traveling body 2.

[0020] The working device 4 includes a boom 4A connected to the revolving body 3, an arm 4B connected to the boom 4A, and a bucket 4C connected to the arm 4B. The hydraulic cylinder 5 includes a boom cylinder 5A for driving the boom 4A, an arm cylinder 5B for driving the arm 4B, and a bucket cylinder 5C for driving the bucket 4C.

[0021] The boom 4A is supported by the revolving body 3 so as to be rotatable about a boom rotation axis AX. The arm 4B is supported by the boom 4A so as to be rotatable about an arm rotation axis BX. The bucket 4C is supported by the arm 4B so as to be rotatable about a bucket rotation axis CX.

[0022] The boom rotation axis AX, the arm rotation axis BX, and the bucket rotation axis CX are parallel to each other. The boom rotation axis AX and an axis parallel to the revolving axis RX are orthogonal to each other. In the following description, the direction parallel to the revolving axis RX is appropriately referred to as the vertical direction, the direction parallel to the boom rotation axis AX is appropriately referred to as the left - right direction, and the direction orthogonal to both the boom rotation axis AX and the revolving axis RX is appropriately referred to as the front - rear direction. The direction in which the working device 4 exists with respect to the revolving axis RX is the front, and the reverse direction of the front is the rear. One of the left - right directions with respect to the revolving axis RX is the right, and the reverse direction of the right is the left. The direction away from the ground contact surface of the traveling body 2 is the up, and the reverse direction of the up is the down.

[0023] The imaging device 30 images the work site. The imaging device 30 images at least the periphery of the working machine 1. The imaging device 30 is disposed on the revolving body 3.

[0024] The imaging device 30 images the objects existing around the working machine 1. Examples of the objects imaged by the imaging device 30 include the construction target of the working machine 1, the excavation target of the working machine 4, the structures existing at the work site, at least a part of the working machine 1, a working machine different from the working machine 1, and the persons (workers) working at the work site.

[0025] [Imaging device] FIG. 3 is a perspective view showing the imaging device 30 according to the embodiment. As shown in FIG. 3, the imaging device 30 is disposed at the upper part of the front portion of the revolving body 3. The imaging device 30 acquires the peripheral image of the working machine 1.

[0026] The imaging device 30 includes an optical system and an image sensor that receives the light that has passed through the optical system. The image sensor includes a CCD (Couple Charged Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0027] In the embodiment, the imaging device 30 includes a front camera 31 that images the front of the revolving body 3, a left camera 32 that images the left side of the revolving body 3, a right camera 33 that images the right side of the revolving body 3, an upper camera 34 that images the diagonally upper front of the revolving body 3, and a lower camera 35 that images the diagonally lower front of the revolving body 3. The front camera 31, the left camera 32, the right camera 33, the upper camera 34, and the lower camera 35 image the periphery of the working machine 1 simultaneously.

[0028] [Remote operation room] FIG. 4 is a view showing the remote operation room 200 according to the embodiment. As shown in FIG. 4, a remote operation device 40 and a display device 50 are disposed in the remote operation room 200.

[0029] The remote control device 40 is operated by an operator seated on the operator's seat 45. The operator sits on the operator's seat 45 so as to face the display screen of the display device 50. The operator operates the remote control device 40 while looking at the display screen of the display device 50.

[0030] The operation signal generated by operating the remote control device 40 is transmitted to the control device 300 of the working machine 1 via the control device 60 and the communication system 400. The control device 300 operates the working machine 1 based on the operation signal acquired via the communication system 400. The operation of the working machine 1 includes at least one of the operations of the traveling body 2, the slewing body 3, and the working implement 4.

[0031] The operation of the traveling body 2 includes the forward and backward operations of the traveling body 2. The operation of the slewing body 3 includes the left and right slewing operations of the slewing body 3. The operation of the working implement 4 includes the raising operation of the boom 4A, the lowering operation of the boom 4A, the dumping operation of the arm 4B, the excavation operation of the arm 4B, the excavation operation of the bucket 4C, and the dumping operation of the bucket 4C.

[0032] The remote control device 40 includes a left working lever 41 and a right working lever 42 that are operated for the operations of the slewing body 3 and the working implement 4, and a left traveling lever 43 and a right traveling lever 44 that are operated for the operation of the traveling body 2.

[0033] The left working lever 41 is disposed to the left of the operator's seat 45. The right working lever 42 is disposed to the right of the operator's seat 45. When the left working lever 41 is operated in the front-rear direction, the arm 4B performs a dumping operation or an excavation operation. When the left working lever 41 is operated in the left-right direction, the slewing body 3 performs a left slewing operation or a right slewing operation. When the right working lever 42 is operated in the left-right direction, the bucket 4C performs an excavation operation or a dumping operation. When the right working lever 42 is operated in the front-rear direction, the boom 4A performs a lowering operation or a raising operation. Note that when the left working lever 41 is operated in the front-rear direction, the slewing body 3 may perform a right slewing operation or a left slewing operation, and when the left working lever 41 is operated in the left-right direction, the arm 4B may perform a dumping operation or an excavation operation.

[0034] The left travel lever 43 and the right travel lever 44 are arranged in front of the operator's seat 45. The left travel lever 43 is arranged to the left of the right travel lever 44. When the left travel lever 43 is operated in the front-rear direction, the crawler on the left side of the traveling body 2 performs a forward movement or a backward movement. When the right travel lever 44 is operated in the front-rear direction, the crawler on the right side of the traveling body 2 performs a forward movement or a backward movement.

[0035] The display device 50 displays a peripheral image of the work machine 1 captured by the imaging device 30. Image data indicating the peripheral image of the work machine 1 acquired by the imaging device 30 is transmitted to the control device 60 of the remote operation system 100 via the control device 300 and the communication system 400. The control device 60 causes the display device 50 to display the peripheral image acquired via the communication system 400.

[0036] The display device 50 includes a flat panel display such as a liquid crystal display (LCD) or an organic EL display (OELD). In an embodiment, the display device 50 includes a plurality of flat panel displays arranged adjacent to each other. In an embodiment, the display device 50 includes a central display 51, a left display 52 arranged to the left of the central display 51, a right display 53 arranged to the right of the central display 51, an upper display 54 arranged above the central display 51, and a lower display 55 arranged below the central display 51.

[0037] The peripheral image acquired by the front camera 31 is displayed on the central display 51. The peripheral image acquired by the left camera 32 is displayed on the left display 52. The peripheral image acquired by the right camera 33 is displayed on the right display 53. The peripheral image acquired by the upper camera 34 is displayed on the upper display 54. The peripheral image acquired by the lower camera 35 is displayed on the lower display 55.

[0038] The display device 50 displays a peripheral image corresponding to the operator's field of view when the operator is assumed to be seated on the driver's seat of the work machine 1. The central display 51, the left display 52, the right display 53, the upper display 54, and the lower display 55 are arranged so as to surround the operator's seat 45. The central display 51 displays a peripheral image corresponding to the field of view in front of the operator when the operator is assumed to be seated on the driver's seat of the work machine 1. The left display 52 displays a peripheral image corresponding to the field of view to the left of the operator when the operator is assumed to be seated on the driver's seat of the work machine 1. The right display 53 displays a peripheral image corresponding to the field of view to the right of the operator when the operator is assumed to be seated on the driver's seat of the work machine 1. The upper display 54 displays a peripheral image corresponding to the field of view diagonally above in front of the operator when the operator is assumed to be seated on the driver's seat of the work machine 1. The lower display 55 displays a peripheral image corresponding to the field of view diagonally below in front of the operator when the operator is assumed to be seated on the driver's seat of the work machine 1. The operator present in the remote control room 200 can obtain a feeling of actually sitting on the driver's seat of the work machine 1.

[0039] In the example shown in FIG. 4, the central display 51 displays the arm 4B and the bucket 4C for excavating the excavation target. The left display 52 displays the dump truck 70, which is a transport vehicle into which the excavated material excavated by the work machine 4 is loaded. The right display 53 displays the boom 4A. The upper display 54 displays the upper end portion of the work machine 4. The lower display 55 displays a part of the traveling body 2.

[0040] [Control device] FIG. 5 is a functional block diagram showing the remote operation system 100 of the working machine 1 according to the embodiment. As shown in FIG. 5, the remote operation system 100 includes a communication device 6 disposed at a remote operation location, a control device 60 connected to the communication device 6, a remote operation device 40 connected to the control device 60, a display device 50 connected to the control device 60, and an alarm device 80 connected to the control device 60. Further, the remote operation system 100 includes a communication device 7 disposed on the working machine 1, a control device 300 connected to the communication device 7, an imaging device 30 connected to the control device 300, a traveling body 2 controlled by the control device 300, a slewing body 3 controlled by the control device 300, and a hydraulic cylinder 5 controlled by the control device 300.

[0041] The control device 300 has a traveling control unit 301, a slewing control unit 302, a work machine control unit 303, and an image data transmission unit 304.

[0042] The traveling control unit 301 receives the operation signal of the remote operation device 40 transmitted from the control device 60. The traveling control unit 301 outputs a control signal for controlling the operation of the traveling body 2 based on the operation signal of the remote operation device 40.

[0043] The slewing control unit 302 receives the operation signal of the remote operation device 40 transmitted from the control device 60. The slewing control unit 302 outputs a control signal for controlling the operation of the slewing body 3 based on the operation signal of the remote operation device 40.

[0044] The work machine control unit 303 receives the operation signal of the remote operation device 40 transmitted from the control device 60. The work machine control unit 303 outputs a control signal for controlling the operation of the work machine 4 based on the operation signal of the remote operation device 40. The control signal for controlling the work machine 4 includes a control signal for controlling the hydraulic cylinder 5.

[0045] The image data transmission unit 304 transmits the image data indicating the peripheral image of the working machine 1 acquired by the imaging device 30 to the control device 60.

[0046] The communication device 7 communicates with the communication device 6 via the communication system 400. The communication device 7 receives the operation signal of the remote control device 40 transmitted from the control device 60 via the communication device 6 and outputs it to the control device 300. The communication device 7 transmits the image data indicating the peripheral image of the working machine 1 received from the image data transmission unit 304 to the communication device 6. The communication device 7 includes an encoder that compresses the image data. The image data is transmitted from the communication device 7 to the communication device 6 in a compressed state.

[0047] The communication device 6 communicates with the communication device 7 via the communication system 400. The communication device 6 transmits the operation signal generated by operating the remote control device 40 to the communication device 7. The communication device 6 receives the image data transmitted from the control device 300 via the communication device 7 and outputs it to the control device 60. The communication device 6 includes a decoder that restores the compressed image data. The image data is output from the communication device 6 to the control device 60 in a restored state.

[0048] The control device 60 includes an operation signal transmission unit 61, an image data reception unit 62, an object detection unit 63, a learning unit 64, a display control unit 65, and an alarm output unit 66.

[0049] The operation signal transmission unit 61 transmits an operation signal for remotely operating the working machine 1. When the remote control device 40 is operated by the operator, an operation signal for remotely operating the working machine 1 is generated. The operation signal transmission unit 61 transmits the operation signal of the remote control device 40 to the control device 300.

[0050] The image data reception unit 62 receives the image data indicating the peripheral image of the working machine 1. The image data reception unit 62 acquires the image data restored by the decoder of the communication device 6.

[0051] The object detection unit 63 detects an object existing around the working machine 1 based on the image data indicating the peripheral image of the working machine 1 received by the image data reception unit 62. In the embodiment, the object detection unit 63 detects a person as an object existing around the working machine 1.

[0052] In an embodiment, the object detection unit 63 detects a person by using artificial intelligence (AI) that analyzes input data by an algorithm and outputs output data. In the embodiment, the input data is image data of a peripheral image, and the output data is a person.

[0053] The learning unit 64 learns feature amounts of a person and generates a learning model. The object detection unit 63 inputs the image data of the peripheral image into the learning model to detect a person.

[0054] FIG. 6 is a functional block diagram showing the object detection unit 63 and the learning unit 64 according to the embodiment. As shown in FIG. 6, a plurality of learning images are input to the learning unit 64. The learning images include images of persons. The learning unit 64 generates a learning model that takes the feature amounts of an object as an input and outputs a person (the presence or absence of a person) by performing machine learning using the learning images including persons as teacher data.

[0055] The object detection unit 63 includes a feature amount extraction unit 63A, a processing unit 63B, and an output unit 63C.

[0056] The peripheral image acquired by the image data reception unit 62 is input to the feature amount extraction unit 63A. The feature amount extraction unit 63A extracts the feature amounts of an object from the peripheral image.

[0057] The processing unit 63B inputs the feature amounts of the object extracted by the feature amount extraction unit 63A into the learning model to detect a person in the peripheral image. Further, when a person exists in the peripheral image, the processing unit 63B may detect the position of the person in the peripheral image.

[0058] The output unit 63C outputs the person detected by the processing unit 63B to the display control unit 65.

[0059] The display control unit 65 causes the display device 50 to display the peripheral image of the working machine 1 acquired by the image data reception unit 62. Further, the display control unit 65 causes the display device 50 to display an image showing the detection result of the object detection unit 63 together with the peripheral image of the working machine 1. As an image showing the detection result of the object detection unit 63, a symbol image that emphasizes a person is exemplified. Note that the symbol image may be distance data indicating the distance between the working machine and the person.

[0060] When a person is detected by the object detection unit 63, the alarm output unit 66 causes the alarm device 80 to output an alarm. Examples of the alarm device 80 include an audio output device, a buzzer, a light emitting device, and a lamp. When a person is detected by the object detection unit 63, the alarm output unit 66 outputs a control signal for driving the alarm device 80 to cause the alarm device 80 to output an alarm.

[0061] [Remote operation method] FIG. 7 is a flowchart showing a remote operation method of the working machine 1 according to the embodiment. Hereinafter, a method for processing image data showing the peripheral image of the working machine 1 will be mainly described. As described above, when the remote operation device 40 is operated, an operation signal for remotely operating the working machine 1 is transmitted from the operation signal transmission unit 61 of the control device 60 to the control device 300.

[0062] The imaging device 30 images the periphery of the working machine 1. The image data transmission unit 304 transmits image data showing the peripheral image of the working machine 1 to the control device 60 via the communication device 7 and the communication system 400 (step SA1).

[0063] The image data reception unit 62 receives the image data of the peripheral image transmitted from the working machine 1 via the communication device 6.

[0064] The learning unit 64 generates a learning model that takes the feature amount of an object as an input and outputs a person. The object detection unit 63 inputs the image data of the peripheral image received by the image data reception unit 62 into the learning model (step SB1).

[0065] In the object detection unit 63, the feature amount extraction unit 63A extracts the feature amount of the object from the peripheral image. The processing unit 63B inputs the feature amount of the object extracted by the feature amount extraction unit 63A into the learning model. The processing unit 63B inputs the feature amount of the object into the learning model to detect a person (step SB2).

[0066] Also, the processing unit 63B specifies the position of the person in the peripheral image.

[0067] The output unit 63C outputs the person detected by the processing unit 63B to the display control unit 65.

[0068] The display control unit 65 generates a symbol image indicating the person output from the output unit 63C. The symbol image is an image for emphasizing the person in the peripheral image.

[0069] The display control unit 65 causes the display device 50 to display the peripheral image acquired by the image data receiving unit 62 and the symbol image indicating the person detected by the object detection unit 63 (step SB3).

[0070] FIG. 8 is a diagram showing the display device 50 according to the embodiment. As shown in FIG. 8, the display control unit 65 causes the display device 50 to display together the peripheral image of the working machine 1 and the symbol image 8 indicating the person WM. The display control unit 65 superimposes and displays the symbol image 8 on the person WM in the peripheral image detected by the processing unit 63B of the object detection unit 63 in step SB2. Note that the symbol image 8 may be displayed adjacent to the person WM.

[0071] In the example shown in FIG. 8, the symbol image 8 is in a frame shape surrounding the person WM in the peripheral image. Note that the shape of the symbol image 8 is arbitrary as long as it can emphasize the person WM in the peripheral image.

[0072] By emphasizing the person WM by the symbol image 8, the operator in the remote operation room 200 can smoothly recognize the presence of the person WM around the working machine 1.

[0073] The display control unit 65 changes the display form of the symbol image 8 based on the position of the person WM. The position of the person WM includes the relative position of the person WM with respect to the working machine 1. In the embodiment, the display control unit 65 changes the display form of the symbol image 8 based on the distance from the working machine 1 to the person WM.

[0074] In the example shown in FIG. 8, a person WM1 exists at a position closer to the working machine 1 than the first distance, a person WM2 exists at a position between the first distance and a second distance farther than the first distance from the working machine 1, and a person WM3 exists at a position between the second distance and a third distance farther than the second distance from the working machine 1.

[0075] The display control unit 65 changes, for example, the color of the symbol image 8 based on the distance from the working machine 1 to the person WM. The display control unit 65 causes the symbol image 8A superimposed on the person WM1 to be displayed in a first color. The display control unit 65 causes the symbol image 8B superimposed on the person WM2 to be displayed in a second color. The display control unit 65 causes the symbol image 8C superimposed on the person WM3 to be displayed in a third color. The first color is, for example, red, the second color is, for example, yellow, and the third color is, for example, blue.

[0076] By changing the display form of the symbol image 8 based on the distance from the working machine 1 to the person WM, the operator in the remote operation room 200 can recognize the distance from the working machine 1 to the person WM by looking at the symbol image 8.

[0077] Note that the display form of the symbol image 8 described with reference to FIG. 8 is an example. The display control unit 65 may change, for example, the size or shape of the symbol image 8 based on the distance from the working machine 1 to the person WM. Also, in the example shown in FIG. 8, the distance from the working machine 1 is divided into three distances: the first distance, the second distance, and the third distance. The number of distance divisions may be any number. The number of distance divisions may be two or less, or any plurality of four or more.

[0078] When the object detection unit 63 detects the person WM, the alarm output unit 66 outputs a control signal for driving the alarm device 80. When an alarm is output from the alarm device 80, the operator in the remote operation room 200 can recognize that there is a person WM around the working machine 1.

[0079] The alarm output unit 66 may change the driving state of the alarm device 80 based on the distance from the working machine 1 to the person WM. For example, when the alarm device 80 includes an audio output device or a buzzer, if a person WM1 exists at a position closer to the working machine 1 than a first distance, the alarm output unit 66 drives the alarm device 80 so that an alarm of a first volume is output. If a person WM2 exists at a position between the first distance and a second distance farther than the first distance from the working machine 1, the alarm output unit 66 drives the alarm device 80 so that an alarm of a second volume smaller than the first volume is output. If a person WM3 exists at a position between the second distance and a third distance farther than the second distance from the working machine 1, the alarm output unit 66 does not drive the alarm device 80.

[0080] [Computer System] FIG. 9 is a block diagram showing a computer system 1000 according to an embodiment. The above-described control device 60 includes the computer system 1000. The computer system 1000 includes a processor 1001 such as a CPU (Central Processing Unit), a main memory 1002 including a non-volatile memory such as a ROM (Read Only Memory) and a volatile memory such as a RAM (Random Access Memory), a storage 1003, and an interface 1004 including an input / output circuit. The functions of the above-described control device 60 are stored in the storage 1003 as a computer program. The processor 1001 reads the computer program from the storage 1003 and expands it in the main memory 1002, and executes the above-described processing according to the program. Note that the computer program may be distributed to the computer system 1000 via a network.

[0081] A computer program or computer system 1000 can receive image data showing a peripheral image of the working machine 1 and detect a person existing around the working machine 1 based on the image data, according to the above-described embodiments.

[0082] [Effect] As described above, the object detection unit 63 is provided in the control device 60 disposed in the remote operation room 200. Since the object detection unit 63 is not provided in the control device 300 of the working machine 1 but in the control device 60 disposed in the remote operation room 200, the function of the object detection unit 63 can be smoothly updated. Also, it becomes easier to use a high-performance general-purpose computer as the control device 60. That is, by providing the object detection unit 63 in the control device 60 instead of the control device 300, the update or improvement of the function of the object detection unit 63 can be flexibly implemented.

[0083] The display control unit 65 causes the display device 50 to display the symbol image 8 indicating the detection result of the object detection unit 63. Therefore, the operator in the remote operation room 200 can recognize the presence of the person WM by looking at the symbol image 8.

[0084] The object detection unit 63 detects the position of the person WM with respect to the working machine 1. The display control unit 65 superimposes the symbol image 8 on the person WM in the peripheral image based on the position of the person WM and causes the display device 50 to display it. Since the person WM is emphasized by the symbol image 8, the operator in the remote operation room 200 can recognize the presence of the person WM by looking at the symbol image 8.

[0085] The display control unit 65 changes the display form of the symbol image 8 based on the position of the person WM with respect to the working machine 1. The operator in the remote operation room 200 can recognize the distance from the working machine 1 to the person WM by looking at the symbol image 8.

[0086] When the object detection unit 63 detects the person WM, the alarm output unit 66 drives the alarm device 80. As a result, the operator in the remote operation room 200 can recognize the presence of the person WM.

[0087] [Other Embodiments] FIG. 10 is a schematic diagram showing a remote operation system 100 for a working machine 1 according to another embodiment. In the example shown in FIG. 10, the working machine 1 remotely operated by the remote operation system 100 includes a first working machine 1A, a second working machine 1B, and a third working machine 1C. The first working machine 1A operates at the first work site. The second working machine 1B operates at the second work site. The third working machine 1C operates at the third work site.

[0088] The control device 60 has a switching unit 67 that switches the transmission destination of the operation signal of the remote operation device 40 and the reception source of the image data of the peripheral image of the working machine 1. The operator in the remote operation room 200 can operate an operation switch 90 provided in the remote operation room 200. The operation signal generated by operating the operation switch 90 is input to the switching unit 67. The switching unit 67 switches the transmission destination of the operation signal of the remote operation device 40 and the reception source of the image data of the peripheral image of the working machine 1 based on the operation signal of the operation switch 90.

[0089] For example, when remotely operating the first working machine 1A, the image data of the peripheral image of the first working machine 1A is transmitted from the first working machine 1A to the control device 60. The control device 60 receives the image data of the peripheral image of the first working machine 1A and causes it to be displayed on the display device 50. The operator in the remote operation room 200 operates the remote operation device 40 to remotely operate the first working machine 1A while viewing the peripheral image of the first working machine 1A displayed on the display device 50. The operation signal of the remote operation device 40 is transmitted to the control device 300 of the first working machine 1A.

[0090] When remotely operating the second working machine 1B, the operator in the remote operation room 200 operates the operation switch 90. When the operation switch 90 is operated, the transmission destination of the operation signal of the remote operation device 40 and the reception source of the image data of the surrounding image are switched from the first working machine 1A to the second working machine 1B. The control device 60 receives the image data of the surrounding image of the second working machine 1B and causes it to be displayed on the display device 50. The operator in the remote operation room 200 operates the remote operation device 40 to remotely operate the second working machine 1B while viewing the surrounding image of the second working machine 1B displayed on the display device 50. The operation signal of the remote operation device 40 is transmitted to the control device 300 of the second working machine 1B.

[0091] When remotely operating the third working machine 1C, the operator in the remote operation room 200 operates the operation switch 90. When the operation switch 90 is operated, the transmission destination of the operation signal of the remote operation device 40 and the reception source of the image data of the surrounding image are switched from the second working machine 1B to the third working machine 1C. The control device 60 receives the image data of the surrounding image of the third working machine 1C and causes it to be displayed on the display device 50. The operator in the remote operation room 200 operates the remote operation device 40 to remotely operate the third working machine 1C while viewing the surrounding image of the third working machine 1C displayed on the display device 50. The operation signal of the remote operation device 40 is transmitted to the control device 300 of the third working machine 1C.

[0092] In this way, a single remote operation device 40 can remotely operate a plurality of working machines 1 (1A, 1B, 1C). Also, the surrounding images of a plurality of working machines 1 (1A, 1B, 1C) can be displayed on a single display device 50. The object detection unit 63 is provided in the control device 60 arranged in the remote operation room 200 instead of in the control device 300 of the working machine 1. Therefore, it is not necessary to provide an object detection unit 63 for each of the plurality of working machines 1. When updating or improving the function of the object detection unit 63, it is only necessary to update or improve the control device 60, and it is not necessary to update or improve the control devices 300 of each of the plurality of working machines 1.

[0093] In the example shown in FIG. 10, the first working machine 1A, the second working machine 1B, and the third working machine 1C may operate at the same work site.

[0094] In the example shown in FIG. 10, all of the plurality of working machines 1 (1A, 1B, 1C) are hydraulic excavators. Some of the plurality of working machines 1 may be hydraulic excavators, some may be wheel loaders, some may be bulldozers, or some may be dump trucks. That is, different types of working machines 1 may be remotely operated by one remote operation device 40.

[0095] In the above-described embodiment, the object detection unit 63 is configured to detect the position of the person WM in the peripheral image in the processing unit 63B. The object detection unit 63 does not necessarily have to detect the position of the person WM. The object detection unit 63 may simply detect the person WM from the peripheral image. The object detection unit 63 may use artificial intelligence to detect the person WM in the peripheral image, and the display control unit 65 may superimpose and display a symbol image on the detected person WM.

[0096] In the above-described embodiment, the object detection unit 63 may detect the person WM based on, for example, a pattern matching method without using artificial intelligence. The object detection unit 63 can detect the person WM by comparing a template indicating the person WM with the peripheral image.

[0097] In the above-described embodiment, the imaging device 30 is arranged at the upper part of the front portion of the revolving body 3. The imaging device 30 may be arranged on at least one of the left and right portions of the revolving body 3, or may be arranged at the rear portion of the revolving body 3. The imaging device 30 arranged on the left portion of the revolving body 3 can acquire a peripheral image on the left side of the revolving body 3. The imaging device 30 arranged on the right portion of the revolving body 3 can acquire a peripheral image on the right side of the revolving body 3. The imaging device 30 arranged at the rear portion of the revolving body 3 can acquire a peripheral image behind the revolving body 3.

[0098] In the above-described embodiment, the object detection unit 63 was configured to detect the distance from the work machine 1 to the person WM. The object detection unit 63 may estimate the distance to the person WM using artificial intelligence, or may geometrically calculate the distance to the person WM based on the installation conditions of the imaging device 30 and the position of the person WM in the surrounding image. The installation conditions of the imaging device 30 include the installation position of the imaging device 30 and the direction of the optical axis of the optical system of the imaging device 30.

[0099] In the above-described embodiment, the object detected by the object detection unit 63 was assumed to be the person WM. The object detected by the object detection unit 63 may not be the person WM. The object detected by the object detection unit 63 may be a work machine that is an obstacle different from the work machine 1 such as a transport vehicle like a dump truck or a service car.

[0100] In the above-described embodiment, one control device 60 was assumed to have the functions of the operation signal transmission unit 61, the image data reception unit 62, the object detection unit 63, the learning unit 64, the display control unit 65, and the alarm output unit 66. At least a part of the functions of the operation signal transmission unit 61, the image data reception unit 62, the object detection unit 63, the learning unit 64, the display control unit 65, and the alarm output unit 66 may be provided in a separate control device provided in the remote operation room 200 and may be realized by two or more control devices.

[0101] In the above-described embodiment, the alarm output unit 66 and the alarm device 80 may be omitted.

[0102] Note that in the above-described embodiment, the work machine 1 was assumed to be a hydraulic excavator. The work machine 1 may be a bulldozer, a wheel loader, or a dump truck.

Explanation of Reference Numerals

[0103] 1... Construction machine, 1A... First construction machine, 1B... Second construction machine, 1C... Third construction machine, 2... Traveling body, 3... Slewing body, 4... Working machine, 4A... Boom, 4B... Arm, 4C... Bucket, 5... Hydraulic cylinder, 5A... Boom cylinder, 5B... Arm cylinder, 5C... Bucket cylinder, 6... Communication device, 7... Communication device, 8... Symbol image, 8A... Symbol image, 8B... Symbol image, 8C... Symbol image, 30... Imaging device, 31... Front camera, 32... Left camera, 33... Right camera, 34... Upper camera, 35... Lower camera, 40... Remote operation device, 41... Left working lever, 42... Right working lever, 43... Left travel lever, 44... Right travel lever, 45... Operator's seat, 50... Display device, 51... Central display, 52... Left display, 53... Right display, 54... Upper display, 55... Lower display, 60... Control device, 61... Operation signal transmission unit, 62... Image data reception unit, 63... Object detection unit, 63A... Feature amount extraction unit, 63B... Processing unit, 63C... Output unit, 64... Learning unit, 65... Display control unit, 66... Alarm output unit, 67... Switching unit, 70... Dump truck, 80... Alarm device, 90... Operation switch, 100... Remote operation system, 200... Remote operation room, 300... Control device, 301... Travel control unit, 302... Slewing control unit, 303... Working machine control unit, 304... Image data transmission unit, 400... Communication system, AX... Boom rotation axis, BX... Arm rotation axis, CX... Bucket rotation axis, RX... Slewing axis, WM... Person, WM1... Person, WM2... Person, WM3... Person.

Claims

1. A computer system disposed at a remote operation location of a work machine having a work implement, wherein the work machine includes a first work machine and a second work machine operating at the same work site, the computer system comprising: an operation signal transmission unit that transmits an operation signal for remotely operating the operation of the work machine; an image data reception unit that receives, from the work machine, image data indicating a peripheral image of the work machine including at least an image of a construction target of the work machine captured by an imaging device provided on the work machine; an object detection unit that detects a person existing around the work machine based on the image data and collation data for detecting a person, and is capable of updating a function of detecting the person; a display control unit that causes a symbol image indicating a detection result of the object detection unit to be displayed on a display device disposed at the remote operation location together with the peripheral image, and changes a display form of the symbol image based on a distance from the work machine to the person; a switching unit that switches the work machine that is a transmission destination of the operation signal and the work machine that is a reception source of the image data; and the display control unit displays the peripheral image of the work machine after switching. A remote operation system for a work machine.

2. A computer system disposed at a remote operation location of a work machine having a work implement, transmits an operation signal for remotely operating the operation of the work machine, receives, from the work machine, image data indicating a peripheral image of the work machine including at least an image of a construction target of the work machine captured by an imaging device provided on the work machine, detects a person existing around the work machine based on the image data and collation data for detecting a person, and updates a function of detecting the person, causes a symbol image indicating a detection result of the person existing around the work machine to be displayed on a display device disposed at the remote operation location together with the peripheral image, and changes a display form of the symbol image based on a distance from the work machine to the person, switches the work machine that is a transmission destination of the operation signal and the work machine that is a reception source of the image data, and displays the peripheral image of the work machine after switching, wherein the work machine includes a first work machine and a second work machine operating at the same work site. A remote operation method for a work machine.

Images