Remote control system, remote operation type work machine system, and work information display control method

Abstract

Provided is a remote control system for remotely controlling a work machine, in order to prevent a decrease in work efficiency due to a video disorder associated with a video transmission delay and packet loss during communication in a series of operations of remote operation of the work machine, and to improve the work efficiency of an operator. Comprising a communication device for communicating with a work machine, and a control device for controlling the operation of the work machine, processing an image of the surroundings from the work machine, and displaying the image on a display device. The communication device receives an image of the surroundings captured by the work machine and assist information including attitude information of the work machine and three-dimensional information of the surroundings of the work machine. The control device executes: a process for generating, from the assist information, an assist pattern indicating the attitude of the work machine and the position of the work object; and superimposing and displaying, on a display device (see Figure 10), a surrounding image acquired at a first timing in the work machine and an assist pattern generated on the basis of assist information acquired at a second timing after the first timing in the work machine.

Description

Technical Field

[0001] The present disclosure relates to a remote control system, a remote-operated working machine system, and a working information display control method. Background Art

[0002] A work machine control system capable of remotely operating a work machine has been put into practical use, but due to the transmission delay caused by transmitting images from the work machine to the remote control system (remote operation room), there is a case where the work efficiency is reduced. In order to cope with this situation, for example, Patent Document 1 proposes processing the image and displaying it on a display to correct the delay time required for the remote operation unit to transmit the image. By displaying such processed images, the efficiency of remote operation can be improved when the image reception delay time in the remote operation room is long.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent No. 6824856 Summary of the invention

[0006] Problems to be solved by the invention

[0007] However, according to Patent Document 1, although the transmission delay of the image around the working machine (the vehicle) is improved, since the vehicle is not taken into account, when a part of the vehicle appears in the image, the display position of the part of the vehicle in the processed image is offset in relation to the surrounding image, which may hinder the operator's efficient operation. In addition, when image disturbances are caused by packet loss during communication, efficient operation may also be hindered.

[0008] In view of the above situation, the present disclosure provides a technology for preventing a decrease in work efficiency caused by image distortion due to image transmission delay and packet loss during communication in a series of actions of remote operation of a working machine and improving the work efficiency of an operator.

[0009] Means for solving problems

[0010] In order to solve the above-mentioned problems, the present disclosure proposes a remote control system that remotely controls a working machine, including a communication device that communicates with the working machine and a control device that controls the movement of the working machine and processes the surrounding images from the working machine and displays them on a display device. The communication device receives the surrounding images taken by the working machine and auxiliary information including posture information of the working machine and three-dimensional information around the working machine, and the control device performs the following processing: generating an auxiliary graphic representing the posture of the working machine and the position of the working object from the auxiliary information; and overlapping the surrounding images obtained at a first timing in the working machine with the auxiliary graphic generated based on the auxiliary information obtained at a second timing in the working machine after the first timing and displaying them on the display device.

[0011] Other features related to the present disclosure can be learned from the description and drawings of this specification. In addition, the present disclosure is implemented through the combination of elements and multiple elements and the detailed description below and the scheme of the claims.

[0012] The descriptions in this specification are only typical examples, and it should be noted that they do not limit the technical solutions or application examples of the present disclosure in any way.

[0013] Effects of the Invention

[0014] According to the technology disclosed in the present invention, it is possible to prevent a decrease in working efficiency due to a delay in image transmission during a series of operations of remote control of a working machine. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 1 is a diagram showing an example of the external configuration of the working machine control system 1 .

[0016] Figure 2 A schematic diagram for explaining the configuration of the surrounding image acquisition unit S1 and the three-dimensional information acquisition unit S2 mounted in the working machine 100.

[0017] Figure 3 1 is a diagram showing an example of the internal configuration of the remote control system (operation room) 300 .

[0018] Figure 4 1 is a diagram showing an example of the internal configuration of the sensor information processing unit 103 mounted on the working machine 100 .

[0019] Figure 5 1032 is a diagram showing an example of the internal configuration of the auxiliary information generating unit 1032 .

[0020] Figure 6 1 is a diagram showing a configuration example of a remote control system (remote operation room) 300 .

[0021] Figure 7This is a flowchart for explaining the image compression processing performed by the sensor information processing unit 103 and the auxiliary information generation processing performed by the auxiliary information generation unit 1032 .

[0022] Figure 8 It is used to illustrate Figure 6 A flowchart of a process of generating a superimposed image having an AR image by the driving control device 305.

[0023] Fig. 9 FIG. 1 is a diagram showing an example of a decompressed image (working machine front image + surrounding image).

[0024] Fig.10 is the image obtained after decompression ( Fig. 9 ) is a diagram showing an example of an overlapping image obtained by overlapping AR images.

[0025] Fig.11 The diagram shows an example of a superimposed image (working machine front image+surrounding image+AR image) displayed when the bucket 113 is lifted vertically.

[0026] Fig.12 1 is a diagram showing an example of the correspondence relationship between the auxiliary lines indicating the orientation of the crawler track 101 in the AR image displayed in the superimposed image and the vehicle body.

[0027] Fig.13 A diagram showing an example of an AR image displayed in a superimposed image.

[0028] Fig.14 It is a diagram showing another example of an AR image displayed in a superimposed image.

[0029] Fig.15 This is a diagram showing an example of display F17 in AR prior rendering in which an AR graphic is superimposed on a video in advance.

[0030] Fig.16 It is a diagram showing the communication delay generated when an overlapping image of an AR image is generated in the working machine 100 and sent to the remote control system 300 without adopting the structure of the first embodiment, or when an image acquired by the surrounding image acquisition unit S1 and the auxiliary information generated by the auxiliary information generation unit 1032 are sent to the remote control system 300 in the same communication packet without performing network slicing and QoS control.

[0031] Fig.17 This is a diagram showing communication delays added to the image acquired by the surrounding image acquisition unit S1 and the auxiliary information generated by the auxiliary information generation unit 1032 when the configuration of the first embodiment is adopted.

[0032] Fig.181 is a diagram showing an example of a superimposed image in which an AR image generated based on auxiliary information 2 transmitted from the communication device 104 to the communication device 306 after the image 1 is superimposed on the image 1 .

[0033] Fig.19 The diagram shows an example of the posture of the working machine 100 when acquiring the image 1 and an example of the posture of the working machine 100 when acquiring the auxiliary information 2 when the working machine 100 operates so that the main body 102 rotates leftward relative to the crawler 101 .

[0034] Fig. 20 It is a diagram showing an example of the internal configuration of the driving control device 305 according to the second embodiment (basic form).

[0035] Fig.21 This is a diagram for explaining the driving control device 305 (see Fig. 20 ) is a flowchart of the process of generating overlapping images with AR images.

[0036] Fig. 22 This is a diagram showing an internal configuration example 1 of the sensor information processing unit 103 in the second embodiment (variation).

[0037] Fig.23 This is a diagram showing an internal configuration example 2 of the sensor information processing unit 103 in the second embodiment (variation).

[0038] Fig.24 1 is a diagram showing an example of a method of embedding information into some pixels of an image acquired by the surrounding image acquisition unit S1 in the information embedding unit 1034 .

[0039] Fig.25 is shown with Fig. 22 and Fig.23 A diagram showing an example of the internal structure of the driving control device 305 corresponding to the sensor information processing unit 103.

[0040] Fig.26 The sensor information processing unit 103 (see Fig.21 or Fig. 22 ) is a flowchart of image compression processing and auxiliary information generation processing.

[0041] Fig. 27 This is a diagram for explaining the driving control device 305 (see Fig.25 ) is a flowchart of the process of generating overlapping images with AR images.

[0042] Fig.28The diagram shows an example of images (surrounding camera images and working machine front image) displayed by the processing of the second embodiment (basic form and modified example) and an AR image displayed superimposed thereon.

[0043] Fig.29 It is a diagram showing an internal configuration example 1 of the driving control device 305 according to the third embodiment.

[0044] Fig.30 It is a diagram showing a second internal configuration example of the driving control device 305 according to the third embodiment.

[0045] Fig.31 The diagram shows an example of images (surrounding camera images and working machine front image) displayed by the processing of the third embodiment and an AR image displayed superimposed thereon.

[0046] Fig.32 It is a diagram showing an example of the internal configuration of the driving control device 305 according to the fourth embodiment.

[0047] Fig.33 The diagram shows an example of images (surrounding camera images and working machine front image) displayed by the processing of the fourth embodiment and an AR image displayed superimposed thereon.

[0048] Fig.34 This is a diagram showing an example of a method of specifying a work site surface position stored in a work site surface storage unit using a GUI in the fourth embodiment.

[0049] Fig.35 It is a conceptual diagram for explaining switching of the display state of the AR image in the working machine control system 1 according to each embodiment. DETAILED DESCRIPTION

[0050] Hereinafter, the present embodiment is described with reference to the accompanying drawings. In the accompanying drawings, there are also cases where elements with the same function are shown with the same number. It should be noted that the accompanying drawings show implementation methods and embodiments that follow the principles of the present disclosure, but they are used to understand the present disclosure and are by no means used to make a restrictive interpretation of the present disclosure. The description in this specification is only a typical example and does not limit the technical solution or application example of the present disclosure in any way.

[0051] In this embodiment, the present disclosure is described in sufficient detail for those skilled in the art to implement the present disclosure, but it should be understood that other installations / modes are also possible, and changes in the composition / structure and replacement of various elements can be made without departing from the technical scope and spirit of the present disclosure. Therefore, the following description should not be interpreted as being limited to this.

[0052] (1) First Embodiment

[0053] <Example of the configuration of a control system for working machines>

[0054] Reference Figures 1 to 3 A schematic configuration example of a working machine control system (remote-operated working machine system) 1 according to the first embodiment will be described. Figure 1 1 is a diagram showing an example of the external configuration of the working machine control system 1 . Figure 2 It is a schematic diagram for explaining the configuration of the surrounding image acquisition unit S1 and the three-dimensional information acquisition unit S2 mounted in the working machine 100 . Figure 3 1 is a diagram showing an example of the internal configuration of the remote control system (operation room) 300 .

[0055] like Figure 1 As shown, the working machine control system 1 includes a remote control system 300 and a working machine (construction machine) 100 controlled by the remote control system 300 .

[0056] As an example, the working machine 100 is a backhoe with a bucket 113 facing the operator's cabin OR. The ground L as a construction target is excavated by the operation of the working machine 100, forming, for example, a plane L0, a plane L1, an inclined plane L2, etc. The soil excavated by the working machine 100 is moved to the bucket 200 of a dump truck or the like by the rotation of the working machine 100, and is discharged (discharged) toward the bucket 200.

[0057] As an example, a working machine 100 as a backhoe includes a crawler 101, a main body 102 supported and transported by the crawler 101, an operation room OR mounted on the main body 102, a sensor information processing unit 103, a communication device 104, various sensors S1 to S4, an antenna 107, a boom 111, an arm 112, and a bucket 113. The boom 111, the arm 112, and the bucket 113 are driven by an engine, a hydraulic cylinder, etc. (not shown) to dig the ground L as a construction object.

[0058] In addition, the crawler 101 is driven by a driving unit (not shown), so that the main body 102 can move forward, backward, left and right. In addition, the main body 102 is configured to be able to rotate relative to the crawler 101, and is driven by a driving unit (not shown) to control the rotation direction and rotation angle. The working machine 100 is configured to be unmanned / remotely controlled by a remote control system 300, without the need for an operator to ride in the operator's cab OR.

[0059] The main body 102 includes (is equipped with) a sensor information processing unit 103 and a communication device 104. The sensor information processing unit 103 compresses the image of the ground and the bucket 113 as the construction object acquired by the surrounding image acquisition unit S1 including the camera element (the image is obtained by shooting the working area including the working machine 100, that is, the working machine 100 and the surrounding area, hereinafter sometimes referred to as the surrounding image.). In addition, the sensor information processing unit 103 generates auxiliary information such as position information (distance information from a reference point) including the current bucket 113 and the working ground surface located directly below the bucket 113 and the bucket 200 of the dump truck, etc., based on the sensor information obtained from the three-dimensional information acquisition unit S2 and the posture information acquisition unit S3. The auxiliary information can be composed of the sensor information itself obtained from the three-dimensional information acquisition unit S2 and the posture information acquisition unit S3, or it can be composed of information that transforms the sensor information into the main body coordinate system (vehicle coordinate system) of the working machine 100. Among them, the latter is preferred considering the amount of calculation in the later stage processing.

[0060] The communication device 104 is configured to transmit the acquired image data and auxiliary information to the remote control system 300 and to receive control information from the remote control system 300 .

[0061] The main body 102 is provided with a surrounding image acquisition unit S1 for acquiring images of the front part (boom 111, arm 112, bucket 113) and surrounding images of the working machine 100, a three-dimensional information acquisition unit S2 for detecting the presence of an object in the field of view and the distance, and a posture information acquisition unit S3 for detecting the posture of the working machine 100 at a predetermined location. Figure 2 As shown, as an example, the surrounding image acquisition unit S1 is an imaging device that captures the surroundings of the working machine 100 , and is a camera having a CMOS sensor that captures a field of view SR1 in front of the working machine 100 .

[0062] The three-dimensional information acquisition unit S2 is an object detection sensor such as LiDAR (Light Detection and Ranging) and an ultrasonic sensor that detects the presence or absence of an object and the distance within the field of view SR2. The three-dimensional information acquisition unit S2 measures the distance from the sensor S2 as sensor information, and outputs three-dimensional information obtained by converting the measured distance into information of the main body coordinate system. The three-dimensional information acquisition unit S2 is not limited to this, and can also be composed of multiple LiDAR and ultrasonic sensors such as a sensor S21 that detects the field of view SR21 and a sensor S22 that detects the field of view SR22. The field of view is not limited to these three directions, and the entire surrounding area can be detected by adjusting the setting position / angle / number of units according to the field of view of the sensor. In addition, the surrounding image acquisition unit S1 and the three-dimensional information acquisition unit S2 can also be composed of a single or multiple stereo cameras that can perform video and stereoscopic observation.

[0063] The posture information acquisition unit S3 is a posture sensor that detects the posture of the working machine 100, and can acquire the postures of the boom 111, the arm 112, and the bucket 113 (for example, the angle between the boom 111 and the main body 102, and the angle between the arm 112 and the bucket 113 relative to the connecting portion) and the rotation angle of the main body 102 relative to the track 101 (the track 101 and the main body 102 can be set to 0 degrees when they are facing the same direction). The posture information acquisition unit S3 can be composed of, for example, any one of a gyro sensor, a tilt sensor, and an acceleration sensor, or a combination thereof. In addition, Figure 2 The example shown is just an example, and the present disclosure is not limited thereto.

[0064] Reference Figure 3 An example of a detailed configuration of the remote control system 300 will be described. As an example, the remote control system 300 includes a driver's seat 301 , an operation control unit 302 , an operation lever 303 , a display 304 , a driving control device 305 , and a communication device 306 .

[0065] The driver's seat 301 is a chair for the operator to sit on, and an operation control unit 302 and an operation lever 303 are provided on the side thereof. In addition to the operation lever 303, various joysticks, switches, pedals, etc. may be configured, and their operations are reflected in the movement of the working machine 100. By operating the operation lever 303 and the like, the operation control unit 302 generates a corresponding operation signal and sends it to the driving control device 305. The driving control device 305 generates a control signal for driving the working machine 100 based on the received operation signal, and sends it to the working machine 100 via the communication device 306.

[0066] The display 304 is disposed in front of the driver's seat 301 and is a display unit that displays the bucket 113, the ground of the construction target, and an AR (Augmented Reality) image to be described later. Figure 3 As shown, it can be composed of a plurality of display devices that give appropriate depression angles to realistically express a three-dimensional space. The display 304 is not limited thereto, and can be composed of only one display device or can be a head-mounted display.

[0067] <Internal Configuration Example of Sensor Information Processing Unit 103>

[0068] Figure 4 1 is a diagram showing an example of the internal configuration of the sensor information processing unit 103 mounted on the working machine 100 .

[0069] As an example, the sensor information processing unit 103 includes an image compression unit 1031, an auxiliary information generation unit 1032, and an auxiliary information encoding unit 1033. The sensor information processing unit 103 may be a general-purpose computer, for example, it may be composed of a CPU 151 (processor), a ROM 152, a RAM 153, a hard disk drive 154, an input device 155, and a display 156 (display unit). The CPU 151 is a form of a processor, and may also be a GPU, a semiconductor device capable of performing other calculations, or a combination thereof.

[0070] The image compression unit 1031 compresses the surrounding image acquired from the surrounding image acquisition unit S1 using a predetermined image compression method and outputs the compressed image to the communication device 104. The image compression method may use a known method such as MPEG, H.264, MotionJPEG, etc., but is not limited thereto.

[0071] The auxiliary information generation unit 1032 calculates the position and posture of the bucket 113, the height of the ground (work surface) vertically below the bucket 113, and the position of the bucket 200 of a dump truck, etc. based on the three-dimensional information obtained from the three-dimensional information acquisition unit S2 and the posture information obtained from the posture information acquisition unit S3.

[0072] The auxiliary information encoding unit 1033 encodes the auxiliary information (the position and posture of the bucket 113, the height of the work surface, the position of the bucket 200 of a dump truck, etc., and the rotation angle of the main body 102 relative to the crawler 101) output from the auxiliary information generating unit 1032 through a predetermined communication protocol and outputs it to the communication device 104. The encoding can use a known method such as the CAN (Controller Area Network) communication protocol specified by ISO11898, but is not limited to this.

[0073] <Internal Configuration Example of the Supplementary Information Generating Unit 1032>

[0074] Figure 5 1 is a diagram showing an example of the internal structure of the auxiliary information generation unit 1032. The auxiliary information generation unit 1032 processes the three-dimensional information acquired from the three-dimensional information acquisition unit S2 and the posture information acquired from the posture information acquisition unit S3, and generates auxiliary information including lightweight (low-capacity) three-dimensional information. The auxiliary information generation unit 1032 includes, as an example, a coordinate system correction information 1 storage unit 10324, a dumping position calculation unit 10321, a bucket position / posture calculation unit 10323, and a work surface height calculation unit 10322.

[0075] The coordinate system correction information 1 storage unit 10324 stores correction information for integrating the coordinate systems of the detection signals of the three-dimensional information acquired from the three-dimensional information acquisition unit S2 and the posture information acquired from the posture information acquisition unit S3. Specifically, the correction information is information of six degrees of freedom (six-dimensional vector: parallel translation and rotation in the xyz direction) that permanently transforms the information coordinates of the sensor reference (sensor coordinate system) into the information of the vehicle body reference (body coordinate system).

[0076] Based on the calibration information, the dump position calculation unit 10321, the bucket position / posture calculation unit 10323, and the work surface height calculation unit 10322 output auxiliary information as data of a coordinate system (main body coordinate system) that rotates in accordance with the rotation of the main body 102 relative to the crawler 101, for example, on the rotation axis of the work machine 100 and on the bottom surface of the main body 102 as a reference point. The calibration information can be calibrated at the time of shipment or inspection of the work machine 100, or can be automatically calibrated when the work machine 100 is started.

[0077] The dump position calculation unit 10321 calculates the position (position in the main body coordinate system) of the dump truck bucket 200 based on the three-dimensional information acquired from the three-dimensional information acquisition unit S2 and the correction information acquired from the coordinate system correction information 1 storage unit 10324. As an example, the position of the dump truck bucket 200 is calculated using the three-dimensional coordinate values ​​representing the positions of the four vertices of the bucket top edge.

[0078] The bucket position / posture calculation unit 10323 calculates the position and posture of the bucket 113 (the position and posture in the main body coordinate system) based on the posture information acquired from the posture information acquisition unit S3 and the correction information acquired from the coordinate system correction information 1 storage unit 10324. The position of the bucket 113 is, for example, a three-dimensional coordinate value of the joint of the bucket 113 and the arm 112. The posture of the bucket 113 is calculated using, for example, a three-dimensional unit direction vector connecting the joint of the bucket 113 and the arm 112 to the center of the front end of the bucket 113.

[0079] The work surface height calculation unit 10322 calculates the height of the work surface (height in the main body coordinate system) based on the three-dimensional information acquired from the three-dimensional information acquisition unit S2, the position of the bucket 113 output from the bucket position / posture calculation unit 10323, and the correction information acquired from the coordinate system correction information 1 storage unit 10324. The calculation of the work surface height can be obtained by calculating, for example, the average height of the three-dimensional shape of the ground located vertically below the position of the bucket 113 based on the three-dimensional information.

[0080] <Configuration Example of Remote Control System 300>

[0081] Figure 630 is a diagram showing a configuration example of a remote control system (remote operation room) 300 . The remote control system 300 includes a communication device 306 , a driving control device 305 , and a display 304 .

[0082] The aforementioned image data subjected to image compression and the aforementioned encoded auxiliary information are transmitted by wireless communication via a wireless base station not shown in the figure through the communication device 104 in the working machine 100 , received by the communication device 306 of the remote control system 300 , and handed over to the driving control device 305 .

[0083] Driving control device 305 such as Figure 6 As shown, as a structure for generating and displaying a superimposed image with an AR image, for example, it includes an image decompression unit 3051, an image superimposition unit 3052, an auxiliary information decoding unit 3053, a coordinate system integrated processing unit 3054, an auxiliary graphic generation unit 3055, and a coordinate system correction information 2 storage unit 3056. The driving control device 305 also has a function of sending a control signal from the operation control unit 302 from the communication device 306 and thereby controlling various actions of the working machine 100, which is omitted in detail. The driving control device 305 can be implemented by a CPU or GPU (processor), a general-purpose computer with various memories, etc., and a computer program for performing the following various actions. That is, the image decompression unit 3051, the image superimposition unit 3052, the auxiliary information decoding unit 3053, the coordinate system integrated processing unit 3054, and the auxiliary graphic generation unit 3055 can be implemented by expanding the corresponding computer program on the internal memory of the processor.

[0084] The image decompression unit 3051 decompresses (decodes) the image data received from the communication device 306 and subjected to the above-mentioned image compression (encoding).

[0085] The auxiliary information decoding unit 3053 decodes the aforementioned encoded auxiliary information received by the communication device 306 .

[0086] The coordinate system correction information 2 storage unit 3056 maintains correction information for integrating the coordinate systems of the auxiliary information decoded by the auxiliary information decoding unit 3053 and the image data decompressed by the image decompression unit 3051. The correction information is information for converting data described in the main body coordinate system into data in the camera coordinate system. In addition, the correction information can be constituted as a 6-dimensional vector representing the setting position and posture of the surrounding image acquisition unit S1 in the main body coordinate system (a coordinate system that uses the bottom surface of the main body 102 on the rotation axis of the working machine 100 as a reference point and rotates in conjunction with the rotation of the main body 102 relative to the crawler 101). It should be noted that the correction information can be generated by correction when the working machine 100 leaves the factory or is inspected, or it can be generated by performing automatic correction processing when the working machine 100 is started.

[0087] The coordinate system integrated processing unit 3054 transforms the auxiliary information (the position and posture of the bucket 113, the height of the work surface, the position of the bucket 200 of the dump truck, etc., and the rotation angle of the main body 102 relative to the track 101) into relative coordinate / posture information based on the setting coordinates of the surrounding image acquisition unit S1 (camera coordinate system) based on the auxiliary information output by the auxiliary information decoding unit 3053 and the correction information output from the coordinate system correction information 2 storage unit 3056.

[0088] The auxiliary graphics generation unit 3055 generates AR image structures (structures in which a plurality of auxiliary graphics are visualized) for representing the position and posture of the bucket 113, the height of the work surface, the position of the bucket 200 of a dump truck, etc., and the posture of the crawler 101, based on the auxiliary information transformed by the coordinate system integration processing unit 3054. The AR image is an image displayed by overlapping with the actual image of the work machine 100, the construction object (ground, etc.), etc., in order to assist the work of the work machine 100.

[0089] The image superimposing unit 3052 superimposes the image decompressed by the image decompressing unit 3051 on the AR image generated by the auxiliary graphic generating unit 3055 to generate a superimposed image. The superimposed image is output to the display 304 .

[0090] <Details of image compression and auxiliary information generation processing>

[0091] Figure 7 This is a flowchart for explaining the image compression processing performed by the sensor information processing unit 103 and the auxiliary information generation processing performed by the auxiliary information generation unit 1032. In the following description, the action subject of each step is set as each processing unit (image compression unit 1031, bucket position / posture calculation unit 10323, etc.), but since it is implemented by a processor, the action subject can also be set as the processor.

[0092] (i) Step S11

[0093] The image compression unit 1031 compresses the image acquired by the surrounding image acquisition unit S1 in a predetermined format (eg, MPEG, H.264, etc.) to generate compressed image data, and outputs the compressed image data to the communication device 104. The communication device 104 transmits the compressed image data to the remote control system 300.

[0094] (ii) Step S12

[0095] The bucket position / posture calculation unit 10323 calculates the position and posture of the bucket 113 based on the posture information acquired from the posture information acquisition unit S3 and the correction information acquired from the coordinate system correction information storage unit 10324. The posture information is information based on the posture information acquisition unit S3 (information of the sensor coordinate system), so the position and posture of the bucket 113 are calculated by converting the coordinate system correction information into information of the main body (vehicle body) coordinate system.

[0096] (iii) Step S13

[0097] The dump position calculation unit 10321 calculates the position of the bucket 200 of the dump truck or the like based on the three-dimensional information acquired from the three-dimensional information acquisition unit S2 and the correction information acquired from the coordinate system correction information storage unit 10324. Similar to the posture information, the three-dimensional information is information based on the three-dimensional information acquisition unit S2 (information of the sensor coordinate system), so the position of the bucket 200 is calculated by converting the coordinate system correction information into information of the main body (vehicle body) coordinate system.

[0098] (iv) Step S14

[0099] The work surface height calculation unit 10322 calculates the work surface height based on the three-dimensional information acquired from the three-dimensional information acquisition unit S2, the bucket position calculated in step S12, and the correction information acquired from the coordinate system correction information 1 storage unit 10324. In other words, the work surface height can be calculated from the three-dimensional information, for example, by calculating the average height of the three-dimensional shape of the ground vertically below the tip of the claw of the bucket 113, and converting it into the main body coordinate system by using the correction information.

[0100] (v) Step S15

[0101] The auxiliary information encoding unit 1033 generates coded auxiliary information by encoding the position and posture of the bucket 113 calculated in step S12, the position of the bucket 200 of the dump truck or the like calculated in step S13, the surface height of the work surface calculated in step S14, and the rotation angle of the main body 102 relative to the crawler 101 acquired from the posture information acquisition unit S3, and outputs the coded auxiliary information to the communication device 104. The communication device 104 transmits the coded auxiliary information to the remote control system (remote operation room) 300.

[0102] It should be noted that the execution order of steps S11 to S13 may be alternated, and other steps may be executed twice or more during the execution of any step among steps S11 to S12 once.

[0103] <Details of the superimposed image generation process>

[0104] Figure 8It is used to illustrate Figure 6 The flowchart of the process of generating an overlapping image with an AR image by the driving control device 305. In the following description, the action subject of each step is set as each processing unit (image decompression unit 3051, auxiliary information decoding unit 3053, etc.), but since it is implemented by a processor, the action subject can also be set as the processor.

[0105] (i) Step S16

[0106] The image decompression unit 3051 decompresses (decodes) compressed image data transmitted from the communication device 104 of the working machine 100 and received by the communication device 306 of the remote control system 300. The decompressed image data is data obtained by projecting the external scenery observed from the camera coordinate system onto the image plane.

[0107] (ii) Step S17

[0108] The auxiliary information decoding unit 3053 decodes the coded auxiliary information transmitted from the communication device 104 of the working machine 100 and received by the communication device 306 of the remote control system 300. The decoded auxiliary information becomes information of the main body coordinate system (vehicle body coordinate system).

[0109] (iii) Step S18

[0110] The coordinate system integrated processing unit 3054 transforms the auxiliary information into relative coordinate / posture information based on the installation position of the surrounding image acquisition unit S1 based on the auxiliary information decoded in step S17 and the correction information output from the coordinate system correction information 2 storage unit 3056. In other words, the auxiliary information is transformed from the information of the main body coordinate system to the information of the camera coordinate system.

[0111] (iv) Step S19

[0112] The auxiliary graphic generation unit 3055 generates an AR image (auxiliary graphic) showing the position and posture of the bucket 113 , the height of the work surface, the position of the bucket 200 of a dump truck, etc., and the posture of the crawler 101 based on the auxiliary information after coordinate conversion in step S18 .

[0113] AR images can be generated by performing calculations to project line segments and graphics in three-dimensional space from camera coordinates to an image plane. For example, in the case of the bucket 200, since the position of the upper end opening is known, the rectangle representing the upper end opening of the bucket 200 is detected and graphed to generate an AR image. In addition, template image data of the part to be displayed (bucket, shovel, etc.) can also be prepared in advance, and an AR image can be generated by appropriately deforming the template image according to the position of each part. The transformation from camera coordinates to the image plane is performed by using, for example, a perspective projection transformation of the internal parameters of the surrounding image acquisition unit S1 (the focal length of the lens, the effective size of each pixel of the camera element, and the center coordinates of the image) pre-stored in a storage device not shown in the figure.

[0114] (v) Step S20

[0115] The image superimposing unit 3052 superimposes the AR image generated in step S19 on the image acquired in step S16 to generate a superimposed image, and outputs the superimposed image to the display 304 .

[0116] <Example of overlapping images>

[0117] Figures 9 to 14 This is a diagram for explaining an example of superimposed images (working machine front image+surrounding image+AR image) in the first embodiment. Fig. 9 FIG. 1 is a diagram showing an example of a decompressed image (working machine front image + surrounding image). Fig.10 is a diagram showing the decompressed image ( Fig. 9 ) is a diagram showing an example of an overlapping image obtained by overlapping an AR image. Fig.11 The diagram shows an example of a superimposed image (working machine front image+surrounding image+AR image) displayed when the bucket 113 is lifted vertically. Fig.12 1 is a diagram showing an example of the correspondence relationship between the auxiliary lines indicating the orientation of the crawler track 101 in the AR image displayed in the superimposed image and the vehicle body. Fig.13 A diagram showing an example of an AR image displayed in a superimposed image. Fig.14 It is a diagram showing another example of an AR image displayed in a superimposed image.

[0118] For example Fig. 9 As shown, the image decompressed by the image decompression unit 3051 is composed of the boom 111, the arm 112, the bucket 113 (the front part of the working machine), the truck bed 200 of the dump truck, etc., and the ground (surrounding image) as the excavation object. Fig. 9 The image shown includes an AR image obtained by overlapping the image overlapping portion 3052, for example, Fig.10 Displayed in the manner shown.

[0119] like Fig.10 As shown, the AR image includes a graphic F1 indicating the position and posture of the bucket 113 during the excavation operation of the working machine 100, a graphic F3 indicating the position corresponding to the moving direction of the bucket 113, in other words, the position of the ground (working surface) to which the bucket 113 is to move, a graphic F2 connecting the two graphics, a graphic F4 indicating the position of the right side portion of the crawler 101, a graphic F5 indicating the position of the left side portion of the crawler 101, a graphic F6 indicating the position of the front end portion of the crawler 101, a graphic F7 indicating the position of the rear end portion of the crawler 101, and a graphic F8 indicating the positions of the four vertices of the top edge of the bucket 200 of a dump truck or the like.

[0120] The graphic F1 is displayed in a manner that reflects the posture and shape of the bucket at the position of the bucket 113 calculated by the bucket position / posture calculation unit 10323. Thus, even when there is an influence of image disturbance due to image transmission delay and packet loss during communication, the state of the bucket 113 can be clearly visually recognized based on the AR image drawn based on auxiliary information that has a lower capacity (lighter weight) than the image and is less susceptible to packet loss due to transmission delay.

[0121] For example, in Fig.11 , an example of an overlapping image displayed on the display 304 when the bucket 113 is lifted vertically by operating the joystick 303 of the remote control system 300 is shown. The image data acquired by the surrounding image acquisition unit S1 is displayed on the display 304 in a delayed state relative to the state of the working machine 100 due to the influence of various delays generated in the compression process of the image compression unit 1031, the decompression process of the image decompression unit 3051, and the transmission from the communication device 104 to the communication device 306. On the other hand, the data capacity of the auxiliary information generated by the auxiliary information generation unit 1032 is small relative to the image data, and the delay in the encoding / decoding process is small. In addition, when the auxiliary information is transmitted in a data packet different from the image data through network slicing and QoS (Quality of Service) control, the delay of the auxiliary information due to the transmission from the communication device 104 to the communication device 306 is less than the delay of the image data. Therefore, the graphic F1 drawn based on the auxiliary information can show the position and posture of the bucket 113 of the working machine 100 with low delay compared with the image data. As a result, as shown in Fig.11 As shown, when the bucket 113 is lifted vertically, the graphic F1 can indicate the movement of the bucket 113 in advance compared with the image data of the bucket 113. In addition, even when a part or the whole of the image is disturbed due to packet loss, the movement state of the bucket 113 can be visually recognized by overlapping the AR image with the driving control device 305 of the remote control system 300.

[0122] The graph F3 is displayed in a manner reflecting the position of the work surface. The position of the work surface is obtained by calculating the average height of the three-dimensional shape of the ground located vertically below the front end position of the bucket 113 by the work surface height calculation unit 10322 based on the above-mentioned three-dimensional information. By displaying the graph F3, even in the presence of image disturbances associated with image transmission delays and packet loss during communication, the positional relationship between the bucket 113 and the work surface can be clearly identified based on the AR image depicted based on auxiliary information that has a low capacity (lightweight) compared to image data and is not easily affected by packet loss due to transmission delays. In addition, by displaying the graph F3 on both sides of the work surface in a manner that sandwiches the work surface (the excavation site of the bucket 113), the operator of the remote control system 300 can clearly observe the work surface through the display 304 while clearly identifying the positional relationship between the bucket 113 and the work surface.

[0123] In addition, the graph F2 is displayed as a graph connecting the graphs F1 and F3. When the bucket 113 digs the horizontal ground, the front end of the bucket 113 and F3 of the graph F1 extend in a substantially horizontal direction, while the graph F2 intersects the graphs F1 and F3 and extends in a substantially vertical direction. By displaying the graph F2, the operator can easily recognize the distance between the bucket 113 and the ground.

[0124] Graphics F4 to F7 display auxiliary lines indicating the directions of the left and right tracks and the front and rear ends of the tracks based on the rotation angle of the main body 102 relative to the tracks 101 and the dimension values ​​of the vehicle body acquired by the posture information acquisition unit S3.

[0125] exist Fig.12 , graph F4 shows the position of the right side of the crawler 101, graph F5 shows the position of the left side of the crawler 101, graph F6 shows the position of the front end of the crawler 101, and graph F7 shows the position of the rear end of the crawler 101. Thus, even when there is an influence of image disturbance due to image transmission delay and packet loss during communication, the rotation angle of the main body 102 relative to the crawler 101 can be clearly visually recognized based on the AR image drawn based on auxiliary information that has a lower capacity (lighter weight) than image data and is less susceptible to packet loss due to transmission delay.

[0126] Graphic F8 shows the positions of the four vertices of the top side of the dump truck or the like based on the position of the dump truck or the like 200 calculated by the dumping position calculation unit 10321. Thus, even when there is an influence of image turbulence associated with image transmission delay and packet loss during communication, the position of the dump truck or the like 200 can be clearly visually recognized based on the AR image drawn based on lightweight auxiliary information that is lighter than image data and less susceptible to packet loss due to transmission delay.

[0127] As an example, Fig.13 The overlapping image displayed on the display 304 when the main body 102 is operated to rotate relative to the crawler 101 by operating the operating lever 303 of the remote control system 300 is shown. The image acquired by the surrounding image acquisition unit S1 is displayed on the display 304 in a delayed state relative to the state of the working machine 100 due to the influence of various delays generated in the compression process of the image compression unit 1031, the transmission from the communication device 104 to the communication device 306, and the decompression process of the image decompression unit 3051. On the other hand, the auxiliary information generated by the auxiliary information generation unit 1032 has a smaller data capacity and a smaller delay in the encoding / decoding process than the image data. In addition, when the auxiliary information is transmitted in a data packet different from the image data by network slicing and QoS control, the delay of the auxiliary information is smaller than the delay of the image data due to the transmission from the communication device 104 to the communication device 306. Therefore, the graphic F8 drawn based on the auxiliary information can show the position of the bucket 200 of the dump truck or the like with a lower delay than the image data. In addition, the graphics F4 to F7 drawn based on the auxiliary information can show the rotation action of the working machine 100 with a lower delay than the image data. Fig.13 As shown, when an operation is performed to rotate the main body 102 relative to the crawler 101, the graph F8 can display the position of the dump truck bucket 200 in advance, and the graphs F4 to F7 can display the rotation of the main body 102 relative to the crawler 101 in advance. In addition, even if a part or the entire image is disturbed due to packet loss, the position of the dump truck bucket 200 and the rotation of the main body 102 relative to the crawler 101 can be visually recognized by overlapping the AR image in the driving control device 305 of the remote control system 300. It should be noted that in Fig.13 In FIG. 1 , the display position of the bucket 200 is offset from the display position of the graphic F8 because the coordinate system is different due to the rotation of the track 101, and thus the overlapping position of the surrounding image (the image of the bucket 200) and the AR image (graphic F8) is offset.

[0128] Fig.14An example of display in which the rotational movement of the main body 102 relative to the crawler 101 is depicted in advance with an AR image is shown. Fig.14 In the figure, the graphic F10 is a graphic display generated based on the auxiliary information, and is a graphic display showing the rotation angle of the main body 102 relative to the crawler 101 of the working machine 100, and the relationship with the surrounding vehicles and terrain. The graphic F10 includes a graphic F11 showing the front of the main body 102, a graphic F12 showing the left 90° direction of the main body 102, a graphic F13 showing the right 90° direction of the main body 102, a graphic F14 showing the direction of the crawler 101, a graphic F15 showing the direction of the working ground surface, and a graphic F16 showing the direction of the bucket 200 of a dump truck or the like. According to this superimposed image, the rotation action of the working machine 100 can be shown with a relatively low delay compared with the image. In addition, even if a part or the entire image is disturbed due to packet loss, the rotation action of the working machine 100 can be shown.

[0129] <Display during AR pre-rendering>

[0130] Fig.15 : This is a diagram showing an example of displaying F17 in the AR advance drawing, which indicates a case where the AR graphic and the image are overlapped in advance. According to the display of F17 in the AR advance drawing, when a dump truck or the like is not arranged within the detection range of the working machine 100, when the working machine 100 is stationary, the AR graphic drawn in advance is shown in the overlapping image for the remote operator of the working machine, so that erroneous operation when the operation starts or the operation is restarted can be prevented. In other words, when the working machine 100 is stationary, there will be no offset between the AR graphic and the image, but if the operation starts, the operator can recognize that the AR graphic is displayed before the image, so that erroneous operation can be prevented. In addition, when each AR graphic display is offset relative to the working image, the operator can recognize that the offset is caused by image delay rather than calibration.

[0131] It should be noted that in Fig.10 , Fig.11 , Figure 13 to Figure 15 In the example shown, all AR graphics of Figures F1 to F8 are displayed overlappingly, but depending on the position and posture of the bucket 113, the position of the work surface, the position of the bucket 200 of the dump truck, etc., and the rotation angle of the main body 102 relative to the crawler 101, Figures F1 to F8 may of course not be displayed in part or in whole in the overlapping image.

[0132] <Communication delay of video and auxiliary information: explanation of timing diagram>

[0133] Reference Figures 16 to 18, which illustrates the communication delays respectively applied to the image acquired by the surrounding image acquisition unit S1 and the auxiliary information generated by the auxiliary information generation unit 1032.

[0134] (i) Fig.16 The figure shows the communication delay generated when an overlapping image of an AR image is generated in the working machine 100 without adopting the configuration of the first embodiment and sent to the remote control system 300, or when an image acquired by the surrounding image acquisition unit S1 and auxiliary information generated by the auxiliary information generation unit 1032 are sent to the remote control system 300 in the same communication packet without performing network slicing and QoS control. In this case, since the image data and the auxiliary information are subject to the same communication delay, each information sent from the communication device 104 is received in the communication device 306 in a state where the communication delay shown in the figure is applied.

[0135] (ii) Fig.17 1 is a diagram showing communication delays respectively applied to the image acquired by the surrounding image acquisition unit S1 and the auxiliary information generated by the auxiliary information generation unit 1032 when the configuration of the first embodiment is adopted. Fig.17 As shown, the image data and the auxiliary information are sent in parallel (for example, using different communication systems) from the communication device 104 to the communication device 306. In the communication device 306, the image data is subjected to a communication delay represented by a communication delay 1, and the auxiliary information is subjected to a communication delay represented by a communication delay 2 different from the communication delay 1. Due to network slicing and QoS control, the communication delay 2 is smaller than the communication delay 1, so the communication delay applied to the auxiliary information received by the communication device 306 becomes smaller than the communication delay applied to the image data.

[0136] Therefore, if Fig.18 As shown in the timing diagram of the first embodiment, according to the configuration of the first embodiment, Fig.18 The image 1 shown in the figure can generate an AR image based on auxiliary information 2 sent from the communication device 104 to the communication device 306 after the image 1, and generate a superimposed image with the AR image superimposed, which can prevent the reduction in work efficiency caused by the delay in image transmission.

[0137] It should be noted that only the communication delay from the communication device 104 to the communication device 306 is considered here for explanation, but in addition to this, even if there is a difference between the processing delay of the compression / decompression processing of the image and the processing delay of the encoding / decoding processing of the auxiliary information, according to the configuration of the first embodiment, the reduction in operating efficiency caused by the delay can be prevented.

[0138] In addition, Fig.18In the embodiment, the timing and cycle of image data and auxiliary information sent from the communication device 104 are consistent, but of course they do not need to be consistent. Fig.18 In the figure, the delay times applied to the auxiliary information 1 and the auxiliary information 2 are made equal, but the respective delay times can vary depending on the working conditions of the communication equipment between the communication device 104 and the communication device 306. Even if such a change occurs, the effect of the first embodiment will not be impaired.

[0139] As described above, according to the first embodiment, based on the auxiliary information including the lightweight three-dimensional information generated by the auxiliary information generating unit 1032 of the working machine 100 and the image acquired by the surrounding image acquiring unit S1, the superimposed image obtained by superimposing the AR image in the driving control device 305 of the remote control system 300 is output to the display 304. As a result, the bucket 113, the bucket 200 of a dump truck, etc., the surface of the work ground, the rotation angle of the main body 102 relative to the crawler 101, etc. can be visually recognized without being hindered by the image disturbance caused by the delay of the image or the packet loss.

[0140] (2) Second Embodiment

[0141] (2-1) Basic method

[0142] Reference Figures 19 to 25 The working machine control system 1 of the second embodiment (basic mode) is described. It should be noted that the overall structure of the working machine control system 1 of the second embodiment is the same as that of the first embodiment, so repeated descriptions are omitted. The coordinate transformation processing applied to the auxiliary information in the coordinate system integration processing unit 3054 of the second embodiment is different from that of the first embodiment.

[0143] <Introduction of coordinate transformation considering rotation>

[0144] In the first embodiment, there is a time difference between the time when the surrounding image acquisition unit S1 of the working machine 100 acquires the image corresponding to the image data received by the communication device 306 of the remote control system 300 and the time when the three-dimensional information acquisition unit S2 and the posture information acquisition unit S3 of the working machine 100 acquire the information that is the basis of the auxiliary information received by the communication device 306 of the remote control system 300. Fig.18 In the case of the situation shown in the timing diagram of , the overlapping image is generated using the image 1 and the auxiliary information 2, and there is a difference in the timing of the image 1 and the auxiliary information 2 being sent from the communication device 104. Therefore, when the working machine 100 moves in a manner that the main body 102 rotates to the left relative to the crawler 101, the posture of the working machine 100 when the image 1 is acquired and the posture of the working machine 100 when the auxiliary information 2 is acquired are as follows: Fig.19Here, the coordinate system C1 is a camera coordinate system (a coordinate system indicating the camera posture) at the time when the working machine 100 acquires the image 1 and the auxiliary information 1. The coordinate system C2 is a camera coordinate system (a coordinate system indicating the camera posture) at the time when the working machine 100 acquires the image 2 and the auxiliary information 2.

[0145] In the first embodiment, when the coordinate system integration processing unit 3054 performs coordinate transformation of the auxiliary information 2, the coordinate system C2 is used as the coordinate system of the transformation target. Fig.13 The superimposed image of the AR image shown is displayed on the display 304. However, in this conversion method, when the working machine 100 turns, the display position of the AR graphic and the image of the surrounding terrain or vehicle may be offset, which may increase the burden on the operator.

[0146] Therefore, in the second embodiment (basic method and modified example), when the coordinate system integration processing unit 3054 performs coordinate transformation of the auxiliary information 2, the transformation is performed taking into account the difference between the auxiliary information 1 and the auxiliary information 2, and the coordinate system C1 is used as the coordinate system of the transformation target, thereby achieving the elimination of the offset between the image and the AR image and the image display that can perform the operation of the working machine 100 without being hindered by the image disturbance caused by the delay of the image data and the packet loss.

[0147] <Internal Configuration Example of Driving Control Device 305>

[0148] Fig. 20 1 is a diagram showing an example of the internal structure of the driving control device 305 according to the second embodiment (basic mode). Figure 6 ) is also provided with a delay time estimation unit 3057.

[0149] The delay time estimation unit 3057 estimates the delay time 1 added to the image data and the delay time 2 added to the auxiliary information based on the data received by the communication device 306. For example, the communication device 306 has a function of monitoring the communication status. In this case, the delay time estimation unit 3057 estimates the degree of delay time currently occurring based on the information on the communication status detected from the communication device 306.

[0150] The coordinate system integrated processing unit 3054 transforms the auxiliary information into information in the coordinate system C1 based on the delay time (estimated value) output from the delay time estimation unit 3057 and the correction information output from the coordinate system correction information 2 storage unit 3056. For example, the coordinate system integrated processing unit 3054 compares the delay time (first delay time) of the surrounding image data with the delay time (second delay time) of the auxiliary information, and determines the auxiliary information (second timing auxiliary information) required for the prior AR image display of the working state of the working machine 100. In addition, the coordinate system integrated processing unit 3054 calculates the difference between the first delay time and the second delay time, and calculates the transformation amount from the coordinate system C2 to the coordinate system C1 based on the difference between the rotation angle of the main body 102 relative to the crawler 101 at the auxiliary time that traces the delay time difference and the rotation angle of the main body 102 relative to the crawler 202 at the latest auxiliary time information. Then, the coordinate system integrated processing unit 3054 transforms the auxiliary information (coordinate system C2) into the coordinate system C1 of the posture of the working machine (the coordinate system of the surrounding image at the first time), generates the auxiliary information after the coordinate transformation, and then generates an AR image (auxiliary graphic). It should be noted that the other components are different from the first embodiment ( Figure 6 ) are repeated and omitted.

[0151] <Overlap image generation processing>

[0152] Fig.21 This is a diagram for explaining the driving control device 305 (see Fig. 20 ) of the flowchart of the overlapping image generation process with the AR image. Figure 8 ) and then execute step S22. It should be noted that steps S16, S17, S19, and S20 are the same as those in the first embodiment ( Figure 8 ) are the same, so the description is omitted.

[0153] In step S22 , the delay time estimation unit 3057 estimates the delay times applied to the image data and the auxiliary information, respectively, based on the data received from the communication device 306 .

[0154] In step S17 , the coordinate system integration processing unit 3054 converts the auxiliary information into information in the coordinate system C1 based on the delay time output from the delay time estimation unit 3057 and the correction information output from the coordinate system correction information 2 storage unit 3056 .

[0155] (2-2) Modification

[0156] The modification does not detect the delay time based on the information of the communication status detected by the communication device 306 as in the above basic method, but detects the delay time based on the information embedded in the image data on the transmitting side (the working machine 100 side) and reflects it in the coordinate conversion process.

[0157] <Internal Configuration Example of Sensor Information Processing Unit 103>

[0158] (i) Fig. 22 This is a diagram showing an internal configuration example 1 of the sensor information processing unit 103 in the second embodiment (variation).

[0159] In the first embodiment ( Figure 4 ) is further provided with an information embedding unit 1034. The information embedding unit 1034 embeds the rotation angle of the main body 102 relative to the crawler 101 obtained by the posture information acquisition unit S3 into some pixels of the image obtained by the surrounding image acquisition unit S1. Figure 4 ) are repeated and omitted.

[0160] (ii) Fig.23 This is a diagram showing an internal configuration example 2 of the sensor information processing unit 103 in the second embodiment (variation).

[0161] The sensor information processing unit 103 Fig. 22 The information embedding unit 1034 embeds the time stamp information in some pixels of the image acquired by the surrounding image acquisition unit S1. The auxiliary information encoding unit 1033 encodes the time stamp output by the time stamp issuing unit 1035 based on the data encoded in the first embodiment. Figure 4 ) are repeated and omitted.

[0162] <Information embedding method>

[0163] Fig.24 1 is a diagram showing an example of a method of embedding information of an image acquired by the surrounding image acquisition unit S1 into some pixels in the information embedding unit 1034 .

[0164] The information embedding unit 1034 replaces the pixel value of the pixel I11 of the part of the image I1 acquired by the surrounding image acquisition unit S1 that does not hinder the operator of the remote control system 300 with the rotation angle of the main body 102 relative to the crawler 101 acquired by the posture information acquisition unit S3 or the time stamp information output by the time stamp issuing unit 1035. It should be noted that the pixel value may be replaced by considering the number of pixels of the image input to the image compression unit 1031 in the interlaced format, the macroblock size in the compression process of the image compression unit 1031, etc. For example, when the interlaced image is compressed by H.264 with a macroblock size of 16×16, the pixel value of 32 pixels in the vertical direction and 16 pixels in the horizontal direction may be replaced with this information.

[0165] <Internal Configuration Example of Driving Control Device 305>

[0166] Fig.25 is shown with Fig. 22 and Fig.23 A diagram showing an example of the internal structure of the driving control device 305 corresponding to the sensor information processing unit 103.

[0167] The driving control device 305 is in the first embodiment ( Figure 6 ) is also provided with an information reading unit 3058.

[0168] The information reading unit 3058 reads the rotation angle of the main body 102 relative to the track 101 embedded by the information embedding unit 1034 of the sensor information processing unit 103 and acquired by the posture information acquisition unit S3 or the timestamp information output by the timestamp issuing unit 1035 from partial pixel values ​​of the image data decompressed by the image decompression unit 3051.

[0169] The coordinate system integration processing unit 3054 transforms the auxiliary information (coordinate system C2) into information in the coordinate system C1 based on the information read by the information reading unit 3058 and the auxiliary information decoded by the auxiliary information decoding unit 3053. When the information read by the information reading unit 3058 is the rotation angle of the main body 102 relative to the crawler 101, the coordinate system integration processing unit 3054 calculates the amount of transformation to the coordinate system C1 based on the difference between the rotation angle of the main body 102 relative to the crawler 101 obtained by decoding the auxiliary information and the read rotation angle. On the other hand, when the information read by the information reading unit 3058 is the time stamp information output by the time stamp issuing unit 1035, the coordinate system integration processing unit 3054 compares the time stamp information of the decoded auxiliary information with the read time stamp information, and calculates the amount of transformation from the coordinate system C2 to the coordinate system C1 based on the difference between the rotation angle of the main body 102 of the auxiliary information with respect to the crawler 101 and the rotation angle of the main body 102 of the latest auxiliary information with respect to the crawler 101. Other components are different from the first embodiment ( Figure 6 ) are repeated and omitted.

[0170] <Image compression processing and auxiliary information generation processing>

[0171] Fig.26 The sensor information processing unit 103 (see Fig. 22 or Fig.23 ) is a flowchart of image compression processing and auxiliary information generation processing. Fig.26 The processing from step S11 to step S15 is the same as the processing of the first embodiment (see Figure 7 ) are the same, so the description of each step is omitted.

[0172] In step S21 , the information embedding unit 1034 replaces part of the pixel values ​​of the image acquired by the surrounding image acquisition unit S1 with the rotation angle of the main body 102 relative to the track 101 acquired by the posture information acquisition unit S3 or the timestamp information output by the timestamp issuing unit 1035 .

[0173] <Overlap image generation processing>

[0174] Fig. 27 This is a diagram for explaining the driving control device 305 (see Fig.25 ) of the flowchart of the overlapping image generation process with the AR image. Figure 8 )The processing of step S23 is performed between the same processing of step S16 and the processing of step S17.

[0175] In step S23, the information reading unit 3058 reads the information embedded by the information embedding unit 1034 of the sensor information processing unit 103 based on the partial pixel value of the image data decompressed in step S16, and the information is the rotation angle of the main body 102 relative to the track 101 obtained by the posture information acquisition unit S3 or the timestamp information output by the timestamp issuing unit 1035.

[0176] In step S18, the coordinate system integration processing unit 3054 transforms the auxiliary information (coordinate system C2) into information in the coordinate system C1 based on the information read in step S23, that is, the rotation angle of the main body 102 relative to the crawler 101 obtained by the posture information acquisition unit S3 or the timestamp information output by the timestamp issuing unit 1035, and the auxiliary information decoded in step S17. It should be noted that the processing of steps S16, S17, S19 and S20 is different from that of the first embodiment ( Figure 8 ) are the same and the description is omitted.

[0177] <Overlap display example>

[0178] Fig.28 1 is a diagram showing an example of an image (surrounding camera image and working machine front image) displayed by processing in the second embodiment (basic mode and modified example) and an AR image displayed superimposed thereon. Fig.28 , an example of a superimposed image displayed on the display 304 when the main body 102 is operated so as to rotate relative to the crawler 101 by operating the operating lever 303 of the remote control system 300 is shown.

[0179] The AR image generated based on the auxiliary information converted into information in the coordinate system C1 is overlapped with the image acquired by the surrounding image acquisition unit S1. As a result, it is possible to draw graphics F3 to F8 whose display positions coincide with the surrounding terrain and vehicle images in the image, and a graphic F1 showing the relative position and posture of the bucket 113 relative to the surrounding terrain and vehicle in the image.

[0180] As described above, according to the second embodiment, it is possible to simultaneously achieve the elimination of the misalignment between the video and the AR image and the video display that allows the operation of the working machine 100 to be performed without being hindered by video disturbances caused by video data delay or packet loss.

[0181] (3) Third Embodiment

[0182] Reference Figure 29 to Figure 31 A working machine control system 1 according to a third embodiment will be described. It should be noted that the overall configuration of the working machine control system 1 according to the third embodiment is the same as that of the second embodiment, and therefore, duplicate descriptions will be omitted.

[0183] In the third embodiment, the auxiliary information is set as the information in the coordinate system C2 to generate an AR image, and the surrounding image is transformed into an image obtained when shooting with the coordinate system C1. That is, in the second embodiment, the image is not processed, and the overlapped image is generated by calculating where the AR image overlaps with the image, while in the third embodiment, the image is projectively transformed (for example, homography: the transformation amount is calculated based on the posture difference between C1 and C2) so that the image is deformed and overlapped with the AR image.

[0184] <Internal Configuration Example 1 of Driving Control Device 305>

[0185] Fig.29 1 is a diagram showing an example 1 of the internal structure of a driving control device 305 according to the third embodiment. The driving control device 305 has the structure of the second embodiment (basic form) (see Fig. 20 ) is provided with an image transformation unit 3060. In this example, the coordinate system integration processing unit 3054 transforms the auxiliary information into information in the coordinate system C2 (camera coordinate system). The image transformation unit 3060 calculates the transformation amount from the coordinate system C2 (camera coordinate system) to the coordinate system C1 (camera coordinate system) based on the delay time output by the delay time estimation unit 3057 and the auxiliary information decoded by the auxiliary information decoding unit 3053, and transforms the image of the coordinate system C2 into an image that can be obtained when shooting with the coordinate system C1. Other components are similar to the second embodiment ( Fig. 20 ) are repeated and the description is omitted.

[0186] <Internal Configuration Example 2 of Driving Control Device 305>

[0187] Fig.30 1 is a diagram showing an example 2 of the internal structure of the driving control device 305 according to the third embodiment. The driving control device 305 has the structure of the second embodiment (modification) (see Fig.25 ) is provided with an image conversion unit 3060. In this example, the coordinate system integration processing unit 3054 converts the auxiliary information into information in the coordinate system C2. The image conversion unit 3060 calculates the amount of conversion from the coordinate system C2 to the coordinate system C1 based on the information output by the information reading unit 3058 (the rotation angle of the main body 102 relative to the crawler 101 obtained by the posture information acquisition unit S3 or the timestamp information output by the timestamp issuing unit 1035) and the auxiliary information decoded by the auxiliary information decoding unit 3053, and converts the image of the coordinate system C2 into an image that can be obtained when shooting with the coordinate system C1. Other components are different from the second embodiment ( Fig.25 ) are repeated and the description is omitted.

[0188] <Example of overlapping images>

[0189] Fig.313 is a diagram showing an example of an image (surrounding camera image and working machine front image) displayed by the processing of the third embodiment and an AR image displayed superimposed thereon. Fig.31 , an example of a superimposed image displayed on the display 304 when the main body 102 is operated so as to rotate relative to the crawler 101 by operating the operating lever 303 of the remote control system 300 is shown.

[0190] The image (deformed image) acquired by the surrounding image acquisition unit S1 and transformed into the state acquired when photographed in the coordinate system C1 is overlapped with the AR image generated based on the auxiliary information transformed into the information in the coordinate system C2. As a result, it is possible to draw graphics F3 to F8 whose display positions coincide with the surrounding terrain and the vehicle image in the image and the graphic F1 showing the relative position and posture of the bucket 113 relative to the surrounding terrain and the vehicle in the image.

[0191] As described above, according to the third embodiment, it is possible to simultaneously achieve the elimination of the offset between the video and the AR image and the video display that allows the operation of the working machine 100 to be performed without being hindered by video disturbance caused by video delay or packet loss.

[0192] (4) Fourth Embodiment

[0193] Reference Figure 32 to Figure 34 A working machine control system 1 according to a fourth embodiment will be described. Note that the overall configuration of the working machine control system 1 according to the fourth embodiment is the same as that of the first embodiment, and therefore, duplicate descriptions thereof will be omitted.

[0194] The fourth embodiment differs from the first embodiment in that a graphic indicating the position of the last (most recent) excavated work surface or the next scheduled excavation work surface is displayed in superimposition on the superimposed image displayed on the display 304. By displaying the position of the work surface, after placing the excavated soil and the like into the bucket 200 of a dump truck or the like, when the main body 102 is rotated relative to the crawler 101 toward the next scheduled excavation work surface, the operation of the working machine 100 can be performed without being hindered by image disturbance caused by delay in image data or packet loss.

[0195] <Internal Configuration Example of Driving Control Device 305>

[0196] Fig.32 1 is a diagram showing an example of the internal configuration of the driving control device 305 according to the fourth embodiment. Figure 6 ) is also provided with a surface storage unit 3059 for the working object.

[0197] The work target ground surface storage unit 3059 stores the position of the work target ground surface that the work machine 100 is scheduled to excavate next (which can be specified in advance by the operator).

[0198] The coordinate system integration processing unit 3054 performs coordinate transformation on the auxiliary information together with the position of the work target ground surface output from the work target ground surface storage unit 3059 .

[0199] The auxiliary graphic generating unit 3055 generates an AR image based on the position of the work target ground surface and the auxiliary information output from the work target ground surface storage unit 3059 after being transformed by the coordinate system integration processing unit 3054 .

[0200] It should be noted that, when the last excavated working surface is displayed, the information of the scheduled excavation working surface is not required, but when the last excavated working surface and the scheduled excavation working surface are displayed at the same time, the necessary information is read from the working surface storage unit 3059. Figure 6 ) are repeated and omitted.

[0201] <Example of overlapping images>

[0202] Fig.33 The diagram shows an example of images (surrounding camera images and working machine front image) displayed by the processing of the fourth embodiment and an AR image displayed superimposed thereon.

[0203] exist Fig.33 In the first embodiment ( Fig.10 ) further includes a graphic F9 representing the last excavated working surface and / or the scheduled excavated working surface. The graphic F9 is a graphic generated based on the position information of the last excavated working surface and / or the position information of the scheduled excavated working surface output from the working surface storage unit 3059. In this way, by overlapping and displaying the last excavated working surface and / or the scheduled excavated working surface as an AR image, the operator can visually recognize the target position when the main body 102 is rotated relative to the crawler 101 toward the next scheduled excavated working surface after placing the excavated sand and the like into the bucket 200 of the dump truck or the like.

[0204] It should be noted that, as for the position of the working object ground surface stored in the working object ground surface storage unit 3059, for example, the coordinate value of the ground where the front end of the bucket 113 touched the ground when the working machine 100 last excavated can be stored in advance, or Fig.34As shown, the positions of the ground to be excavated a predetermined number of times are stored in advance by designating them with marks P1 to P4 on a GUI (Graphical User Interface) that displays the vehicle body in a bird's-eye view.

[0205] As described above, according to the fourth embodiment, when the main body 102 is rotated relative to the crawler belt 101 toward the next work surface to be excavated, the operation of the working machine 100 can be performed without being hindered by image disturbance caused by delay or packet loss of image data.

[0206] (5) Modification

[0207] In the first to fourth embodiments, the case where the working machine 100 is a so-called backhoe has been described. However, even when the bucket of the working machine 100 is a loading shovel facing forward, the same configuration and processing as those in the first to fourth embodiments can be applied.

[0208] <AR image display status switching function>

[0209] Fig.35 It is a conceptual diagram for explaining the switching of the display state of the AR image in the working machine control system 1 according to each of the above-mentioned embodiments.

[0210] After the working machine 100 moves to the excavation site (S31) and waits (S32), it drives the bucket 113 to perform excavation (S33). When performing excavation, the working machine 100 turns (moves forward) (S34) and discharges soil into the bucket 200 (S35). When the discharge of soil is completed, the bucket 113 turns (returns) to the excavation site again (S36) and continues excavation (S33).

[0211] In the excavation stage (S33), it can be set to display a graphic F1 representing the bucket 113, a graphic F3 representing the position of the surface of the working object, and a graphic F2 connecting the graphic F1 and the graphic F3. In addition, when rotating (S34 and S36), it can be set to further display graphics F4 to F7 representing the direction of the crawler 101, a graphic F8 representing the position of the bucket 200 of the dump truck, etc., and a graphic F9 representing the surface of the working ground scheduled to be excavated next. These display settings can be made for each operator (a user who remotely operates the loading shovel in the operation room). For example, the setting of the display state switching is pre-stored in a storage device not shown in the figure according to the operator (what setting value is displayed at which stage), and the setting value is read from the storage device during operation. In this way, the display switching of easy-to-operate AR images customized for each user can be realized.

[0212] In addition, the brightness and contrast of the captured image change according to the weather, time, and shooting direction (front light or backlight, etc.), so adjusting the brightness and contrast of the AR image displayed in overlap (reducing the brightness in the case of dark images such as at night and overlapping them in the case of bright images such as during the day) is also effective in improving visual recognition. The setting value of the brightness can also be pre-stored by the user and read during operation.

[0213] (6) Summary

[0214] (i) In the embodiment of the present disclosure, the driving control device 305 generates an AR image (auxiliary graphic) representing the posture of the working machine 100 and the position of the working object based on the auxiliary information received from the working machine 100, and displays the surrounding image (image 1: see Fig.18 ) and auxiliary information (auxiliary information 2: see Fig.18 ) and the AR image generated is overlapped and displayed on the display device. Thus, even if the surrounding images of the working machine 100 are delayed, the remote operator of the working machine 100 can grasp the actual working status based on the AR image generated from the low-capacity (light) auxiliary information, thereby preventing the working efficiency from being reduced. Here, the driving control device 305 transforms the auxiliary information represented by the body coordinate system of the working machine 100 into the camera coordinate system, and generates an AR image (auxiliary graphic) based on the auxiliary information after the coordinate transformation. Thus, the AR image can be displayed in the same coordinate system as the coordinate system of the surrounding images, and the AR image corresponding to the actual working action of the working machine 100 can be displayed in advance with respect to the delayed surrounding images.

[0215] (ii) The driving control device 305 generates a graphic including a graphic indicating the position and posture of the working machine 100 (see Fig.10 etc.), a graphic showing the topographic position of the work object of the work machine 100 (see Fig.33 ), a graphic showing the position of a vehicle (for example, a dump truck) that is the work object of the work machine 100 (see Fig.10 etc.), or a graphic showing the posture of the traveling body (crawler) of the working machine 100 (see Fig.10 In this way, by displaying a minimal AR image, it is possible to provide operation assistance that avoids confusion of the remote operator (displaying too much information may cause confusion).

[0216] (iii) The driving control device 305 displays information indicating that an AR image based on auxiliary information (auxiliary information 2) acquired later than the work image (image 1) is displayed superimposed on the work image together with the superimposed image (see Fig.15 ). This makes it possible to prevent erroneous operations by the remote operator when starting or resuming an operation.

[0217] (iv) In the second embodiment, the driving control device 305 changes the overlapping position and shape of the AR image (auxiliary graphic) based on the angle difference between the first angle formed by the traveling body (crawler) of the working machine 100 and the rotating body (main body) of the working machine at the first timing and the second angle formed by the traveling body and the rotating body at the second timing (see Fig.28 ).

[0218] When calculating the above-mentioned angle difference, the delay time of the surrounding image (first delay time) and the delay time of the auxiliary information (second delay time) can be used. That is, the driving control device 305 compares the first delay time with the second delay time to determine the surrounding image to be generated with the first time (image 1: see Fig.18 ) The second timing auxiliary information of the AR image overlapped (auxiliary information 2: see Fig.18 ), transform the auxiliary information at the second timing into the coordinate system of the posture of the working machine 100 at the first timing, and generate an AR image based on the auxiliary information at the second timing after the coordinate transformation.

[0219] In addition, as another example of calculating the above-mentioned angle difference, the angle information (the first angle information) embedded in the surrounding image can also be used. In this case, the driving control device 305 calculates the angle difference based on the first angle information obtained (separated) from the data of the surrounding image and the second angle information obtained from the second timing auxiliary information (auxiliary information 2).

[0220] As another example of calculating the above-mentioned angle difference, the timestamp (the first moment information corresponding to the first timing) embedded in the working pixel may also be used. In this case, the driving control device 305 calculates the above-mentioned angle difference based on the angle of the traveling body (crawler track) and the rotating body (main body) of the auxiliary information that traces the difference between the timestamp (the first moment information) obtained from the data of the surrounding image and the second moment information obtained from the auxiliary information and the angle of the traveling body and the rotating body of the latest auxiliary information (the auxiliary information of the second timing).

[0221] (v) In the third embodiment, the driving control device 305 deforms the surrounding image based on the angle difference between the first angle formed by the traveling body (crawler) of the working machine 100 and the rotating body (main body) of the working machine 100 at the first timing and the second angle formed by the traveling body (crawler) and the rotating body (main body) at the second timing (see Fig.33 ) In the third embodiment, the same method of calculating the angle difference as in the second embodiment can also be adopted.

[0222] (vi) In the fourth embodiment, the driving control device 305 overlaps an AR image (auxiliary graphic) indicating a planned position (planned excavation position) of the work object based on the work machine 100 with the surrounding image based on the information on the position of the work object (see Fig.33 ). Thus, the remote operator can easily recognize the position to be worked on next in the working machine 100. In addition, the position of the work object of the last work may be displayed in an AR image.

[0223] (vii) In the modified example, the driving control device 305 switches the display mode of the auxiliary graphics (AR images) according to the working state (moving, waiting, digging, swinging, earth dropping, etc.) of the working machine 100 (for example, only displaying a part of the necessary AR images, etc.).

[0224] (viii) The functions of the embodiments of the present disclosure can also be implemented by the program code of the software. In this case, a storage medium having the program code recorded thereon is provided to a system or device, and the computer (or CPU, MPU) of the system or device reads the program code stored in the storage medium. In this case, the program code read from the storage medium itself implements the functions of the aforementioned embodiments, and the program code itself and the storage medium storing the program code constitute the present disclosure. As a storage medium for supplying such a program code, for example, a floppy disk, a CD-ROM, a DVD-ROM, a hard disk, an optical disk, a magneto-optical disk, a CD-R, a magnetic tape, a non-volatile memory card, a ROM, etc. are used.

[0225] In addition, based on the instructions of the program code, the OS (operating system) running on the computer may perform part or all of the actual processing, and the functions of the aforementioned embodiments may be realized through the processing. In addition, after the program code read from the storage medium is written into the memory on the computer, based on the instructions of the program code, the CPU of the computer may perform part or all of the actual processing, and the functions of the aforementioned embodiments may be realized through the processing.

[0226] In addition, the program code of the software that realizes the functions of the implementation method can also be published via the network, thereby storing it in a storage device such as a hard disk or memory of the system or device or a storage medium such as CD-RW or CD-R. When in use, the computer (or CPU, MPU) of the system or device reads and executes the program code stored in the storage device or the storage medium.

[0227] In addition, the technology disclosed in the present invention is not limited to the above-mentioned embodiments, and includes a variety of modified examples. For example, the above-mentioned embodiments are described in detail in order to clearly and easily explain the technology disclosed in the present invention, but are not necessarily limited to having all the described structures. In addition, a part of the structure of a certain embodiment can be replaced with the structure of other embodiments, and the structure of other embodiments can be added to the structure of a certain embodiment. In addition, other structures can be added / deleted / replaced for a part of the structure of each embodiment. In addition, the above-mentioned structures, functions, processing parts, processing units, etc. can also be partially or completely implemented by hardware through, for example, integrated circuit design, etc.

[0228] Description of Reference Numerals

[0229] 1 operating machine control system, 100 operating machine (construction machine), 101 crawler, 102 main body, 103 sensor information processing unit, 104 communication device, 107 antenna, 111 boom, 112 arm, 113 bucket, 200 bucket, 300 remote control system, 301 driver's seat, 302 operation control unit, 303 input device (operating lever), 304 display, 305 driving control device, 306 communication device, 1031 image compression unit, 1032 auxiliary information generation unit, 1033 auxiliary information encoding unit, 1034 information embedding unit, 1035 time stamp issuing unit, 10321 dumping position calculation unit, 10322 working ground surface height calculation unit, 10323 Bucket position / posture calculation unit, 10324 coordinate system correction information 1 storage unit, 3051 image decompression unit, 3052 image overlapping unit, 3053 auxiliary information decoding unit, 3054 coordinate system comprehensive processing unit, 3055 auxiliary graphics generation unit, 3056 coordinate system correction information 2 storage unit, 3057 delay time estimation unit, 3058 information reading unit, 3059 work object ground surface storage unit, 3060 image transformation unit, L0, L1, L2 ground, L4 wall, Lg corresponding ground, OR operation room, S1 surrounding image acquisition unit, S2 three-dimensional information acquisition unit, S3 posture information acquisition unit, F1~F17 graphics, C1, C2 coordinate system, I1 surrounding image, P1~P4 marks.

Claims

1. A remote control system comprising a display device that receives and displays a variety of information from a working machine via a communication device and a control device that controls the display device, and remotely controls the working machine. The remote control system is characterized in that: The communication device receives an image of the surroundings of the working machine photographed by the working machine and auxiliary information including posture information of the working machine and three-dimensional information of the surroundings of the working machine acquired by the working machine. The control device performs the following processing: generating an auxiliary graphic indicating a posture of the working machine and a position of a working object based on the auxiliary information received by the communication device; and The surrounding image captured at a first timing in the working machine and received by the communication device is overlapped with the auxiliary graphic and displayed on the display device. The auxiliary graphic is generated based on the auxiliary information acquired at a second timing after the first timing in the working machine and received by the communication device.

2. The remote control system according to claim 1, characterized in that: The control device transforms the auxiliary information represented by the vehicle body coordinate system of the working machine into a camera coordinate system, and generates the auxiliary graphic from the auxiliary information after the coordinate transformation.

3. The remote control system according to claim 1, characterized in that: The control device generates the auxiliary graphics including at least one of a graphic representing the position and posture of the working machine, a graphic representing the terrain position of the working object of the working machine, a graphic representing the position of the vehicle of the working object of the working machine, or a graphic representing the posture of the traveling body of the working machine.

4. The remote control system according to claim 1, characterized in that: The control device changes the overlapping position and shape of the auxiliary graphics based on the angle difference between a first angle and a second angle, wherein the first angle is the angle formed by the traveling body of the working machine and the rotating body of the working machine at the first timing, and the second angle is the angle formed by the traveling body and the rotating body at the second timing.

5. The remote control system according to claim 1, characterized in that: The control device deforms the surrounding image based on the angle difference between a first angle and a second angle, wherein the first angle is an angle formed by a traveling body of the working machine and a rotating body of the working machine at the first timing, and the second angle is an angle formed by the traveling body and the rotating body at the second timing.

6. The remote control system according to claim 1, characterized in that: The control device performs the following processing: estimating a first delay time of the surrounding image and a second delay time of the auxiliary information based on information on the communication status detected by the communication device; determining the auxiliary information of the second timing based on a difference between the first delay time and the second delay time, the auxiliary information being used to generate the auxiliary pattern to be overlapped with the surrounding image of the first timing; transforming the auxiliary information at the second timing into a coordinate system of the posture of the working machine at the first timing, thereby generating auxiliary information at the second timing after coordinate transformation; as well as The auxiliary pattern is generated from the auxiliary information at the second timing after the coordinate conversion.

7. The remote control system according to claim 4, characterized in that: The data of the surrounding image includes a data portion representing information of the first angle, The control device calculates the angle difference based on the information of the first angle acquired from the data of the surrounding image and the information of the second angle acquired from the auxiliary information at the second timing.

8. The remote control system according to claim 5, characterized in that: The data of the surrounding image includes a data portion representing information of the first angle, The control device calculates the angle difference based on the information of the first angle acquired from the data of the surrounding image and the information of the second angle acquired from the auxiliary information at the second timing.

9. The remote control system according to claim 4, characterized in that: The data of the surrounding image includes a data portion indicating first time information corresponding to the first timing. The auxiliary information includes second time information corresponding to the second timing. The control device calculates the angle difference based on a difference between the first time information acquired from the data of the surrounding image and the second time information acquired from the auxiliary information.

10. The remote control system according to claim 5, characterized in that: The data of the surrounding image includes first time information corresponding to the first timing. The auxiliary information includes second time information corresponding to the second timing. The control device calculates the angle difference based on a difference between the first time information acquired from the data of the surrounding image and the second time information acquired from the auxiliary information.

11. The remote control system according to claim 1, characterized in that: The control device generates the auxiliary graphic based on the information on the position of the work object, and the auxiliary graphic includes a graphic indicating a predetermined position of the work object by the work machine.

12. The remote control system according to claim 1, characterized in that: The control device switches the display mode of the auxiliary graphics according to the working state of the working machine.

13. The remote control system according to claim 1, characterized in that: The control device displays information indicating that the auxiliary graphic acquired at a later time than the surrounding image is displayed superimposed on the surrounding image together with the superimposed image of the auxiliary graphic and the surrounding image.

14. A remote-operated working machine system, characterized in that: A remote control system according to claim 1 for remotely controlling a working machine and the working machine, The working machine comprises: A camera device for capturing the surrounding images; A posture sensor for acquiring posture information of the working machine; an object detection sensor that acquires three-dimensional information of the working machine; and A working machine communication device communicates with the remote control system and sends the surrounding image and the auxiliary information including the posture information and the three-dimensional information.

15. A method for controlling a work information display, which displays work information received from a work machine on a screen, The operation information display control method is characterized by comprising the following processing: The control device for controlling the operation of the working machine acquires a surrounding image of the working machine photographed by the working machine and auxiliary information including posture information of the working machine and three-dimensional information of the surroundings of the working machine; The control device generates an auxiliary graphic indicating the posture of the working machine and the position of a working object from the auxiliary information; and The control device displays the surrounding image captured at a first timing in the working machine on a display device by superimposing the auxiliary graphic generated based on the auxiliary information acquired at a second timing after the first timing in the working machine.

Images