Remote control system, remotely operated work machine system, and work information display control method

Abstract

To improve work efficiency of an operator by preventing a decrease in the work efficiency due to video disturbance caused by video transmission delay and communication packet loss during communication in a series of operations during remote control of a work machine.SOLUTION: In order to solve the above problems, the present disclosure proposes a remote control system that remotely controls a work machine comprising a communication device that communicates with the work machine, and a control device that controls the operation of the work machine and also processes surrounding images from the work machine and displays it on a display device. The communication device receives a surrounding image taken by the work machine and support information including posture information of the work machine and three-dimensional information of the surroundings of the work machine. The control device executes processing for generating a support figure indicating a posture of the work machine and a position of a work target from the support information, and processing for superimposing a surrounding image acquired at a first timing in the work machine and a support figure generated from support information acquired at a second timing after the first timing in the work machine, and displaying it on the display device.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] The present disclosure relates to a remote control system, a remotely operated work machine system, and a work information display control method.

Background Art

[0002] Although a work machine control system that enables remote operation of a work machine has been put into practical use, the work efficiency may decrease due to the transmission delay caused by transmitting video from the work machine to a remote control system (remote operation room). To address this, for example, Patent Document 1 proposes processing video so as to correct the delay time involved in transmitting video for remote operation unit work and displaying it on a display. By such processed video display, when the reception delay time of the video in the remote operation room where remote operation is performed is large, the efficiency of remote operation can be improved.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, according to Patent Document 1, although the transmission delay of the video around the work machine (own vehicle) is improved, since it does not assume that the own vehicle will be shown, when a part of the own vehicle is shown in the video, the display position of the part of the own vehicle in the processed video may shift in relation to the surrounding image, which may prevent efficient operation by the operator. Also, in the case where video disturbance occurs due to communication packet loss during communication, there is a possibility of preventing efficient operation.

[0005] In light of these circumstances, this disclosure provides a technology that prevents a decrease in work efficiency due to video transmission delays and video distortion caused by communication packet loss during communication in a series of operations while remotely controlling a work machine, thereby improving the work efficiency of the operator. [Means for solving the problem]

[0006] To solve the above problems, this disclosure proposes a remote control system for remotely controlling a work machine, comprising: a communication device that communicates with the work machine; and a control device that controls the operation of the work machine and processes ambient video from the work machine and displays it on a display device, wherein the communication device receives ambient video taken by the work machine and support information including the posture information of the work machine and three-dimensional information of the surroundings of the work machine, and the control device performs the following processes: generating a support figure from the support information that indicates the posture of the work machine and the position of the work object, and superimposing the ambient video acquired by the work machine at a first timing and the support figure generated from the support information acquired by the work machine at a second timing later than the first timing, and displaying it on a display device.

[0007] Further features relating to this disclosure will become apparent from the description herein and the accompanying drawings. Furthermore, aspects of this disclosure are achieved and realized by elements and various combinations of elements and the modes of the claims described herein and the accompanying claims. It should be understood that the descriptions herein are merely typical examples and do not limit in any way the claims or applications of this disclosure. [Effects of the Invention]

[0008] According to the technology disclosed herein, it is possible to prevent a decrease in work efficiency due to video transmission delays during a series of operations while remotely controlling a work machine. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the external configuration of the work machine control system 1. [Figure 2] Figure 2 is a schematic diagram illustrating the configuration of the surrounding image acquisition unit S1 and the three-dimensional information acquisition unit S2 mounted on the work machine 100. [Figure 3] This figure shows an example of the internal configuration of the remote control system (operation room) 300. [Figure 4] This figure shows an example of the internal configuration of the sensor information processing unit 103 mounted on the work machine 100. [Figure 5] This figure shows an example of the internal configuration of the support information generation unit 1032. [Figure 6] This is a diagram showing an example configuration of the remote control system (remote control room) 300. [Figure 7] This flowchart illustrates the video compression processing performed by the sensor information processing unit 103 and the support information generation processing performed by the support information generation unit 1032. [Figure 8] Figure 6 is a flowchart illustrating the process of generating superimposed images with AR images using the operation control device 305. [Figure 9] This figure shows an example of an image obtained by stretching (image of the front of the work machine + image of the surroundings). [Figure 10] This figure shows an example of a superimposed image obtained by superimposing an AR image onto an image obtained by stretching (Figure 9). [Figure 11] This figure shows an example of the superimposed image (front view of the work machine + surrounding view + AR image) displayed when the bucket 113 is lifted vertically. [Figure 12] This figure shows an example of the correspondence between the auxiliary lines indicating the orientation of the crawler 101 and the vehicle body in the AR image displayed in the superimposed image. [Figure 13] This figure shows an example of an AR image displayed in an overlaid image. [Figure 14] This figure shows another example of an AR image displayed in an overlaid image. [Figure 15] This figure shows an example of AR pre-drawing display F17, which indicates that AR shapes are being superimposed on the video in advance. [Figure 16]When the configuration of the first embodiment is not adopted, a superimposed video with an AR image superimposed therein is generated within the working machine 100 and transmitted to the remote control system 300, or when network slicing and QoS control are not performed and the video acquired by the surrounding video acquisition unit S1 and the support information generated by the support information generation unit 1032 are transmitted to the remote control system 300 in the same communication packet, this is a diagram showing the communication delay that occurs. [Figure 17] This is a diagram showing the communication delay added to each of the video acquired by the surrounding video acquisition unit S1 and the support information generated by the support information generation unit 1032 when the configuration of the first embodiment is adopted. [Figure 18] This is a diagram showing an example of a superimposed video obtained by superimposing an AR image generated based on support information 2 transmitted from the communication device 104 to the communication device 306 after video 1 on video 1. [Figure 19] This is a diagram showing an example of the posture of the working machine 100 at the time of acquiring video 1 and an example of the posture of the working machine 100 at the time of acquiring support information 2 when the working machine 100 operates to turn the main body 102 to the left with respect to the crawler 101. [Figure 20] This is a diagram showing an example of the internal configuration of the operation control device 305 according to the second embodiment (basic form). [Figure 21] This is a flowchart for explaining the superimposed video generation process having an AR image by the operation control device 305 (refer to FIG. 20) of the second embodiment (basic form). [Figure 22] This is a diagram showing an example 1 of the internal configuration of the sensor information processing unit 103 in the second embodiment (modified example). [Figure 23] This is a diagram showing an example 2 of the internal configuration of the sensor information processing unit 103 in the second embodiment (modified example). [Figure 24] This is a diagram showing an example of a method of embedding information into some pixels of the video acquired by the surrounding video acquisition unit S1 in the information embedding unit 1034. [Figure 25] This is a diagram showing an example of the internal configuration of the operation control device 305 corresponding to the sensor information processing unit 103 in FIGS. 22 and 23. [Figure 26]This is a flowchart illustrating the video compression processing and support information generation processing by the sensor information processing unit 103 (see Figure 21 or Figure 22) of the second embodiment (modified version). [Figure 27] This is a flowchart illustrating the superimposed image generation process having an AR image by the operation control device 305 (see Figure 25) of the second embodiment (modified version). [Figure 28] This figure shows examples of video (surrounding image and front view of the work machine) and AR images superimposed thereon, which are displayed by the processing of the second embodiment (basic form and modified form). [Figure 29] This figure shows an example of the internal configuration 1 of the operation control device 305 according to the third embodiment. [Figure 30] This figure shows an example of the internal configuration 2 of the operation control device 305 according to the third embodiment. [Figure 31] This figure shows examples of video (surrounding image and front view of the work machine) and AR images superimposed thereon, which are displayed by the processing of the third embodiment. [Figure 32] This figure shows an example of the internal configuration of the operation control device 305 according to the fourth embodiment. [Figure 33] This figure shows examples of video (surrounding image and front view of the work machine) and AR images superimposed thereon, which are displayed by the processing of the fourth embodiment. [Figure 34] This figure shows an example of a GUI method for specifying the work surface location stored in the work surface memory unit in the fourth embodiment. [Figure 35] This is a conceptual diagram illustrating the switching of the display state of AR images in the work machine control system 1 of each embodiment. [Modes for carrying out the invention]

[0010] This embodiment will be described below with reference to the attached drawings. In the attached drawings, functionally identical elements may be indicated by the same number. The attached drawings show embodiments and implementation examples in accordance with the principles of this disclosure, but they are for the purpose of understanding this disclosure and are not to be used in any way to restrict the interpretation of this disclosure. The descriptions in this specification are merely typical examples and do not limit the claims or applications of this disclosure in any way.

[0011] While this embodiment is described in sufficient detail for those skilled in the art to implement the disclosure, it is important to understand that other implementations and forms are possible, and that the configuration and structure can be modified and various elements replaced without departing from the scope and spirit of the technical idea of ​​this disclosure. Therefore, the following description should not be construed as limiting to this.

[0012] (1) First Embodiment <Example of a control system configuration for industrial machinery> Referring to Figures 1 to 3, a schematic configuration example of a work machine control system (remotely operated work machine system) 1 according to the first embodiment will be described. Figure 1 is a diagram showing an example of the external configuration of the work machine control system 1. Figure 2 is a schematic diagram illustrating the configuration of the surrounding image acquisition unit S1 and the three-dimensional information acquisition unit S2 mounted on the work machine 100. Figure 3 is a diagram showing an example of the internal configuration of the remote control system (operation room) 300.

[0013] As shown in Figure 1, the work machine control system 1 comprises a remote control system 300 and a work machine (construction machine) 100 controlled by the remote control system 300.

[0014] The working machine 100 is, for example, a backhoe with a bucket 113 facing the control room OR. The operation of the working machine 100 excavates the ground L that is the target of construction, creating, for example, a flat surface L0, a flat surface L1, a slope L2, etc. The soil excavated by the working machine 100 is discharged (released) towards the vessel 200, such as a dump truck, when the working machine 100 rotates and the bucket 113 moves to the vessel 200.

[0015] The backhoe work machine 100 includes, for example, a crawler 101, a main body 102 supported and transported by the crawler 101, an operating room OR mounted on the main body 102, a sensor information processing unit 103, a communication device 104, various sensors S1 to S4, an antenna 107, a boom 111, an arm 112, and a bucket 113. The boom 111, arm 112, and bucket 113 are driven by an engine, hydraulic cylinder, etc. (not shown) to perform excavation of the ground L to be constructed.

[0016] Furthermore, the crawler 101 is driven by a drive unit (not shown), allowing the main body 102 to move forward, backward, left, and right. The main body 102 is also configured to rotate relative to the crawler 101, and its rotation direction and angle are controlled by a drive unit (not shown). The work machine 100 is configured to be operated unmanned and remotely controlled by a remote control system 300, and the operator does not need to be in the control room OR.

[0017] The main unit 102 includes (is mounted on) a sensor information processing unit 103 and a communication device 104. The sensor information processing unit 103 compresses the image of the ground and bucket 113, which are the construction target, acquired by the surrounding image acquisition unit S1 including an image sensor (this is an image of the area including the work area of ​​the work machine 100, which is the work area of ​​the work machine 100 and its surroundings, and may be referred to as the surrounding image below). The sensor information processing unit 103 also generates support information, including the current position of the bucket 113, the work ground surface located directly below the bucket 113, and position information (distance information from a reference point) of the vessel 200 of a dump truck, etc., based on the sensor information obtained from the three-dimensional information acquisition unit S2 and the attitude information acquisition unit S3. This support information may consist of the sensor information itself obtained from the three-dimensional information acquisition unit S2 and the attitude information acquisition unit S3, or it may consist of information obtained by converting the sensor information to the coordinate system of the main unit (vehicle body coordinate system) of the work machine 100. However, the latter is preferable considering the amount of computation required for subsequent processing.

[0018] The communication device 104 is configured to transmit acquired video data and support information to the remote control system 300, and to receive control information from the remote control system 300.

[0019] The main unit 102 includes, at a predetermined location, an ambient image acquisition unit S1 that acquires images of the front part (boom 111, arm 112, bucket 113) of the work machine 100 and surrounding images, a three-dimensional information acquisition unit S2 that detects the presence and distance of objects within the field of view, and a posture information acquisition unit S3 that detects the posture of the work machine 100. As shown in Figure 2, as an example, the ambient image acquisition unit S1 is an imaging device that images the area around the work machine 100, and is, for example, a camera having a CMOS sensor that images the field of view SR1 in front of the work machine 100.

[0020] The three-dimensional information acquisition unit S2 is an object detection sensor such as a LiDAR (Light Detection and Ranging) or ultrasonic sensor that detects the presence and distance of objects within the field of view SR2. The three-dimensional information acquisition unit S2 measures the distance from sensor S2 as sensor information, and outputs three-dimensional information obtained by converting the measured distance into information in the main unit's coordinate system. The three-dimensional information acquisition unit S2 is not limited to this and may consist of multiple LiDAR or ultrasonic sensors, such as sensor S21 that detects the field of view SR21 and sensor S22 that detects the field of view SR22. The field of view is not limited to these three directions, and detection of the entire surroundings may be performed by adjusting the installation position, angle, and number of sensors according to the field of view of the sensors. In addition, the surrounding image acquisition unit S1 and the three-dimensional information acquisition unit S2 may consist of one or more stereo cameras capable of imaging and stereoscopic viewing.

[0021] The posture information acquisition unit S3 is a posture sensor that detects the posture of the work machine 100, and is capable of acquiring the posture of the boom 111, arm 112, and bucket 113 (for example, the angle between the boom 111 and the main body 102, and the angles of the arm 112 and bucket 113 from the connecting part) and the rotation angle of the main body 102 relative to the crawler 101 (0 degrees can be defined as the state in which the crawler 101 and the main body 102 are facing the same direction). The posture information acquisition unit S3 can be composed of, for example, a gyro sensor, a tilt sensor, an acceleration sensor, or a combination thereof. Furthermore, the example shown in Figure 2 is just one example, and this disclosure is not limited thereto.

[0022] Referring to Figure 3, an example of the configuration details of the remote control system 300 will be described. This remote control system 300, as an example, includes a driver's seat 301, an operation control unit 302, an operation rod 303, a display 304, an operation control device 305, and a communication device 306.

[0023] The driver's seat 301 is a chair in which the operator sits, and an operation control unit 302 and an operation lever 303 are provided on its side. In addition to the operation lever 303, various levers, switches, pedals, etc., may be arranged, and their operation may be reflected in the operation of the work machine 100. When the operation lever 303, etc. are operated, a corresponding operation signal is generated by the operation control unit 302 and transmitted to the operation control device 305. The operation control device 305 generates a control signal to drive the work machine 100 according to the received operation signal and transmits it to the work machine 100 via the communication device 306.

[0024] The display 304 is positioned in front of the driver's seat 301 and is a display unit that displays superimposed images having the bucket 113, the ground to be constructed, and the AR (Augmented Reality) image described later. As shown in Figure 3, the display 304 can be composed of multiple display devices given appropriate depression angles to realistically represent three-dimensional space. The display 304 is not limited to this and may consist of only one display device or may be a head-mounted display.

[0025] <Example of internal configuration of sensor information processing unit 103> Figure 4 shows an example of the internal configuration of the sensor information processing unit 103 mounted on the work machine 100.

[0026] The sensor information processing unit 103 includes, as an example, a video compression unit 1031, a support information generation unit 1032, and a support information encoding unit 1033. The sensor information processing unit 103 may be a general-purpose computer, and may consist of, for example, a CPU 151 (processor), ROM 152, RAM 153, a hard disk drive 154, an input device 155, and a display 156 (display unit). The CPU 151 is one form of processor and may be a GPU or other semiconductor device capable of arithmetic processing, or a combination thereof.

[0027] The video compression unit 1031 compresses the ambient video acquired from the ambient video acquisition unit S1 using a predetermined video compression method and outputs it to the communication device 104. The video compression method can be a well-known method such as MPEG, H.264, or MotionJPEG, but is not limited to these.

[0028] Based on the three-dimensional information acquired from the three-dimensional information acquisition unit S2 and the attitude information acquired from the attitude information acquisition unit S3, the support information generation unit 1032 calculates the position and attitude of the bucket 113, the height of the ground (working surface) directly below the bucket 113, and the position of the vessel 200 of the dump truck or the like.

[0029] The support information encoding unit 1033 encodes the support information (position and orientation of the bucket 113, height of the work surface, position of the vessel 200 of the dump truck, etc., and the turning angle of the main body 102 relative to the crawler 101) output from the support information generation unit 1032 using a predetermined communication protocol and outputs it to the communication device 104. Well-known methods such as the CAN (Controller Area Network) communication protocol defined in ISO 11898 can be used for encoding, but are not limited to these.

[0030] <Example of internal configuration of support information generation unit 1032> Figure 5 shows an example of the internal configuration of the support information generation unit 1032. The support information generation unit 1032 processes the three-dimensional information acquired from the three-dimensional information acquisition unit S2 and the attitude information acquired from the attitude information acquisition unit S3 to generate support information that includes lightweight (low-capacity) three-dimensional information. As an example, this support information generation unit 1032 includes a coordinate system correction information 1 storage unit 10324, a dump position calculation unit 10321, a bucket position / attitude calculation unit 10323, and a work ground surface height calculation unit 10322.

[0031] The coordinate system correction information 1 storage unit 10324 stores correction information for integrating the coordinate systems of the detection signals of the three-dimensional information acquired from the three-dimensional information acquisition unit S2 and the attitude information acquired from the attitude information acquisition unit S3. Specifically, this correction information is 6-degree-of-freedom (6-dimensional vector: translation and rotation in the xyz directions) information for transforming the sensor reference (sensor coordinate system) information to the vehicle body reference (main body coordinate system) information.

[0032] The dump position calculation unit 10321, the bucket position and attitude calculation unit 10323, and the work surface height calculation unit 10322 output support information based on the correction information, for example, as data for a coordinate system (body coordinate system) that rotates in accordance with the rotation of the body 102 relative to the crawler 101, with the pivot axis of the work machine 100 and the bottom surface of the body 102 as the reference point. This correction information may be calibrated when the work machine 100 is shipped or inspected, or an automatic calibration process may be performed when the work machine 100 is started.

[0033] The dump position calculation unit 10321 calculates the position of the vessel 200 of a dump truck or the like (position in the main body coordinate system) based on the three-dimensional information acquired from the three-dimensional information acquisition unit S2 and the correction information acquired from the coordinate system correction information 1 storage unit 10324. The position of the vessel 200 of a dump truck or the like is calculated as three-dimensional coordinate values ​​indicating the positions of the four vertices at the top of the vessel, as an example.

[0034] The bucket position and attitude calculation unit 10323 calculates the position and attitude of the bucket 113 (position and attitude in the main body coordinate system) based on attitude information acquired from the attitude information acquisition unit S3 and correction information acquired from the coordinate system correction information 1 storage unit 10324. The position of the bucket 113 is, for example, the three-dimensional coordinate value of the joint between the bucket 113 and the arm 112. The attitude of the bucket 113 is calculated, for example, as a three-dimensional unit direction vector connecting the joint between the bucket 113 and the arm 112 and the center of the tip of the bucket 113.

[0035] The work surface height calculation unit 10322 calculates the height of the work surface (height in the main body coordinate system) based on the three-dimensional information acquired from the three-dimensional information acquisition unit S2, the position of the bucket 113 output from the bucket position / attitude calculation unit 10323, and the correction information acquired from the coordinate system correction information 1 storage unit 10324. The work surface height can be calculated, for example, by calculating the average height of the three-dimensional shape of the ground vertically downward from the position of the bucket 113 from the three-dimensional information.

[0036] <Example configuration of remote control system 300> Figure 6 shows an example configuration of a remote control system (remote control room) 300. The remote control system 300 includes a communication device 306, an operation control device 305, and a display 304.

[0037] The video data with the aforementioned video compression applied and the aforementioned encoded support information are transmitted wirelessly via a wireless base station (not shown) by the communication device 104 in the work machine 100, received by the communication device 306 of the remote control system 300, and passed on to the operation control device 305.

[0038] As shown in Figure 6, the operation control device 305 is configured for generating and displaying superimposed images having AR images and includes, for example, an image expansion unit 3051, an image superposition unit 3052, a support information decoding unit 3053, a coordinate system integration processing unit 3054, a support figure generation unit 3055, and a coordinate system correction information 2 storage unit 3056. In addition, the operation control device 305 also has the function of transmitting control signals from the operation control unit 302 via the communication device 306 to control various operations of the work machine 100, but the details of this are omitted. The operation control device 305 can be realized by a general-purpose computer equipped with a CPU or GPU (processor), various types of memory, etc., and a computer program for executing the various operations shown below. In other words, the image expansion unit 3051, the image superposition unit 3052, the support information decoding unit 3053, the coordinate system integration processing unit 3054, and the support figure generation unit 3055 can be realized by deploying the corresponding computer program on the internal memory of the processor.

[0039] The video decompression unit 3051 decompresses (decodes) the video data that has been compressed (encoded) as described above and received by the communication device 306.

[0040] The support information decoding unit 3053 decodes the aforementioned encoded support information received by the communication device 306.

[0041] The coordinate system correction information 2 storage unit 3056 stores correction information for integrating the coordinate systems of the support information decoded by the support information decoding unit 3053 and the video data expanded by the video expansion unit 3051. This correction information is information for converting data described in the main unit coordinate system into data in the camera coordinate system. Furthermore, this correction information can be configured, for example, as a 6-dimensional vector indicating the installation position and orientation of the surrounding video acquisition unit S1 in the main unit coordinate system (a coordinate system that uses the pivot axis of the work machine 100 and the bottom surface of the main unit 102 as the reference point, and rotates in accordance with the rotation of the main unit 102 relative to the crawler 101). Note that this correction information may be generated by calibration during on-site shipment or inspection of the work machine 100, or it may be generated by performing an automatic calibration process when the work machine 100 is started.

[0042] The coordinate system integration processing unit 3054 converts the support information (position and orientation of the bucket 113, height of the work surface, position of the vessel 200 of the dump truck, etc., and rotation angle of the main body 102 relative to the crawler 101) into relative coordinate and orientation information based on the installation coordinates of the surrounding image acquisition unit S1 (camera coordinate system), based on the support information output by the support information decoding unit 3053 and the correction information output by the coordinate system correction information 2 storage unit 3056.

[0043] The support shape generation unit 3055 generates AR image configurations (images of various support shapes) to show the position and orientation of the bucket 113, the height of the work surface, the position of the vessel 200 of the dump truck or the like, and the orientation of the crawler 101, based on the support information converted by the coordinate system integration processing unit 3054. The AR image is an image that is superimposed on actual video footage of the work machine 100 or the work target (ground, etc.) to support the work performed by the work machine 100.

[0044] The video overlay unit 3052 overlays the AR image generated by the support figure generation unit 3055 onto the video obtained by expanding the video in the video expansion unit 3051, thereby generating an overlaid video. This overlaid video is output to the display 304.

[0045] <Details of video compression and support information generation processing> Figure 7 is a flowchart illustrating the video compression processing by the sensor information processing unit 103 and the support information generation processing by the support information generation unit 1032. In the following explanation, the main operator of each step is the respective processing unit (video compression unit 1031, bucket position / attitude calculation unit 10323, etc.), but since these are implemented by the processor, the main operator may also be the processor.

[0046] (i) Step S11 The video compression unit 1031 compresses the video acquired by the ambient video acquisition unit S1 using a predetermined method (e.g., MPEG, H.264, etc.) to generate compressed video data, which is then output to the communication device 104. The communication device 104 transmits the compressed video data to the remote control system 300.

[0047] (ii) Step S12 The bucket position and attitude calculation unit 10323 calculates the position and attitude of the bucket 113 based on the attitude information acquired from the attitude information acquisition unit S3 and the correction information acquired from the coordinate system correction information 1 storage unit 10324. Since this attitude information is based on the attitude information acquisition unit S3 (sensor coordinate system information), it is converted into information of the main body (vehicle body) coordinate system by the coordinate system correction information, and the position and attitude of the bucket 113 are calculated.

[0048] (iii) Step S13 The dump position calculation unit 10321 calculates the position of the vessel 200 of a dump truck or the like based on the three-dimensional information acquired from the three-dimensional information acquisition unit S2 and the correction information acquired from the coordinate system correction information 1 storage unit 10324. Similar to the attitude information, the three-dimensional information is information based on the three-dimensional information acquisition unit S2 (sensor coordinate system information), so it is converted into information of the main body (vehicle body) coordinate system by the coordinate system correction information, and the position of the vessel 200 is calculated.

[0049] (iv) Step S14 The work surface height calculation unit 10322 calculates the work surface height based on the three-dimensional information acquired from the three-dimensional information acquisition unit S2, the bucket position calculated in step S12, and the correction information acquired from the coordinate system correction information 1 storage unit 10324. In other words, the work surface height can be determined by, for example, calculating the average height of the three-dimensional shape of the ground vertically downward from the tip position of the claw portion of the bucket 113 from the three-dimensional information, and converting this to the main body coordinate system using the correction information.

[0050] (v) Step S15 The support information encoding unit 1033 generates encoding support information by encoding the position and orientation of the bucket 113 calculated in step S12, the position of the vessel 200 of the dump truck or the like calculated in step S13, the working ground surface height calculated in step S14, and the rotation angle of the main body 102 relative to the crawler 101 obtained from the orientation information acquisition unit S3, and outputs it to the communication device 104. The communication device 104 transmits the encoding support information to the remote control system (remote operation room) 300.

[0051] Note that the execution order of steps S11 to S13 may be changed, and steps S11 to S12 may be executed two or more times while any of the other steps are executed once.

[0052] <Details of superimposed image generation process> Figure 8 is a flowchart illustrating the process of generating superimposed images with AR images using the operation control device 305 shown in Figure 6. In the following explanation, the main operator of each step is the respective processing unit (image decompression unit 3051, support information decoding unit 3053, etc.), but since these are implemented by the processor, the main operator may also be the processor.

[0053] (i) Step S16 The video decompression unit 3051 decompresses (decodes) compressed video data transmitted from the communication device 104 of the work machine 100 and received by the communication device 306 of the remote control system 300. The resulting video data is data that projects the external scenery as seen from the camera coordinate system onto the image plane.

[0054] (ii) Step S17 The support information decoding unit 3053 decodes the coded support information transmitted from the communication device 104 of the work machine 100 and received by the communication device 306 of the remote control system 300. The decoded support information is information in the main unit coordinate system (vehicle body coordinate system).

[0055] (iii) Step S18 The coordinate system integration processing unit 3054 converts the support information obtained by decoding in step S17 and the correction information output from the coordinate system correction information 2 storage unit 3056 into relative coordinate and orientation information based on the installation position of the surrounding image acquisition unit S1. In other words, the support information is converted from information of the main unit coordinate system to information of the camera coordinate system.

[0056] (iv) Step S19 Based on the support information transformed in step S18, the support shape generation unit 3055 generates an AR image (support shape) showing the position and orientation of the bucket 113, the height of the work surface, the position of the vessel 200 of the dump truck or the like, and the orientation of the crawler 101.

[0057] AR images can be generated by performing calculations that project line segments and shapes in three-dimensional space from camera coordinates onto the image plane. For example, in the case of the Vessel 200, since the position of the upper opening is known, an AR image is generated by detecting a rectangle representing the upper opening of the Vessel 200 and converting it into a shape. Alternatively, template image data of the parts to be displayed (Vessel, bucket, etc.) may be prepared in advance, and an AR image may be generated by appropriately transforming the template image according to the position of each part. The conversion from camera coordinates to the image plane is performed, for example, by perspective projection transformation using internal parameters of the ambient image acquisition unit S1 (focal length of the lens, effective size per pixel of the image sensor, and center coordinates of the image) stored in a memory device (not shown).

[0058] (v) Step S20 The video superposition unit 3052 superimposes the AR image generated in step S19 onto the video obtained in step S16 to generate a superimposed video, which is then output to the display 304.

[0059] <Example of superimposed image> Figures 9 to 14 are diagrams illustrating examples of superimposed images (front view of the work machine + surrounding view + AR image) in the first embodiment. Figure 9 is a diagram showing an example of an image obtained by stretching (front view of the work machine + surrounding view). Figure 10 is a diagram showing an example of a superimposed image obtained by superimposing an AR image on the stretched image obtained (Figure 9). Figure 11 is a diagram showing an example of a superimposed image (front view of the work machine + surrounding view + AR image) displayed when the bucket 113 is lifted vertically. Figure 12 is a diagram showing an example of the correspondence between auxiliary lines indicating the orientation of the crawler 101 and the vehicle body among the AR images displayed in the superimposed image. Figure 13 is a diagram showing an example of an AR image displayed in the superimposed image. Figure 14 is a diagram showing another example of an AR image displayed in the superimposed image.

[0060] The image extended by the image extension unit 3051 consists of, for example, the boom 111, arm 112, bucket 113 (front part of the work machine), vessel 200 of a dump truck or the like, and the ground to be excavated (surrounding image), as shown in Figure 9. The display 304 displays the image as shown in Figure 9, including the AR image superimposed by the image superimposition unit 3052, as shown in Figure 10.

[0061] As shown in Figure 10, the AR image includes a figure F1 showing the position and orientation of the bucket 113 during the excavation work of the work machine 100, a figure F3 showing the position corresponding to the direction of movement of the bucket 113, in other words, the position of the ground (work surface) to which the bucket 113 is moving, a figure F2 connecting the two figures, a figure F4 showing the position of the right side of the crawler 101, a figure F5 showing the position of the left side of the crawler 101, a figure F6 showing the position of the front end of the crawler 101, a figure F7 showing the position of the rear end of the crawler 101, and a figure F8 showing the positions of the four top vertices of the vessel 200 of a dump truck or the like.

[0062] Figure F1 is displayed so as to reflect the orientation and shape of the bucket 113 at the position calculated by the bucket position and orientation calculation unit 10323. This makes it possible to clearly see the state of the bucket 113 even when there is video transmission delay and video distortion due to communication packet loss during communication, by using an AR image drawn based on support information that is less cumbersome (lighter) than video and less affected by packet loss due to transmission delay.

[0063] For example, Figure 11 shows an example of a superimposed image displayed on the display 304 when the bucket 113 is lifted vertically by operating the control rod 303 of the remote control system 300. The video data acquired by the ambient video acquisition unit S1 is displayed on the display 304 with a delay relative to the state of the work machine 100 due to various delays caused by compression processing in the video compression unit 1031, decompression processing in the video decompression unit 3051, and transmission from the communication device 104 to the communication device 306. On the other hand, the support information generated by the support information generation unit 1032 has a smaller data capacity than the video data, and the delay required for encoding and decoding processing is small. Furthermore, by transmitting the support information in different packets from the video data through network slicing or QoS (Quality of Service) control, the delay required for transmission of the support information from the communication device 104 to the communication device 306 is also less than the delay required for the video data. As a result, the figure F1 drawn based on the support information can indicate the position and orientation of the bucket 113 of the work machine 100 with lower latency compared to the video data. As a result, as shown in Figure 11, when performing the operation to lift the bucket 113 vertically, it becomes possible to make the figure F1 represent the movement of the bucket 113 ahead of the video data of the bucket 113. Furthermore, even if part or all of the video is distorted due to communication packet loss, the operation of the bucket 113 can be visually confirmed by superimposing the AR image on the operation control device 305 of the remote control system 300.

[0064] Figure F3 is displayed to reflect the position of the work surface. The position of the work surface is determined by the work surface height calculation unit 10322, which calculates the average height of the three-dimensional shape of the ground vertically downward from the tip position of the bucket 113 using the three-dimensional information described above. By displaying Figure F3, even if there is an effect of video transmission delay and video distortion due to communication packet loss during communication, it becomes possible to clearly recognize the positional relationship between the bucket 113 and the work surface using an AR image drawn based on support information that is less cumbersome (lighter) than video data and less affected by packet loss due to transmission delay. Furthermore, by displaying Figure F3 on both sides of the work surface (the excavation area by the bucket 113), the operator of the remote control system 300 can clearly recognize the positional relationship between the bucket 113 and the work surface while also clearly observing the work surface on the display 304.

[0065] Furthermore, shape F2 is displayed as a shape connecting shapes F1 and F3. When bucket 113 excavates horizontal ground, shape F1 extends along a roughly horizontal direction from the tip of bucket 113 to F3, whereas shape F2 intersects with shapes F1 and F3 and extends along a roughly vertical direction. Displaying shape F2 makes it easier for the operator to recognize the distance between bucket 113 and the ground.

[0066] Figures F4 to F7 display auxiliary lines indicating the orientation of the left and right crawlers and the front and rear ends of the crawlers, based on the turning angle of the main body 102 relative to the crawler 101 and the dimensions of the vehicle body acquired by the attitude information acquisition unit S3.

[0067] In Figure 12, Figure F4 shows the position of the right side of the crawler 101, Figure F5 shows the position of the left side of the crawler 101, Figure F6 shows the position of the front end of the crawler 101, and Figure F7 shows the position of the rear end of the crawler 101. This makes it possible to clearly visualize the rotation angle of the main body 102 relative to the crawler 101 by using an AR image drawn based on support information that is less cumbersome (lighter) than video data and less affected by packet loss due to transmission delay, even when there is an effect of video transmission delay and video distortion due to communication packet loss.

[0068] Figure F8 displays the positions of the four top vertices of the vessel 200 of a dump truck, etc., based on the position of the vessel 200 of the dump truck, etc., calculated by the dump position calculation unit 10321. This makes it possible to clearly visualize the position of the vessel 200 of the dump truck, etc., using an AR image drawn based on lightweight support information that is lighter than video data and less affected by packet loss due to transmission delay, even when there is an effect of video transmission delay and video distortion due to communication packet loss.

[0069] Figure 13 shows, as an example, the superimposed image displayed on the display 304 when the main body 102 is rotated relative to the crawler 101 by operating the control pole 303 of the remote control system 300. The image acquired by the surrounding image acquisition unit S1 is displayed on the display 304 with a delay relative to the state of the work machine 100 due to various delays caused by the compression process in the image compression unit 1031, transmission from the communication device 104 to the communication device 306, and decompression process in the image decompression unit 3051. On the other hand, the support information generated by the support information generation unit 1032 has a smaller data capacity and less delay in encoding and decoding compared to the image data. Furthermore, by transmitting the support information in different packets from the image data through network slicing or QoS control, the delay associated with the transmission of the support information from the communication device 104 to the communication device 306 is also less than the delay associated with the image data. Therefore, the figure F8 drawn based on the support information can indicate the position of the vessel 200 of a dump truck or the like with lower latency compared to the image data. Furthermore, the figures F4 to F7, drawn based on the support information, can show the rotational movement of the work machine 100 with lower latency compared to the video data. As a result, as shown in Figure 13, when operating to rotate the main body 102 relative to the crawler 101, figure F8 can display the position of the vessel 200 of a dump truck or the like, and figures F4 to F7 can display the rotational movement of the main body 102 relative to the crawler 101 in advance. In addition, even if part or all of the video is distorted due to communication packet loss, the position of the vessel 200 of a dump truck or the like, and the rotational movement of the main body 102 relative to the crawler 101 can be visually confirmed by superimposing the AR image on the operation control device 305 of the remote control system 300. Note that the display position of the vessel 200 and the display position of figure F8 in Figure 13 are misaligned because the coordinate system changes due to the rotational movement of the crawler 101, causing the superimposed positions of the surrounding image (image of the vessel 200) and the AR image (figure F8) to be misaligned.

[0070] FIG. 14 shows a display example in which the turning operation of the main body 102 with respect to the crawler 101 is pre-drawn with an AR image. In FIG. 14, the graphic F10 is a graphic display generated based on the support information, and is a graphic display showing the relationship between the turning angle of the main body 102 with respect to the crawler 101 of the working machine 100, the surrounding vehicles, and the terrain. The graphic F10 includes a graphic F11 showing the front of the main body 102, a graphic F12 showing the left 90° direction of the main body 102, a graphic F13 showing the right 90° direction of the main body 102, a graphic F14 showing the direction of the crawler 101, a graphic F15 showing the direction of the position of the working ground surface, and a graphic F16 showing the direction of the position of the vessel 200 such as a dump truck. According to this superimposed video, it is possible to show the turning operation of the working machine 100 with low latency as compared with the video. Further, even when a part or the whole of the video is disturbed due to communication packet loss, it is possible to show the turning operation of the working machine 100.

[0071] <Display during AR pre-drawing> FIG. 15 is a diagram showing an example of the AR pre-drawing display F17 indicating that the AR graphic is pre-superimposed on the video. According to the AR pre-drawing display F17, when there is no dump truck or the like arranged within the detection range of the working machine 100 or when the working machine 100 is stationary, it is shown to the remote operator of the working machine that the AR graphic is pre-drawn in the superimposed video, and it is possible to prevent misoperations at the start or resumption of the operation. That is, when the working machine 100 is stationary, there is no deviation between the AR graphic and the video, but when the operation is started, the operator can recognize that the AR graphic is displayed in advance of the video, so that misoperations can be prevented. Further, when each AR graphic display is deviated from the working video, it is possible to make the operator recognize that the deviation is due to video delay rather than calibration deviation.

[0072] In the examples shown in Figures 10, 11, and 13-15, the AR shapes F1-F8 are displayed as if they are all superimposed. However, it goes without saying that depending on the position and orientation of the bucket 113, the position of the work surface, the position of the vessel 200 of the dump truck, etc., and the rotation angle of the main body 102 relative to the crawler 101, it is not necessary for some or all of the shapes F1-F8 to be displayed in the superimposed image.

[0073] <Communication delays for video and support information: Explanation using a timing chart> Referring to the timing charts in Figures 16 to 18, the communication delays added to the video acquired by the ambient video acquisition unit S1 and the support information generated by the support information generation unit 1032 will be explained.

[0074] (i) Figure 16 shows the communication delay that occurs when the configuration of the first embodiment is not adopted, and an overlaid image is generated by superimposing an AR image within the work machine 100 and transmitted to the remote control system 300, or when network slicing and QoS control are not performed, and the video acquired by the ambient video acquisition unit S1 and the support information generated by the support information generation unit 1032 are transmitted to the remote control system 300 in the same communication packet. In this case, since the video data and support information are subjected to the same communication delay, each piece of information transmitted from the communication device 104 is received by the communication device 306 with the communication delay shown in the figure.

[0075] (ii) Figure 17 shows the communication delays added to the video acquired by the ambient video acquisition unit S1 and the support information generated by the support information generation unit 1032 when the configuration of the first embodiment is adopted. As shown in Figure 17, the video data and support information are transmitted in parallel (for example, via different communication systems) from the communication device 104 to the communication device 306. At the communication device 306, the video data is subjected to a communication delay of the amount indicated by communication delay 1, and the support information is subjected to a communication delay of the amount indicated by communication delay 2, which is different from communication delay 1. Due to network slicing and QoS control, communication delay 2 is smaller than communication delay 1, so the communication delay added to the support information received at the communication device 306 is smaller than the communication delay added to the video data.

[0076] As a result, as shown in the timing chart of Figure 18, according to the configuration of the first embodiment, it is possible to generate an AR image based on the support information 2 transmitted from the communication device 104 to the communication device 306 after the video 1 shown in Figure 18, and to generate a superimposed video with the AR image superimposed, thereby preventing a decrease in work efficiency due to video transmission delay.

[0077] In this explanation, we have only considered the communication delay from communication device 104 to communication device 306. However, even if there is a difference between the processing delay of video compression / decompression and the processing delay of support information encoding / decoding, the configuration of the first embodiment makes it possible to prevent a decrease in work efficiency due to delays.

[0078] Furthermore, although Figure 18 shows that the timing and frequency of transmission of video data and support information from the communication device 104 were the same, it goes without saying that these do not need to be the same. Also, although Figure 18 shows that the delay times added to support information 1 and support information 2 are equal, these delay times may vary depending on the operating status of the communication equipment between communication device 104 and communication device 306, and even if such variations occur, the effects of the first embodiment will not be impaired.

[0079] As described above, according to the first embodiment, based on the support information including the three-dimensional information reduced in the support information generation unit 1032 of the work machine 100 and the video acquired by the surrounding video acquisition unit S1, the operation control device 305 of the remote control system 300 outputs a superimposed video with an AR image superimposed on it to the display 304. This makes it possible to visually confirm the bucket 113, the vessel 200 of the dump truck, the work surface, the turning angle of the main body 102 relative to the crawler 101, etc., without being hindered by video delay or video distortion due to communication packet loss.

[0080] (2) Second embodiment (2-1) Basic form Referring to Figures 19 to 25, the second embodiment (basic form) of the work machine control system 1 will be described. Note that the overall configuration of the work machine control system 1 in the second embodiment is the same as in the first embodiment, so redundant explanations will be omitted. In the second embodiment, the coordinate transformation processing added to the support information in the coordinate system integration processing unit 3054 differs from that of the first embodiment.

[0081] <Introduction of coordinate transformation that takes rotational motion into account> In the first embodiment, there is a time difference between the time when the surrounding image acquisition unit S1 of the work machine 100 acquires the image corresponding to the image data received by the communication device 306 of the remote control system 300, and the time when the three-dimensional information acquisition unit S2 and the posture information acquisition unit S3 of the work machine 100 acquire the information that forms the basis of the support information received by the communication device 306 of the remote control system 300. For example, in the situation shown in the timing chart of Figure 18, superimposed images are generated using image 1 and support information 2, but there is a difference in the timing at which image 1 and support information 2 are transmitted from the communication device 104. Therefore, when the work machine 100 operates so that the main body 102 rotates to the left relative to the crawler 101, the posture of the work machine 100 when image 1 is acquired and the posture of the work machine 100 when support information 2 is acquired are as shown in Figure 19. Here, coordinate system C1 is the camera coordinate system (coordinate system representing the camera posture) at the time when image 1 and support information 1 are acquired in the work machine 100. Coordinate system C2 is the camera coordinate system (a coordinate system representing the camera orientation) at the time when video 2 and support information 2 were acquired by the work machine 100.

[0082] In the first embodiment, when the coordinate system integration processing unit 3054 performs coordinate transformation on the support information 2, coordinate system C2 is used as the target coordinate system. As a result, a superimposed image with the AR image superimposed, as shown in Figure 13, is displayed on the display 304. However, with this transformation method, when the work machine 100 turns, the display position of the AR figure may become misaligned with the surrounding terrain and vehicle images, potentially increasing the burden on the operator.

[0083] Therefore, in the second embodiment (basic form and modified form), when the coordinate system integration processing unit 3054 performs coordinate transformation of support information 2, it performs a transformation that takes into account the difference between support information 1 and support information 2, and uses coordinate system C1 as the target coordinate system. This achieves both the elimination of the discrepancy between the video and the AR image and the display of video that allows the operation of the work machine 100 without being hindered by delays in video data or video distortion due to communication packet loss.

[0084] <Example of internal configuration of the operation control device 305> Figure 20 shows an example of the internal configuration of the operation control device 305 according to the second embodiment (basic form). In addition to the configuration of the first embodiment (Figure 6), the operation control device 305 is equipped with a delay time estimation unit 3057.

[0085] The delay time estimation unit 3057 estimates the delay time 1 added to the video data and the delay time 2 added to the support information based on the data received by the communication device 306. For example, the communication device 306 has a function to monitor the communication status. In such a case, the delay time estimation unit 3057 estimates how much delay time is currently occurring from the communication status information detected by the communication device 306.

[0086] The coordinate system integration processing unit 3054 converts the support information into information in coordinate system C1 based on the delay time (estimated value) output from the delay time estimation unit 3057 and the correction information output from the coordinate system correction information 2 storage unit 3056. For example, the coordinate system integration processing unit 3054 compares the delay time of the surrounding video data (first delay time) with the delay time of the support information (second delay time) and determines the support information (second timing support information) necessary to display the working state of the work machine 100 in advance as an AR image. The coordinate system integration processing unit 3054 also calculates the difference between the first delay time and the second delay time and calculates the amount of conversion from coordinate system C2 to coordinate system C1 based on the difference between the rotation angle of the main body 102 relative to the crawler 101 at the support time going back by the difference in delay time and the rotation angle of the main body 102 relative to the crawler 202 of the latest support time information. The coordinate system integration processing unit 3054 then converts the support information (coordinate system C2) to the coordinate system C1 of the working machine's posture (the coordinate system from which the surrounding image at the first timing was acquired), generates coordinate-transformed support information, and from this, generates an AR image (support figure). Note that other components are omitted as they overlap with the first embodiment (Figure 6).

[0087] <Superimposed image generation processing> Figure 21 is a flowchart illustrating the superimposed image generation process with AR images by the operation control device 305 (see Figure 20) of the second embodiment (basic form). Step S22 is executed after steps S16 and S17, which are the same as in the first embodiment (Figure 8). Steps S16, S17, S19, and S20 are the same as in the first embodiment (Figure 8), so their explanation is omitted.

[0088] In step S22, the delay time estimation unit 3057 estimates the delay time added to the video data and the support information, respectively, based on the data received by the communication device 306.

[0089] In step S17, the coordinate system integration processing unit 3054 converts the support information into information in coordinate system C1 based on the delay time output from the delay time estimation unit 3057 and the correction information output from the coordinate system correction information 2 storage unit 3056.

[0090] (2-2) Variations In the modified form, instead of detecting the delay time from the communication status information detected by the communication device 306 as in the basic form described above, the transmitting side (work machine 100 side) detects the delay time based on information embedded in the video data and processes it to reflect in the coordinate transformation.

[0091] <Example of internal configuration of sensor information processing unit 103> (i) Figure 22 is a diagram showing an example of the internal configuration 1 of the sensor information processing unit 103 in the second embodiment (modified version).

[0092] The sensor information processing unit 103 includes an information embedding unit 1034 in addition to the configuration of the first embodiment (Figure 4). The information embedding unit 1034 embeds the rotation angle of the main body 102 relative to the crawler 101, acquired by the attitude information acquisition unit S3, into some of the pixels of the image acquired by the surrounding image acquisition unit S1. Other components are omitted as they overlap with those of the first embodiment (Figure 4).

[0093] (ii) Figure 23 is a diagram showing an example of the internal configuration 2 of the sensor information processing unit 103 in the second embodiment (modified version). The sensor information processing unit 103 includes a timestamp issuing unit 1035 in addition to the configuration shown in Figure 22. The information embedding unit 1034 embeds timestamp information into some pixels of the video acquired by the ambient video acquisition unit S1. The support information encoding unit 1033 encodes the timestamp output by the timestamp issuing unit 1035 in addition to the data encoded in the first embodiment. Other components are omitted as they overlap with the first embodiment (Figure 4).

[0094] <How to embed information> Figure 24 shows an example of a method for embedding information into some pixels of the video acquired by the ambient video acquisition unit S1 in the information embedding unit 1034.

[0095] The information embedding unit 1034 replaces the pixel values ​​of some pixels I11 in the video I1 acquired by the ambient video acquisition unit S1 that do not interfere with the operator of the remote control system 300 with the rotation angle of the main unit 102 relative to the crawler 101 acquired by the attitude information acquisition unit S3, or with timestamp information output by the timestamp issuing unit 1035. It is sufficient to replace the pixel values ​​for a number of pixels that takes into account the video input to the interlaced video compression unit 1031 and the macroblock size during the compression process in the video compression unit 1031. For example, when compressing interlaced video using H.264 with a macroblock size of 16x16, it is sufficient to replace the pixel values ​​of 32 vertical pixels and 16 horizontal pixels with this information.

[0096] <Example of internal configuration of the operation control device 305> Figure 25 shows an example of the internal configuration of the operation control device 305, which corresponds to the sensor information processing unit 103 in Figures 22 and 23.

[0097] In addition to the configuration of the first embodiment (Figure 6), the operation control device 305 includes an information reading unit 3058.

[0098] The information reading unit 3058 reads, from some of the pixel values ​​of the video data expanded by the video expansion unit 3051, the rotation angle of the main body 102 relative to the crawler 101 acquired by the attitude information acquisition unit S3, or the timestamp information output by the timestamp issuing unit 1035, which is embedded by the information embedding unit 1034 of the sensor information processing unit 103.

[0099] The coordinate system integration processing unit 3054 converts the support information (coordinate system C2) to information in coordinate system C1 based on the information read by the information reading unit 3058 and the support information decoded by the support information decoding unit 3053. If the information read by the information reading unit 3058 is the rotation angle of the main body 102 relative to the crawler 101, the coordinate system integration processing unit 3054 calculates the amount of conversion to coordinate system C1 based on the difference between the rotation angle of the main body 102 relative to the crawler 101 in the decoded support information and the rotation angle read. On the other hand, if the information read by the information reading unit 3058 is timestamp information output by the timestamp issuing unit 1035, the coordinate system integration processing unit 3054 compares the timestamp information of the decoded support information with the timestamp information read, and calculates the amount of conversion from coordinate system C2 to coordinate system C1 based on the difference between the rotation angle of the main unit 102 relative to the crawler 101 of the support information going back by the time difference of the timestamp information and the rotation angle of the main unit 102 relative to the crawler 101 of the latest support information. Other components are omitted as they overlap with the first embodiment (Figure 6).

[0100] <Video compression processing and support information generation processing> Figure 26 is a flowchart illustrating the video compression processing and support information generation processing by the sensor information processing unit 103 (see Figure 22 or Figure 23) of the second embodiment (modified version). In Figure 26, the processing from steps S11 to S15 is the same as the processing of the first embodiment (see Figure 7), so the explanation of those steps is omitted.

[0101] In step S21, the information embedding unit 1034 replaces some of the pixel values ​​of the video acquired by the ambient video acquisition unit S1 with the rotation angle of the main unit 102 relative to the crawler 101 acquired by the attitude information acquisition unit S3, or with timestamp information output by the timestamp issuing unit 1035.

[0102] <Superimposed image generation processing> Figure 27 is a flowchart illustrating the superimposed image generation process having an AR image by the operation control device 305 (see Figure 25) of the second embodiment (modified version). The process of step S23 is executed between the processes of step S16 and step S17, which are the same as in the first embodiment (Figure 8).

[0103] In step S23, the information reading unit 3058 reads information embedded in the information embedding unit 1034 of the sensor information processing unit 103, based on some of the pixel values ​​of the video data expanded in step S16, which is the rotation angle of the main unit 102 relative to the crawler 101 acquired by the attitude information acquisition unit S3, or the timestamp information output by the timestamp issuing unit 1035.

[0104] In step S18, the coordinate system integration processing unit 3054 converts the support information (coordinate system C2) into information in coordinate system C1 based on the information read in step S23, which is the rotation angle of the main body 102 relative to the crawler 101 acquired by the attitude information acquisition unit S3 or the timestamp information output by the timestamp issuing unit 1035, and the support information decoded in step S17. Note that the processing in steps S16, S17, S19, and S20 is the same as in the first embodiment (Figure 8), so the explanation is omitted.

[0105] <Example of superimposed display> Figure 28 shows examples of images (surrounding image and front view of the work machine) and AR images superimposed thereon, which are displayed by the processing of the second embodiment (basic form and modified form). More specifically, Figure 28 shows an example of a superimposed image displayed on the display 304 when the main body 102 is rotated relative to the crawler 101 by operating the operation rod 303 of the remote control system 300.

[0106] An AR image generated based on the support information converted into information in coordinate system C1 is superimposed on the video acquired by the surrounding video acquisition unit S1. This makes it possible to draw figures F3 to F8 whose display position matches the surrounding terrain and vehicle images in the video, as well as figure F1 which shows the relative position and orientation of the bucket 113 with respect to the surrounding terrain and vehicle in the video.

[0107] As described above, the second embodiment makes it possible to eliminate the discrepancy between the video and the AR image, and to display video in a way that allows the operation of the work machine 100 without being hindered by delays in video data or video distortion due to communication packet loss.

[0108] (3) Third Embodiment The third embodiment of the work machine control system 1 will be described with reference to Figures 29 to 31. Note that the overall configuration of the third embodiment of the work machine control system 1 is the same as that of the second embodiment, so redundant explanations will be omitted.

[0109] In the third embodiment, an AR image is generated using support information as information in coordinate system C2, and the surrounding video is converted into the video obtained when the video is captured in coordinate system C1. In other words, in the second embodiment, the video is not processed, and the superimposed image is generated by calculating where to superimpose the AR image on the video, but in the third embodiment, the video is distorted by oblique transformation (for example, homography transformation: the amount of transformation is calculated from the difference in orientation between C1 and C2) before the AR image is superimposed.

[0110] <Example of internal configuration of the operation control device 305 1> Figure 29 shows an example of the internal configuration 1 of the operation control device 305 according to the third embodiment. In addition to the configuration of the second embodiment (basic form) (see Figure 20), this operation control device 305 includes a video conversion unit 3060. In this example, the coordinate system integration processing unit 3054 converts the support information into information in coordinate system C2 (camera coordinate system). The video conversion unit 3060 calculates the amount of conversion from coordinate system C2 (camera coordinate system) to coordinate system C1 (camera coordinate system) based on the delay time output by the delay time estimation unit 3057 and the support information decoded by the support information decoding unit 3053, and converts the video in coordinate system C2 into video that can be obtained when filmed in coordinate system C1. Other components are the same as in the second embodiment (Figure 20), so their explanation is omitted.

[0111] <Example of internal configuration of the operation control device 305 2> Figure 30 shows an example of the internal configuration 2 of the operation control device 305 according to the third embodiment. In addition to the configuration of the second embodiment (modified version) (see Figure 25), this operation control device 305 includes a video conversion unit 3060. In this example, the coordinate system integration processing unit 3054 converts the support information into information in coordinate system C2. The video conversion unit 3060 calculates the amount of conversion from coordinate system C2 to coordinate system C1 based on the information output by the information reading unit 3058 (the rotation angle of the main body 102 relative to the crawler 101 acquired by the attitude information acquisition unit S3, or the timestamp information output by the timestamp issuance unit 1035) and the support information decoded by the support information decoding unit 3053, and converts the video in coordinate system C2 into video that can be acquired when filmed in coordinate system C1. Other components are the same as in the second embodiment (Figure 25), so their explanation is omitted.

[0112] <Example of superimposed image> Figure 31 shows an example of an image (surrounding image and image of the front of the work machine) and an AR image superimposed thereon, which are displayed by the processing of the third embodiment. Specifically, Figure 31 shows an example of a superimposed image displayed on the display 304 when the main body 102 is rotated relative to the crawler 101 by operating the operation rod 303 of the remote control system 300.

[0113] The surrounding image acquisition unit S1 acquires an image (distorted image) that has been converted to the state obtained when shooting in coordinate system C1. An AR image generated based on support information converted to information in coordinate system C2 is then superimposed on this image. This makes it possible to draw figures F3 to F8 whose display position matches the surrounding terrain and vehicle images in the video, as well as figure F1 which shows the relative position and orientation of the bucket 113 with respect to the surrounding terrain and vehicle in the video.

[0114] As described above, according to the third embodiment, it is possible to achieve both the elimination of the discrepancy between the video and the AR image, and the display of video that allows the operation of the work machine 100 without being hindered by video delay or video distortion due to communication packet loss.

[0115] (4) Fourth Embodiment Referring to Figures 32 to 34, the fourth embodiment of the work machine control system 1 will be described. Note that the overall configuration of the fourth embodiment of the work machine control system 1 is the same as that of the first embodiment, so redundant explanations will be omitted.

[0116] The fourth embodiment differs from the first embodiment in that it superimposes a figure indicating the position of the last (most recent) excavated work surface or the next work surface to be excavated onto the superimposed image displayed on the display 304. By displaying the position of the work ground, after discharging excavated soil and other materials into the vessel 200 of a dump truck or the like, it becomes possible to operate the work machine 100 without being hindered by delays in video data or distortion of the image due to loss of communication packets when rotating the main body 102 relative to the crawler 101 toward the next work surface to be excavated.

[0117] <Example of internal configuration of the operation control device 305> Figure 32 shows an example of the internal configuration of the operation control device 305 according to the fourth embodiment. In addition to the configuration of the first embodiment (see Figure 6), the operation control device 305 includes a work target ground surface memory unit 3059. The work surface memory unit 3059 stores the location of the work surface that the work machine 100 plans to excavate next (which can be specified in advance by the operator). The coordinate system integration processing unit 3054 performs a coordinate transformation on the position of the work target ground surface output from the work target ground surface storage unit 3059, along with the support information.

[0118] The support shape generation unit 3055 generates an AR image based on the position of the work target ground surface and support information output from the work target ground surface storage unit 3059, which have been converted by the coordinate system integration processing unit 3054.

[0119] Note that when displaying the last excavated work area, information about the work area to be excavated is not required. However, when displaying both the last excavated work area and the work area to be excavated simultaneously, the necessary information is read from the work area surface memory unit 3059. Other components are omitted as they overlap with the first embodiment (Figure 6).

[0120] <Example of superimposed image> Figure 33 shows an example of video (surrounding image and front view of the work machine) and an AR image superimposed thereon, which are displayed by the processing of the fourth embodiment.

[0121] In Figure 33, the AR image superimposed on the video includes, in addition to the configuration of the first embodiment (Figure 10), a figure F9 indicating the last excavated work area and / or the work area to be excavated. Figure F9 is a figure generated based on the position information of the last excavated work area and / or the position information of the work area to be excavated output from the work area surface memory unit 3059. By superimposing the last excavated work area and / or the work area to be excavated as an AR image in this way, the operator can visually confirm the target position when rotating the main body 102 relative to the crawler 101 toward the next work area to be excavated after discharging the excavated soil and sand into the vessel 200 of a dump truck or the like.

[0122] Note that the position of the work target ground surface stored in the work target ground surface memory unit 3059 may be, for example, the coordinate value of the ground where the tip of the bucket 113 touches the ground when the work machine 100 last performed excavation, or as shown in FIG. 34, on the GUI (Graphical User Interface) displaying the vehicle body from a top view perspective, the positions of the ground to be excavated for a predetermined number of times may be specified and stored in advance by indicating them with markers P1 to P4.

[0123] As described above, according to the fourth embodiment, when the main body 102 is swung with respect to the crawler 101 toward the work ground surface of the next excavation plan, the work machine 100 can be operated without being hindered by the delay of video data or the disturbance of the video due to communication packet loss.

[0124] (5) Variation In the first to fourth embodiments, the case where the work machine 100 is a so-called backhoe has been described. However, even when the work machine 100 is a loading shovel with the bucket facing forward, the same configuration and processing as in the first to fourth embodiments can be applied.

[0125] <AR Image Display State Switching Function> FIG. 35 is a conceptual diagram for explaining the switching of the display state of the AR image in the work machine control system 1 of each of the above embodiments.

[0126] The work machine 100 moves to the excavation location (S31), waits (S32), and then drives the bucket 113 to perform excavation (S33). When the excavation is executed, the work machine 100 turns (goes) (S34) and discharges soil to the hopper 200 (S35). When the soil discharge is completed, the bucket 113 turns back (returns) to the excavation location again (S36), and the excavation (S33) continues.

[0127] During the excavation phase (S33), the system can be configured to display a figure F1 representing the bucket 113, a figure F3 representing the position of the work surface, and a figure F2 connecting figures F1 and F3. Furthermore, during rotation (S34 and S36), the system can be configured to display figures F4 to F7 representing the direction of the crawler 101, a figure F8 representing the position of the vessel 200 such as a dump truck, and a figure F9 representing the next planned work surface for excavation. These display settings can be configured for each operator (the user remotely controlling the loading shovel from the control room). For example, the settings for switching display states can be stored in an unillustrated memory device, with each operator having their own set values ​​(settings for what to display at each stage), and these settings can be read from the memory device during operation. In this way, customized and easy-to-use AR image display switching can be achieved for each user.

[0128] Furthermore, since the brightness and contrast of captured images change depending on the weather, time of day, and shooting direction (whether it's in direct sunlight or backlighting, etc.), it is also effective to improve visibility by adjusting the brightness and contrast of the superimposed AR image (lowering the brightness for dark images such as those at night, and increasing the brightness for bright images such as those taken during the day to make them stand out). These brightness settings can also be stored for each user and recalled during operation.

[0129] (6) Summary (i) In the embodiments of this disclosure, the operation control device 305 generates an AR image (support figure) showing the posture of the work machine 100 and the position of the work object from the support information received from the work machine 100, and displays the AR image generated from the support information (support information 2: see Figure 18) acquired from the work machine 100 at a second timing after the first timing, superimposed on the surrounding video (video 1: see Figure 18) acquired from the work machine 100, and the AR image generated from the support information (support information 2: see Figure 18) acquired from the work machine 100 at a second timing after the first timing, on the display device. As a result, even if there is a delay in the surrounding video of the work machine 100, the remote operator of the work machine 100 can know the actual work situation from the AR image generated from low-capacity (light) support information, thereby preventing a decrease in work efficiency. Here, the operation control device 305 converts the support information represented in the vehicle body coordinate system of the work machine 100 to the camera coordinate system, and generates an AR image (support figure) from the support information after the coordinate conversion. This allows AR images to be displayed in the same coordinate system as the surrounding video, enabling the display of AR images corresponding to the actual work movements of the work machine 100 in advance of the delayed surrounding video.

[0130] (ii) The operation control device 305 generates an AR image that includes at least one of the following: a figure showing the position and orientation of the work machine 100 (see Figure 10, etc.), a figure showing the terrain position of the work target of the work machine 100 (see Figure 33), a figure showing the position of the work target vehicle (e.g., the vessel of a dump truck) of the work machine 100 (see Figure 10, etc.), or a figure showing the orientation of the crawler body of the work machine 100 (see Figure 10, etc.). By displaying the minimum necessary AR image in this way, operation support can be provided without confusing the remote operator (displaying too much information may actually cause confusion).

[0131] (iii) The operation control device 305 displays information indicating that it is displaying an AR image based on support information (support information 2) acquired later than the work video (video 1) superimposed on the work video (see Figure 15). This prevents erroneous operation by the remote operator at the start or restart of the operation.

[0132] (iv) In the second embodiment, the operation control device 305 changes the superimposed position and shape of the AR image (support figure) based on the angle difference between the first angle formed by the crawler body and the slewing body of the work machine 100 at the first timing and the second angle formed by the crawler body and the slewing body at the second timing (see Figure 28).

[0133] When calculating the angle difference mentioned above, the delay time of the ambient video (first delay time) and the delay time of the support information (second delay time) can be used. In other words, the operation control device 305 compares the first delay time and the second delay time to determine the support information for the second timing (support information 2: see Figure 18) to be superimposed on the ambient video at the first timing (video 1: see Figure 18), converts the support information for the second timing into the coordinate system of the posture of the work machine 100 at the first timing, and generates an AR image from the coordinate-transformed support information for the second timing.

[0134] Another example of calculating the angle difference is to use angle information (first angle information) embedded in the surrounding video. In this case, the driving control device 305 calculates the angle difference from the first angle information acquired (separated) from the surrounding video data and the second angle information acquired from the support information (support information 2) at the second timing.

[0135] As yet another example of calculating the above angle difference, a timestamp embedded in the work overload (first time information corresponding to the first timing) can also be used. In this case, the operation control device 305 calculates the above angle difference from the angles of the crawler and turning body (main body) in the support information obtained by going back only by the difference between the timestamp (first time information) obtained from the surrounding video data and the second time information obtained from the support information, and the angles of the crawler and turning body in the latest support information (support information for the second timing).

[0136] (v) In the third embodiment, the operation control device 305 deforms the surrounding image based on the angle difference between the first angle formed by the crawler body and the slewing body (main body) of the work machine 100 at the first timing and the second angle formed by the crawler body and the slewing body (main body) at the second timing (see Figure 33). In the third embodiment, the same method for calculating the angle difference as in the second embodiment can be adopted.

[0137] (vi) In the fourth embodiment, the operation control device 305 superimposes an AR image (support figure) representing the planned position of the work target (planned excavation position) by the work machine 100 onto the surrounding image based on the information of the work target's position (see Figure 33). This allows the remote operator to easily recognize the next position to be worked on by the work machine 100. The position of the last work target may also be displayed in the AR image.

[0138] (vii) In a modified example, the operation control device 305 switches the display mode of the support figures (AR images) (for example, displaying only some of the necessary AR images) according to the working status of the work machine 100 (movement, standby, excavation, rotation, earth discharge, etc.).

[0139] (viii) The functions of the embodiments of the present disclosure can also be realized by software program code. In this case, a storage medium on which the program code is recorded is provided to a system or device, and the computer (or CPU or MPU) of the system or device reads the program code stored on the storage medium. In this case, the program code read from the storage medium itself realizes the functions of the embodiments described above, and the program code itself and the storage medium on which it is stored constitute the present disclosure. Examples of storage media for supplying such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, etc.

[0140] Furthermore, based on the instructions in the program code, the operating system (OS) running on the computer may perform some or all of the actual processing, thereby realizing the functions of the embodiment described above. In addition, after the program code read from the storage medium is written to the computer's memory, the computer's CPU may perform some or all of the actual processing based on the instructions in the program code, thereby realizing the functions of the embodiment described above.

[0141] Furthermore, the program code for the software that realizes the functions of the embodiment may be distributed via a network and stored in a storage means such as a hard disk or memory of the system or device, or in a storage medium such as a CD-RW or CD-R, so that when in use the system or device's computer (or CPU or MPU) reads and executes the program code stored in the storage means or storage medium.

[0142] Furthermore, the technology of this disclosure is not limited to the embodiments described above, and includes various modifications. For example, the embodiments described above are described in detail to make the technology of this disclosure easier to understand, and are not necessarily limited to those having all the configurations described. Also, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is possible to add, delete, or replace parts of the configuration of each embodiment with other configurations. Moreover, some or all of the above configurations, functions, processing units, processing means, etc., may be implemented in hardware, for example, by designing them as integrated circuits. [Explanation of Symbols]

[0143] 1 Work machine control system, 100 Work machine (construction machine), 101 Crawler, 102 Main body, 103 Sensor information processing unit, 104 Communication device, 107 Antenna, 111 Boom, 112 Arm, 113 Bucket, 200 Vessel, 300 Remote control system, 301 Driver's seat, 302 Operation control unit, 303 Input device (operating rod), 304 Display, 305 Driving control device, 306 Communication device, 1031 Video compression unit, 1032 Support information generation unit, 1033 Support information encoding unit, 1034 Information embedding unit, 1035 Timestamp issuance unit, 10321 Dump position calculation unit, 10322 Work ground surface height calculation unit, 10323 Bucket position / attitude calculation unit, 10324 Coordinate system correction information 1 storage unit, 3051 Video decompression unit, 3052 3053 Image overlay unit, 3054 Support information decoding unit, 3055 Coordinate system integration processing unit, 3056 Support figure generation unit, 3056 Coordinate system correction information 2 storage unit, 3057 Delay time estimation unit, 3058 Information reading unit, 3059 Work target ground surface storage unit, 3060 Image conversion unit, L0, L1, L2 Ground, L4 Wall surface, Lg Corresponding ground, OR Operation room, S1 Surrounding image acquisition unit, S2 Three-dimensional information acquisition unit, S3 Pose information acquisition unit, F1~F17 Figure, C1, C2 Coordinate system, I1 Surrounding image, P1~P4 Marker

Claims

1. A remote control system for a work machine, comprising a display device that receives and displays various information from a work machine via a communication device, and a control device that controls the display device, for remotely operating the work machine, The communication device receives an image of the surroundings of the work machine captured by the work machine, and support information including posture information of the work machine and three-dimensional information of the surroundings of the work machine acquired by the work machine. The control device is A process for generating a support figure indicating the posture of the work machine and the position of the work object based on the support information received by the communication device, A process of superimposing the surrounding image captured at a first timing in the work machine and received by the communication device, and the support figure generated from the support information acquired at a second timing later than the first timing in the work machine and received by the communication device, and displaying them on the display device, A remote control system that performs this task.

2. In claim 1, The control device is a remote control system that converts the support information, which is represented in the vehicle body coordinate system of the work machine, into a camera coordinate system, and generates the support figure from the support information after the coordinate conversion.

3. In claim 1, The control device is a remote control system that generates the support figure, which includes at least one of the following: a figure showing the position and orientation of the work machine; a figure showing the terrain position of the work target of the work machine; a figure showing the position of the vehicle that the work target of the work machine; or a figure showing the orientation of the running body of the work machine.

4. In claim 1, The control device is a remote control system that changes the superimposed position and shape of the support figure based on the angle difference between a first angle formed by the traveling body and the rotating body of the work machine at the first timing and a second angle formed by the traveling body and the rotating body at the second timing.

5. In claim 1, The control device is a remote control system that deforms the surrounding image based on the angle difference between a first angle formed by the traveling body and the rotating body of the work machine at the first timing and a second angle formed by the traveling body and the rotating body at the second timing.

6. In claim 1, The control device is Based on the communication status information detected by the communication device, the process estimates the first delay time of the surrounding video and the second delay time of the support information. A process for determining the support information for the second timing to generate the support figure to be superimposed on the surrounding image at the first timing, based on the difference between the first delay time and the second delay time, A process to convert the support information for the second timing into the coordinate system of the posture of the work machine at the first timing, and to generate the coordinate-transformed support information for the second timing, A process for generating the support figure from the coordinate-transformed support information of the second timing, A remote control system that performs this task.

7. In claim 4, The surrounding image data includes a data portion indicating the information of the first angle, The control device is a remote control system that calculates the angle difference from the first angle information obtained from the surrounding video data and the second angle information obtained from the support information at the second timing.

8. In claim 5, The surrounding image data includes a data portion indicating the information of the first angle, The control device is a remote control system that calculates the angle difference from the first angle information obtained from the surrounding video data and the second angle information obtained from the support information at the second timing.

9. In claim 4, The surrounding video data includes a data portion indicating first time information corresponding to the first timing, The support information includes a second time information corresponding to the second timing, The control device is a remote control system that calculates the angle difference based on the difference between the first time information obtained from the surrounding video data and the second time information obtained from the support information.

10. In claim 5, The surrounding video data includes first time information corresponding to the first timing, The support information includes a second time information corresponding to the second timing, The control device is a remote control system that calculates the angle difference based on the difference between the first time information obtained from the surrounding video data and the second time information obtained from the support information.

11. In claim 1, The control device is a remote control system that generates the support figure, which includes a figure representing the planned position of the work object by the work machine, based on information about the position of the work object.

12. In claim 1, The control device is a remote control system that switches the display format of the support figure according to the working state of the work machine.

13. In claim 1, The control device is a remote control system that displays information indicating that the support figure, which was acquired later than the surrounding image, is superimposed on the surrounding image, along with the superimposed image of the support figure and the surrounding image.

14. The invention comprises a work machine and a remote control system according to claim 1 for remotely controlling the work machine, The aforementioned work machine is The imaging device for capturing the surrounding image, A posture sensor that acquires posture information of the aforementioned work machine, An object detection sensor that acquires three-dimensional information of the aforementioned work machine, A work machine communication device that communicates with the remote control system and transmits the surrounding video, the posture information and the support information including the three-dimensional information, A remotely operated work machine system equipped with [a specific feature].

15. A work information display control method that displays work information received from a work machine on a screen, The control device that controls the operation of the work machine acquires a video of the work machine's surroundings captured by the work machine, and support information including the work machine's posture information and three-dimensional information of the work machine's surroundings. The control device generates support figures from the support information that indicate the posture of the work machine and the position of the work object, The control device superimposes the surrounding image captured by the work machine at a first timing and the support figure generated from the support information acquired by the work machine at a second timing later than the first timing onto a display device, A method for controlling the display of work information, including the method described above.

Images