Image generation device and operation support system

Abstract

To provide a remote operation support technique allowing a remote operator to easily and quickly comprehend the state of a work machine based on information on the machine and environmental information.SOLUTION: An image generating device in accordance with the present invention generates an AR image representing machine information and environmental information based on the machine information representing a state of the work machine and the environmental information representing a state of peripheral environment around the work machine, and places the AR image representing the machine information at a position represented by the machine information or places the AR image representing the environmental information at a position represented by the environmental information.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a technique for generating images used to assist an operator who remotely operates a work machine. [Background technology]

[0002] Technologies for remotely controlling work machines such as hydraulic excavators have been developed. The work machine is equipped with an imaging device that captures images of the surrounding environment, and by transmitting the captured images to a remote control room, the remote control room can obtain images as seen from, for example, the driver's cab of the work machine. The remote operator uses the images to remotely control the work machine.

[0003] The following Patent Document 1 describes a technology that aims to "provide a technology that enables an operator to comfortably remotely control a work machine and suppresses a decline in work efficiency" and that "includes a target image generating unit that generates a target image showing a virtual viewpoint image of a target as seen from a virtual viewpoint outside the work machine based on detection data from a distance detection device that is mounted on a work machine having a work implement and detects the distance to an object around the work machine, a work machine image generating unit that generates a work machine image showing a virtual viewpoint image of the work machine as seen from the virtual viewpoint based on detection data from an attitude detection device that is mounted on the work machine and detects the attitude of the work machine, a synthesis unit that generates a synthetic image by superimposing the target image and the work machine image, and a display control unit that simultaneously displays an actual image captured by an imaging device mounted on the work machine and the synthetic image on a display device located outside the work machine" (see abstract). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-054464 A Summary of the Invention [Problem to be solved by the invention]

[0005] When remotely operating a work machine, machine information that indicates the state of the work machine and environmental information that indicates the state of the environment surrounding the work machine are both required. However, the information required differs depending on the work performed by the work machine, etc. Also, rather than simply displaying the machine information and environmental information at an arbitrary position on the remote operation screen, it is desirable for the remote operator to easily and quickly grasp the state of the work machine based on the machine information and environmental information. In conventional techniques such as Patent Document 1, although machine information and environmental information are displayed on the remote operation screen, sufficient consideration is not necessarily given to devising ways to enable the remote operator to easily and quickly grasp the state of the work machine.

[0006] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a remote operation support technology that enables a remote operator to easily and quickly grasp the condition of a work machine based on the machine information and environmental information. [Means for solving the problem]

[0007] The image generation device of the present invention generates an AR image showing machine information and environmental information based on machine information representing the state of a work machine and environmental information representing the state of the surrounding environment of the work machine, and places the AR image showing the machine information at a position indicated by the machine information, or places the AR image showing the environmental information at a position indicated by the environmental information. Effect of the Invention

[0008] According to the image generating device of the present invention, a remote operator can easily and quickly grasp the state of a work machine based on the machine information and environmental information. Other objects, configurations, advantages, etc. of the present invention will become apparent from the description of the following embodiments. [Brief description of the drawings]

[0009] [Figure 1] 1 is a schematic diagram showing the surrounding environment of a work machine 100 according to a first embodiment. [Diagram 2]FIG. 2 is a top view showing the imaging ranges of sensors S1 to S3. [Diagram 3] FIG. 2 is a schematic diagram showing the configuration of a remote control room 300. [Figure 4] FIG. 2 is a block diagram showing a configuration of an image generating device 103. [Diagram 5] FIG. 11 is a functional block diagram of an image generating device 103 when a free viewpoint image is not generated. [Figure 6] FIG. 3 is a functional block diagram of an operation control device 305. [Figure 7] 11 is a flowchart illustrating a procedure in which the image generating device 103 generates a free viewpoint image. [Figure 8] 11 is a flowchart illustrating a procedure in which the image generating device 103 generates a normal image. [Figure 9] 13 is an example of a dot pattern used in calibration in the coordinate system integration process performed by the coordinate system processing unit 1031. [Figure 10] FIG. 13 is a schematic diagram illustrating calibration 2. [Figure 11] FIG. 13 is a schematic diagram illustrating calibration 3. [Figure 12] This indicates the coordinate system used when generating free viewpoint video. [Figure 13] 13 shows examples of images generated by the image generating device 103 for each viewpoint. [Figure 14] 13 shows examples of images generated by the image generating device 103 for each viewpoint. [Figure 15] 13 shows examples of images generated by the image generating device 103 for each viewpoint. [Figure 16] This is a display example in which the boom 111 to the bucket 113 are colored in the image generated by the image generating device 103. [Figure 17] This is a display example in which the boom 111 to the bucket 113 are colored in the image generated by the image generating device 103. [Figure 18] This is a display example in which the boom 111 to the bucket 113 are colored in the image generated by the image generating device 103. [Figure 19] 1 is an example of an image of the surrounding environment of the work machine 100 generated by the image generating device 103. [Figure 20] 1 is an example of an image of the surrounding environment of the work machine 100 generated by the image generating device 103. [Figure 21] 1 is an example of an image of the surrounding environment of the work machine 100 generated by the image generating device 103. [Figure 22] 1 is an example of an image generated by the image generating device 103, showing the range of movement of a rotating body 102. [Diagram 23] 13 is an example of another image generated by the image generating device 103, showing the range of movement of the rotating body 102. [Figure 24] 13 is an example of an image generated by the image generating device 103 to warn against watermarking. [Diagram 25] FIG. 11 is a functional block diagram of an image generating device 103 according to a fourth embodiment. [Figure 26] FIG. 11 is a functional block diagram of an image generating device 103 when a free viewpoint image is not generated. [Figure 27] 2 is a state transition diagram of the work machine 100. FIG. [Figure 28] 13 is an example of a threshold value that serves as a trigger for switching the free viewpoint image. [Figure 29] A modified example of the image generating device 103 is shown. [Diagram 30] FIG. 3 is a functional block diagram of an operation control device 305. [Diagram 31] A modified example of the image generating device 103 is shown. [Diagram 32] FIG. 1 is a schematic diagram of the OR. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] <Embodiment 1> FIG. 1 is a schematic diagram showing the surrounding environment of a work machine 100 according to a first embodiment of the present invention. The work machine 100 in this example is a hydraulic excavator. The work machine 100 includes a traveling body 101, a revolving body 102, a cab OR, and a work machine (111 to 113 described below). The work machine 100 is connected to a remote control room 300 via a communication device 104. The work machine is composed of a boom 111, an arm 112, and a bucket 113. Sensors S1, S2, S3, and S4 are disposed on the right side, front, left side, and top of the revolving body 102, respectively. An image generating device 103 is disposed inside the cab OR. An antenna 107 communicates with the remote control room 300 by wireless communication.

[0011] The sensors S1 to S3 are capable of capturing images of the ranges described below. The sensor S4 is a sensor that detects mechanical information of the work machine 100 (for example, the attitude of the work machine 100). The sensors S1 to S3 can also be configured as a sensor that combines a LiDAR (three-dimensional measuring device) and a camera. The LiDAR is a device that measures the three-dimensional distance between the surrounding environment and the work machine 100 by emitting laser light. In the following, the sensors S1 to S3 are assumed to be configured as a combination of a camera and a LiDAR, and are configured to acquire environmental information around the work machine 100.

[0012] The work machine 100 performs excavation work in a work environment L. The work environment L is made up of flat land L0, a plateau L1, and a slope L2. The work machine 100 loads the soil and sand collected in the bucket 113 by performing the excavation work into a dump truck 200.

[0013] Fig. 2 is a top view showing the imaging ranges of sensors S1 to S3. Sensor S1 is disposed on the right side of the revolving structure 102, sensor S2 is disposed on the front side of the revolving structure 102, and sensor S3 is disposed on the left side of the revolving structure 102. Sensors S1 to S3 can capture images of arc-shaped imaging ranges SR1 to SR3, respectively, as shown in Fig. 2. These sensors can obtain images of the work machine 100 and images of the environment surrounding the work machine 100. The measurement range by the LiDAR is also the same as that in Fig. 2.

[0014] 3 is a schematic diagram showing the configuration of a remote control room 300. The remote control room 300 includes a driver's seat 301, an operation device 302, an input device 303, a display 304, an operation control device 305, and a communication device 306. The communication device 306 acquires the image generated by the image generation device 103 through communication. The operation control device 305 displays the image on the display 304. The remote operator inputs operation instructions using the input device 303 while viewing the image. The operation device 302 transmits the operation instructions to the work machine 100 via the operation control device 305 and the communication device 306. The work machine 100 operates in accordance with the operation instructions. The image generation device 103 and the remote control room 300 can be configured as an operation support system that supports the operation of the work machine 100.

[0015] Fig. 4 is a block diagram showing the configuration of the image generating device 103. The image generating device 103 is a device that integrates measurement data acquired by each sensor equipped in the work machine 100 and transmits it as an image to the remote control room 300. The image generating device 103 acquires measurement results from sensors S1 to S3 (imaging devices and three-dimensional measuring devices) and vehicle sensors. The vehicle sensors are sensors that measure the state of the work machine (e.g. internal pressure, etc.). S4 is one of the vehicle sensors. The processing of each unit equipped in the image generating device 103 will be explained below.

[0016] The machine information acquisition unit 1030A and the environment information acquisition unit 1030B acquire data (measurement data) describing the results of measurements by each sensor from each sensor. The same role as the machine information acquisition unit 1030A and the environment information acquisition unit 1030B can also be performed by a functional unit (for example, the coordinate system processing unit 1031 and the figure position and shape calculation unit 1032 in FIG. 4) that acquires measurement data from each sensor. Therefore, in the following description, for convenience of description, the machine information acquisition unit 1030A and the environment information acquisition unit 1030B may be omitted.

[0017] The coordinate system processing unit 1031 acquires measurement values ​​from each sensor and performs processing to align the coordinate systems of each sensor. A specific example of the processing to align the coordinate systems will be described later. The figure position and shape calculation unit 1032 calculates the part to be highlighted by changing, for example, the color or the line type of the image of the work machine 100 and the image of the surrounding environment of the work machine 100. The free viewpoint image generation unit 1033 generates a free viewpoint image in which the work machine 100 or its surrounding environment is viewed from a free viewpoint using the measurement values ​​of each sensor with the aligned coordinate system. The AR image generation unit 1034 generates an image of the highlighted part calculated by the figure position and shape calculation unit 1032 (an image with changed color, line type, etc.: called an AR image). The video superimposition unit 1035 synthesizes the free viewpoint image and the AR image. The video compression unit 1036 compresses the synthesized image and transmits it to the remote control room 300 via the communication device 104.

[0018] The image generating device 103 can be configured with hardware such as a circuit device that implements the operation of each functional unit, or can be configured by a computing device executing software that implements the operation of each functional unit. In the latter case, the image generating device 103 has the configuration shown in the upper right corner of Fig. 4. A CPU (Central Processing Unit) 151 executes software that implements the operation of each functional unit. ROM 152, RAM 153, and HDD (Hard Disk Storage Device) 154 store data used by the CPU 151. An operation interface 155 is an interface for inputting operation instructions to the image generating device 103. A display interface 156 is an image display interface.

[0019] In the configuration example shown in Fig. 4, the state of the work machine 100 and the state of the surrounding environment of the work machine 100 are detected by a group of sensors provided on the work machine 100. The image generating device 103 is provided on the work machine 100, and an operation assistance image used to operate the work machine 100 is generated by the work machine 100. The remote operation room 300 receives the operation assistance image as a two-dimensional image and displays it.

[0020] Fig. 5 is a functional block diagram of the image generating device 103 when a free viewpoint image is not generated. When a free viewpoint image is not generated, the image superimposing unit 1035 synthesizes the images acquired by each sensor with an AR image. The rest is the same as in Fig. 4. For convenience of description, the hardware configuration is omitted. The same applies to the following embodiments.

[0021] 6 is a functional block diagram of the operation control device 305. The operation control device 305 includes a video decompression unit 3051, an information addition / display switching unit 3052, and a display control unit 3053. The video decompression unit 3051 acquires a composite image from the image generation device 103 via the communication device 306, and decompresses the compressed composite image. The information addition / display switching unit 3052 adds information to the image displayed by the display 304 or switches the display contents according to an operation instruction from the input device 303. This point will be described later. The display control unit 3053 outputs the image displayed by the display 304.

[0022] FIG. 7 is a flowchart for explaining the procedure for the image generating device 103 to generate a free viewpoint image. The coordinate system processing unit 1031 calculates correction information for aligning the coordinate system for each sensor using a procedure described later (S11). The free viewpoint image generating unit 1033 calculates a free viewpoint video using the coordinate system correction information (S12). The figure position and shape calculation unit 1032 calculates the position, shape, etc. of the AR image using the coordinate system correction information (S13). The AR image generating unit 1034 generates an AR image to be superimposed on the free viewpoint video (S14). The video superimposition unit 1035 superimposes the free viewpoint video and the AR image (S15). The video compression unit 1036 compresses the video (S16).

[0023] Fig. 8 is a flowchart for explaining the procedure for generating a normal image by the image generating device 103. The procedure for generating a normal image is generally similar to that in Fig. 7, but the step of generating a free viewpoint video (S12) is omitted.

[0024] 9 is an example of a dot pattern used in calibration in the coordinate system integration process performed by the coordinate system processing unit 1031. The coordinate system processing unit 1031 performs processing to align the coordinate systems of the sensors (coordinate system conversion). In principle, this processing needs to be performed only once after various sensors are installed on the work machine 100. This processing includes the following four calibrations: (a) Calibration 1: correcting internal parameters and distortion of the sensors (S1 to S3); (b) Calibration 2: coordinate system conversion between the image sensor and the three-dimensional measuring device; (c) Calibration 3: coordinate system conversion between the three-dimensional measuring devices; (d) Calibration 4: coordinate system conversion between the three-dimensional measuring device and the work machine 100.

[0025] In calibration 1, the coordinate system processing unit 1031 calculates the internal parameters of the image sensor (camera matrix consisting of focal lengths fx, fy and image center coordinates cx, cy) and calculates a distortion coefficient that represents distortion caused by the lens. For example, a calibration process is performed using images of a pattern in which dots are arranged in a grid as shown in FIG. 9, captured from various directions, as input data. In the first step, the center coordinates of each dot are extracted from the input image and listed. In the second step, the internal parameters of the camera and distortion coefficients are calculated based on the list of center coordinates of the dots. By using the internal parameters and distortion coefficients obtained in the above calibration process and performing distortion removal correction on the camera image, it is possible to remove distortion of the subject image that was significantly distorted at the outer periphery of the angle of view.

[0026] FIG. 10 is a schematic diagram for explaining calibration 2. The purpose of calibration 2 is to calculate a coordinate transformation matrix between an image sensor (camera) and a three-dimensional measurement device (LiDAR). The image sensor and the three-dimensional measurement device capture a common subject (hereinafter, a disk shown in FIG. 10), and the center coordinates of the disk are used as a common feature point. N feature points [q1, q2, q N ] and N feature points in the LiDAR coordinate system [Q1, Q2, . . . Q N ], q i =HQ i (i = 1, 2, , N). Calculate H, which satisfies (i = 1, 2, , N). The coordinates Q of the feature points in the coordinate system of the LiDAR point cloud. i is a three-dimensional vector, and the coordinates of the feature point in the camera image coordinate system q i is a two-dimensional vector. q i and Q i are the same feature points expressed in each coordinate system.

[0027] The feature points may be extracted by selecting the center of the disk on the user interface using the mouse and pointer, or by recognizing the disk and determining its center of gravity.

[0028] The calculation of the transformation matrix H using the feature point list extracted in the above manner can be regarded as a Perspective n Points (PnP) problem, and algorithms for solving it efficiently are widely known.

[0029] In the above description, a disk is used as an example of the subject, but the subject is not limited to this, and calibration may be performed using the terrain around the work area, buildings, other shovels, dump trucks, etc. This eliminates the need to prepare and install special targets, and makes it possible to constantly correct deviations that accumulate over time while working.

[0030] FIG. 11 is a schematic diagram for explaining calibration 3. The purpose of calibration 3 is to calculate a coordinate transformation matrix between LiDARs. When the overlap of the fields of view between the LiDARs is large, a method similar to calibration 2 can be used. However, when the overlap of the fields of view between the LiDARs is small, it is difficult to take such measures. Therefore, by having each LiDAR continuously acquire a point cloud while rotating the rotating body 102 and registering (aligning) the acquired point clouds, the field of view of each LiDAR is effectively expanded and the overlap of the fields of view between the LiDARs is effectively increased.

[0031] First, as shown in FIG. 11, a registration process is performed for each LiDAR on the LiDAR point cloud acquired while rotating the rotating body 102, thereby effectively expanding the field of view and generating an integrated point cloud. By performing this process for all LiDARs mounted on the work machine 100, the same number of integrated point clouds 1101 to 1103 are obtained. Next, a registration process is performed between the integrated point clouds to calculate a coordinate transformation matrix between the point clouds. This coordinate transformation matrix corresponds to information representing the relative attitude and position between the LiDARs. At each step, a registration process combining the Iterative Closest Point (ICP) method and the Random Sample Consensus (RANSAC) method can be used.

[0032] The purpose of calibration 4 is to calculate a coordinate transformation matrix between the LiDAR and the work machine 100. The coordinate transformation matrix between the LiDAR and the work machine 100 can be calculated from the LiDAR displacement and the amount of change in the point cloud when the vehicle turns. Furthermore, it can be performed based on the amount of change in each LiDAR point cloud when the vehicle turns, which is calculated in the first step of calibration 3, and it is possible to reduce the amount of calculation.

[0033] Fig. 12 shows the coordinate system used when generating a free viewpoint video. The free viewpoint image generator 1033 needs to integrate the acquired camera video, 3D measurement data, and vehicle sensor measurement data. First, the coordinate system integration process result and LiDAR point cloud data are used to generate a depth image (a two-dimensional array storing depth information) as seen from the viewpoint of the LiDAR's paired camera.

[0034] The free viewpoint image generating unit 1033 first converts the LiDAR point cloud obtained as coordinate information in the LiDAR coordinate system into an expression based on the camera coordinate system. The coordinates of a certain point in the point cloud in the LiDAR coordinate system are expressed as (x L ,y L ,z L ), and the coordinates of the same point in the camera coordinate system are (x c ,y c ,z c ), these two coordinates are expressed by Equation 1. The 3 × 4 matrix in Equation 1 is the attitude transformation matrix calculated in Calibration 2.

[0035]

number

[0036] Next, the free viewpoint image generation unit 1033 converts the LiDAR point cloud from the camera coordinate system to the image coordinate system, and calculates which pixel of the image the point cloud will be projected onto when it is projected onto the camera image. c ,y c ,z cThe coordinates (pixel index) of one point of the point cloud represented as (x img ,y img ), this transformation is expressed by Equation 2. The 3 × 3 matrix in Equation 2 is the camera internal parameters calculated in Calibration 1. The coefficient s is a scaling coefficient such that the z component of the coordinate value in the image coordinate system becomes 1.

[0037]

number

[0038] Finally, the free viewpoint image generator 1033 outputs the (x img ,y img ,z c ) and stores a value in the depth image D according to Equation 3. The free viewpoint image generation unit 1033 creates a depth image by repeatedly executing the above process for each point in the LiDAR point cloud.

[0039]

number

[0040] Next, the free viewpoint image generation unit 1033 generates a colored point cloud based on the vehicle body coordinate system. First, distortion correction processing is performed on the camera image. In the distortion correction processing, the distortion coefficient obtained in calibration 1 is used. After that, the free viewpoint image generation unit 1033 uses the camera image and the depth image after distortion correction to generate a colored point cloud based on the camera coordinate system. img ,y img A point in the colored point cloud generated from the RGB elements is expressed by Equation 4. In Equation 4, R, G, and B are arrays that correspond to the RGB three channels of the camera image.

[0041]

number

[0042] Next, the free viewpoint image generating unit 1033 converts the colored point cloud expressed with the camera coordinate system as the base into an expression in the LiDAR coordinate system. And finally, it converts from the LiDAR coordinate system to the vehicle body coordinate system. In this conversion, by using the coordinate conversion matrix obtained in the calibrations 3 and 4, it is possible to convert into a coordinate system with the bottom surface of the rotating body 102 as the origin. Next, the free viewpoint image generating unit 1033 calculates a conversion matrix to the vehicle body coordinate system with the bottom surface of the running body 101 as the origin based on the vehicle body attitude information and the vehicle body dimension values, and performs the same calculation using the matrix to convert the colored point cloud into an expression with the vehicle body coordinate system with the bottom surface of the running body 101 as the origin as the base. By the above processing, colored point cloud information can be generated.

[0043] Furthermore, the free viewpoint image generator 1033 integrates the CG model of the work machine 100 generated based on the above surrounding colored point cloud and vehicle body posture information, and generates an image from an arbitrary viewpoint using an application such as a 3D viewer. This makes it possible to generate a free viewpoint image including the work machine 100 itself, even if an image of the work machine 100 itself is not captured.

[0044] The above method has been described using an example of processing via a depth image, but this is implemented in consideration of cases where AR needs to be overlaid on the video. If only a free viewpoint video is to be obtained, processing may be performed without going through a depth image.

[0045] Fig. 13 to Fig. 15 show examples of images generated by the image generating device 103 for each viewpoint. Fig. 13 is an example of an image viewed from the cab OR. Fig. 14 is an example of an image when the work machine 100 is viewed from the side as a free viewpoint. Fig. 15 is an example of an image when the work machine 100 is viewed from above as a free viewpoint.

[0046] 16 to 18 are display examples in which the boom 111 to the bucket 113 are colored in an image generated by the image generating device 103. FIG. 16 to FIG. 18 correspond to FIG. 13 to FIG. 15, respectively. As exemplified in FIG. 16 to FIG. 18, drawing attributes (such as line type, line thickness, color, etc., the same applies to the case where drawing attributes are changed below) may be changed for some or all of the boom 111 to the bucket 113. For example, when the cylinder pressure of each of the boom 111 to the bucket 113 exceeds a threshold, the image generating device 103 can change the drawing attributes to be different from those of other parts so as to visually display the range where the pressure is exceeded. This allows the operator to visually and quickly grasp the part where an abnormality has occurred.

[0047] <Embodiment 2> 19 to 21 are examples of images of the surrounding environment of the work machine 100 generated by the image generating device 103. FIG. 19 is an example in which an excavation site 1901 is highlighted when the work machine 100 excavates flat land L0. FIG. 20 is an example in which earth and sand loaded into a dump truck 200 is highlighted. Areas 201 to 203 represent the shape of the earth and sand. The boundaries between the areas 201 to 203 correspond to contour lines. Each area may be given a different color. FIG. 21 is an example in which an edge 2101 of a plateau L1 is highlighted.

[0048] <Embodiment 3> 22 is an example of an image generated by the image generating device 103, showing the range of movement of the revolving body 102. The free viewpoint image generating unit 1033 generates a cylinder 2201 having a maximum radius with the rotation center of the revolving body 102 as the central axis, as an image showing the range of movement of the revolving body 102. The range 2202 is obtained by changing the drawing attributes of a portion where the revolving body 102 and a surrounding object may collide.

[0049] Fig. 23 is an example of another image generated by the image generating device 103, showing the movement range of the revolving body 102. Unlike Fig. 22, the free viewpoint image generating unit 1033 generates a cylinder 2301 having the maximum radius including the work machine (boom 111 to bucket 113) as an image showing the movement range of the revolving body 102. The range 2302 has drawing attributes changed for a portion where the work machine may collide with a surrounding object.

[0050] Fig. 24 is an example of an image generated by the image generating device 103 to warn of openwork. Openwork is digging into the lower part of a downward surface that falls in a vertical direction (or a direction close to this). In Fig. 24, this corresponds to digging into a surface 2402 located below a front surface 2401 of the running object 101. The free viewpoint image generating unit 1033 can identify the surface 2402 by identifying the front surface 2401 and the plateau L1. The free viewpoint image generating unit 1033 generates an image in which the surface 2402 is highlighted.

[0051] <Fourth embodiment> Fig. 25 is a functional block diagram of an image generating device 103 according to a fourth embodiment of the present invention. In addition to the configuration described in Fig. 4 of the first embodiment, the bold line portion has been newly added. The switching determination unit 1037 switches the free viewpoint image according to the operating state of the work machine 100 (i.e., the work content instructed by the driver or remote operator to the work machine 100). Examples of the content of the image to be switched include the following:

[0052] (Image example 1: Machine information) As an example of the free viewpoint image, machine information indicating the state of the work machine 100 can be displayed within the free viewpoint image. Examples of the machine information include, but are not limited to, the following:

[0053] (Image example 1: Machine information: Cylinder pressure) When the work machine 100 is performing excavation work, an image showing the pressure of a hydraulic cylinder equipped in the work machine 100 can be displayed within the free viewpoint image. For example, by applying a color representing the cylinder pressure to each of the boom 111, arm 112, and bucket 113, cylinder pressure information can be superimposed on the free viewpoint image. Figures 16 to 18 are one example.

[0054] (Image example 1: Machine information: Rotating body movement range) When the work machine 100 rotates the revolving body 102, an image showing the movement range (or rotation trajectory) of the revolving body 102 can be displayed in the free viewpoint image. For example, a cylinder showing the movement range of the revolving body 102 can be superimposed on the free viewpoint image. Figures 22 to 23 are one example.

[0055] (Image example 1: Machine information: Watermark area) When the work machine 100 is performing excavation work, the area in the free viewpoint image that corresponds to the openwork excavation can be highlighted, for example, by coloring it. Figure 24 shows one example.

[0056] (Image example 2: Environmental information) As an example of a free viewpoint image, environmental information that indicates the state of the surrounding environment of the work machine 100 can be displayed within the free viewpoint image. Examples of environmental information include, but are not limited to, the following:

[0057] (Image example 2: Environmental information: Loading shape) When the work machine 100 loads the excavated earth and sand onto a dump truck or the like, an image showing the shape of the loaded earth and sand can be displayed within the free viewpoint image. Figure 20 shows one example.

[0058] (Image example 2: Environmental information: Excavation shape) When the work machine 100 performs excavation work, an image showing the shape of the excavated area can be displayed in the free viewpoint image. Fig. 19 is one example. The shape of the excavated area 1901 may also be displayed using contour lines and colors, similar to the areas 201 to 203 in Fig. 20.

[0059] (Image example 2: Environment information: Surrounding terrain) When the work machine 100 is traveling, an image showing the unevenness of the surrounding terrain and the shape of its edges can be displayed within the free viewpoint image. Fig. 21 shows an example in which an edge 2101 is highlighted. The unevenness of the terrain may also be displayed in shape using contour lines or colors, similar to the areas 201-203 in Fig. 20.

[0060] Machine information and environmental information have been described as examples of types of images that can be superimposed on a free viewpoint image. When these are superimposed on a free viewpoint image, the superimposed parts may be emphasized by appropriately adjusting the transparency, color, line thickness, etc. of the superimposed image. Furthermore, the switching determination unit 1037 may switch the free viewpoint depending on the work performed by the work machine 100 and the type of superimposed image. For example, when displaying a watermark area, a side viewpoint is desirable as shown in FIG. 24.

[0061] Fig. 26 is a functional block diagram of the image generating device 103 when a free viewpoint image is not generated. In addition to the configuration described in Fig. 5 of the first embodiment, the parts indicated by bold lines are newly added. The added parts are the same as those in Fig. 25.

[0062] Fig. 27 is a state transition diagram of the work machine 100. While moving (S31), other operations such as excavation and turning are not performed. In the standby state (S32), excavation (S33) or turning (forward) (S34) is performed. Turning (forward) is a preliminary operation to releasing the excavated soil (S35). After releasing the soil, the rotating body 102 is returned to its original position (S36: turning (return)), and a standby or excavation operation is performed. During each operation, the above-mentioned machine information and environmental information can be displayed together with an appropriate free viewpoint.

[0063] <Fifth Preferred Embodiment> Fig. 28 shows an example of a threshold value that serves as a trigger for switching the free viewpoint image. As one example, during a period in which the detection value of the hydraulic cylinder sensor exceeds the threshold value, the switching determination unit 1037 outputs a switching signal to the free viewpoint image generation unit 1033 to instruct it to switch the free viewpoint image. In this example, the detection value of the vehicle sensor that detects the state of the work machine 100 is used as the trigger, but similar triggers can also be set for machine information and environmental information. For example, the following examples are possible.

[0064] (Display switching example 1) The image generating device 103 acquires the pressure of the cylinder equipped in the work machine 100 as information representing the state of the work machine 100. The sensor S4 (vehicle sensor) can be configured with a boom cylinder pressure sensor, an arm cylinder pressure sensor, a bucket cylinder pressure sensor, etc. If the acquired cylinder pressure exceeds a threshold value, the image generating device 103 changes the drawing attributes of the portion where the pressure exceeds the threshold value. Figures 16 to 18 are one example. For the free viewpoint, the image is configured to include at least the portion where the cylinder pressure exceeds the threshold value in the free viewpoint image. This makes it possible to notify the operator that the operating state of the work machine 100 is abnormal.

[0065] (Display switching example 2) The image generating device 103 can also obtain the degree of roughness of the arrival location where the loaded vehicle carrying the earth and sand excavated by the work machine 100 is scheduled to arrive, as information representing the state of the environment surrounding the work machine 100. For example, the degree of roughness can be obtained by capturing images of the arrival location with sensors S1 to S3, or by measuring the three-dimensional shape of the arrival location with LiDAR. If the obtained degree of roughness exceeds a threshold, the image generating device 103 changes the drawing attribute of the arrival location. The free viewpoint is configured to include at least the arrival location. This makes it possible to notify the operator that the arrival location is not suitable for the arrival of a loaded vehicle.

[0066] (Display switching example 3) The image generating device 103 can also obtain the predicted weight of the soil on the loading vehicle that carries the soil excavated by the work machine 100 as information representing the state of the surrounding environment of the work machine 100. For example, the accumulation of soil can be calculated by imaging the soil with the sensors S1 to S3 or measuring the three-dimensional shape of the soil with LiDAR. The predicted weight of the soil can be calculated by multiplying the calculated volume by the average volume density of the soil. If the calculated predicted weight exceeds a threshold, the image generating device 103 changes the drawing attribute of the loaded soil. FIG. 20 is one example. The free viewpoint is configured to include at least the loaded soil. This makes it possible to notify the operator of overloading of soil.

[0067] (Display switching example 4) The image generating device 103 can also obtain the distance between the front of the work machine 100 (the rear end when retreating) and a slope that exists in the traveling direction of the work machine 100 as information that represents the state of the surrounding environment of the work machine 100. For example, the distance from the work machine 100 to the slope can be obtained by capturing images of the surrounding terrain of the work machine 100 with the sensors S1 to S3 or measuring the surrounding terrain with LiDAR. The image generating device 103 (the AR image generating unit 1034 or the free viewpoint image generating unit 1033) changes the drawing attribute of the end of the slope if the obtained distance is less than a threshold value (or calculates the risk of the work machine 100 sliding down the slope based on the obtained distance and the risk is equal to or greater than the threshold value). FIG. 21 is one example. The risk of sliding down depends in principle on the distance between the work machine 100 and the slope, but appropriate parameters such as the hardness of the ground may be taken into account. The free viewpoint is configured to include at least the end of the slope. This makes it possible to notify the operator of the possibility of the work machine 100 sliding down the slope.

[0068] (Display switching example 5) The image generating device 103 may acquire the range of movement (swivel range) of the revolving unit 102 or the rotating body including the revolving unit 102 and the boom 111 to the bucket 113 when the work machine 100 rotates, as information representing the state of the work machine 100. The range of movement can be calculated using the size and extension range of each part, which are stored in advance as shape information of the work machine 100. The image generating device 103 can further acquire the positions of objects present around the work machine 100, as information representing the state of the environment around the work machine 100. For example, the positions of surrounding objects can be acquired by capturing images of the periphery of the work machine 100 with sensors S1 to S3, or by measuring the surrounding topography with LiDAR. If the distance between the peripheral object and the turning range is less than a threshold value (or the risk of the work machine 100 colliding with the peripheral object due to a turning operation is calculated based on the position of the peripheral object and the risk is equal to or greater than a threshold value), the image generating device 103 (the AR image generating unit 1034 or the free viewpoint image generating unit 1033) changes the drawing attributes of the portion of the turning range that may collide with the peripheral object. Figs. 22 to 23 are one example. The collision risk depends in principle on the distance between the work machine 100 and the peripheral object, but appropriate parameters such as the movable range due to the extension and retraction operation of the boom 111 to the bucket 113 may also be taken into consideration. The free viewpoint is configured to include at least the entire turning range. This makes it possible to notify the operator of the possibility of the work machine 100 colliding with the peripheral object.

[0069] (Display switching example 6) The image generating device 103 may acquire the distance between the front of the work machine 100 and a slope present in front of the work machine 100 as information representing the state of the surrounding environment of the work machine 100. For example, the distance from the work machine 100 to the slope can be acquired by capturing an image of the surrounding terrain of the work machine 100 with the sensors S1 to S3 or measuring the surrounding terrain with LiDAR. The image generating device 103 (the AR image generating unit 1034 or the free viewpoint image generating unit 1033) changes the rendering attribute of the slope if the acquired distance is less than a threshold value (or calculates the risk of the work machine 100 digging the ground below itself (deep trench risk) based on the acquired distance and the risk is equal to or greater than the threshold value). FIG. 24 is one example. The deep trench risk depends in principle on the distance between the work machine 100 and the slope, but appropriate parameters such as the hardness of the ground may be added. The free viewpoint is configured to include at least the side of the work machine 100 and the slope. This makes it possible to notify the operator of the possibility that the work machine 100 will perform trench digging.

[0070] <Sixth embodiment> In the above embodiment, it has been described that the image generating device 103 provided in the work machine 100 captures an image of the surrounding environment of the work machine 100, performs coordinate system processing, and transmits the results to the remote control room 300. Part of the processing performed by the image generating device 103 may be performed in the remote control room 300. A specific example will be described in embodiment 6 of the present invention.

[0071] 29 shows a modified example of the image generation device 103. In this configuration example, a group of sensors provided on the work machine 100 detects the state of the work machine 100 and the state of the environment surrounding the work machine 100. The image generation device 103 performs coordinate conversion on the data acquired by the sensors and transmits it to the remote operation room 300. The remote operation room 300 uses this to generate an operation assistance image and displays it.

[0072] 29, since the image generating device 103 does not generate a free viewpoint image (or a normal viewpoint image), the free viewpoint image generating unit 1033, the graphic position and shape calculating unit 1032, the AR image generating unit 1034, and the video superimposing unit 1035 are provided on the remote control room 300 side. Instead of these, a coordinate conversion unit 1038 and a 3D information compressing unit 1039 are provided. Although it is possible to transmit the data measured by the imaging device, the 3D measuring device, and the vehicle sensor directly to the remote control room 300, there are cases where the data becomes redundant due to overlapping fields of view, etc. Therefore, the coordinate conversion unit 1038 integrates the coordinate systems of these data before transmitting them to the remote control room 300.

[0073] The coordinate conversion unit 1038 converts the coordinate system of the 3D measurement data acquired by the 3D measurement device and the vehicle information acquired by the vehicle sensor, based on the 2D image acquired by the imaging device. The 3D measurement data is a combination of the 3D measurement data from each of the sensors S1 to S3. The image compression unit 1036 compresses the 2D image from the imaging device, and the 3D information compression unit 1039 compresses the 3D measurement data. The vehicle information does not necessarily need to be compressed because it is relatively small in size. The communication device 104 transmits these data to the remote operation room 300.

[0074] Fig. 30 is a functional block diagram of an operation control device 305. The operation control device 305 in Fig. 30 receives measurement data from each sensor from the image generation device 103 in Fig. 29, and generates a free viewpoint image (or a normal viewpoint image). The operation control device 305 includes a free viewpoint image generation unit 1033 transferred from the image generation device 103, a figure position and shape calculation unit 1032, an AR image generation unit 1034, and a video superimposition unit 1035. It further includes a three-dimensional information decompression unit 3054. The three-dimensional information decompression unit 3054 decompresses the three-dimensional measurement data received from the image generation device 103. The operation of each of the other units is similar to that of the above embodiments.

[0075] Fig. 31 shows a modified example of the image generation device 103. Fig. 31 shows a modified example different from that of Fig. 29. In this configuration example, a group of sensors provided in the work machine 100 detects the state of the work machine 100 and the state of the surrounding environment of the work machine 100. The image generation device 103 performs coordinate transformation on the data acquired by the sensors, generates an operation assistance image, and displays this.

[0076] The image generation device 103 shown in Fig. 31 is mounted on the work machine 100 and is a device that displays an operation assistance image on the work machine 100. Therefore, in addition to the configurations described in Fig. 4 and Fig. 5, it has the configuration described in Fig. 6. However, since there is no need to transmit and receive measurement data between the image generation device 103 and the remote operation room 300, the video compression unit 1036, communication device 306, and video decompression unit 3051 are not required. The input device 311 and display 312 are the same devices as the input device 303 and display 304, respectively.

[0077] Fig. 32 is a configuration diagram of the driver's cab OR. Fig. 32 shows a configuration example in which the image generating device 103 of Fig. 31 is provided. The driver's cab OR includes a driver's seat 321, an operating device 322, an input device 311, a display 312, and an image generating device 103. The driver sits in the driver's seat 321 and inputs operation instructions via the input device 311. The operating device 322 passes the operation instructions to the image generating device 103. The image generating device 103 displays an operation assistance image on the display 312.

[0078] <Modifications of the present invention> The present invention is not limited to the above-described embodiment, and includes various modified examples. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to those having all of the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0079] In the above embodiment, when the image generating device 103 superimposes information representing the state of the work machine 100 (machine information) or information representing the state of the surrounding environment of the work machine 100 (environmental information) on the free viewpoint image (or normal viewpoint image), this information can also be expressed and superimposed using text images. The text images in this case are also an example of machine information or environmental information.

[0080] In the above embodiment, the user may select information to be superimposed on the image generated by the image generating device 103. For example, an experienced operator and a novice operator require different information for operation assistance. Thus, the user may select which of the machine information and the environmental information to superimpose on the operation assistance screen. This is the same whether the operation assistance image is displayed in the remote operation room 300 or in the work machine 100.

[0081] In the above embodiment, the user himself may switch the free viewpoint. At each free viewpoint, machine information and environmental information that are desired to be displayed at the free viewpoint at that time may be superimposed. For example, when displaying an image seen from the cab OR, information exemplified in Figs. 16 to 17, 19 to 20, etc. may be superimposed. The user himself may set in advance which information is to be superimposed at each viewpoint, or it may be preset for each viewpoint.

[0082] In the above embodiment, the information to be superimposed on each viewpoint image may be switched on the device that displays the image, or the display side may receive all the information and then switch which information to superimpose on the image. The switching may be performed, for example, by a user instruction.

[0083] In the above embodiment, a backhoe has been given as an example of the work machine 100, but the configuration example according to the present invention can also be applied when assisting the operation of other work machines. In other words, the same effect as that of the present invention can be achieved with other work machines by presenting machine information and environmental information on a screen viewed by the operator. [Explanation of symbols]

[0084] 100:Work machines 103: Image generating device 1030A: Machine information acquisition section 1030B:Environmental information acquisition department 1031: Coordinate system processing unit 1033: Free viewpoint image generation unit 1034: AR image generation unit 300: Remote Control Room

Claims

1. An image generating device that acquires an image captured by an imaging device provided in a work machine and generates an image used to assist an operator who remotely operates the work machine, a machine information acquisition unit that acquires machine information representing a state of the work machine from a vehicle sensor provided on the work machine; an environmental information acquisition unit that acquires environmental information representing a state of the surrounding environment of the work machine from a three-dimensional measuring device provided on the work machine and an imaging device provided on the work machine; an AR image generating unit that generates an AR image indicating at least one of the mechanical information and the environmental information by superimposing the AR image on the captured image based on the mechanical information and the environmental information; Equipped with The AR image generation unit includes: generating a machine information AR image, which is the AR image representing the machine information, to be superimposed on the captured image at a position corresponding to a part of the work machine indicated by the machine information; or An environmental information AR image is generated, which is an AR image representing the environmental information to be superimposed at a position corresponding to the surrounding environment of the work machine indicated by the environmental information, among the captured images.

1. An image generating apparatus comprising:

2. a free viewpoint image generating unit that generates a free viewpoint image of the work machine viewed from an arbitrary viewpoint from the captured image, The AR image generation unit generates the machine information AR image or the environmental information AR image corresponding to the free viewpoint image.

2. The image generating apparatus according to claim 1.

3. A coordinate system processing unit that converts a coordinate system, the coordinate system processing unit converts a coordinate system of three-dimensional measurement data measured by a three-dimensional measurement device equipped in the work machine into a coordinate system of an imaging device equipped in the work machine; the coordinate system processing unit converts the three-dimensional measurement data converted into the coordinate system of the imaging device into a coordinate system of the free viewpoint image generated by the free viewpoint image generating unit; The free viewpoint image generating unit generates the free viewpoint image by projecting the three-dimensional measurement data converted into a coordinate system of the free viewpoint image onto the free viewpoint image.

3. The image generating apparatus according to claim 2.

4. The AR image generation unit is configured to be able to switch between the machine information, the environmental information, or both the machine information and the environmental information based on the machine information and the environmental information, or in response to a selection by an operator who operates the work machine.

2. The image generating apparatus according to claim 1.

5. The AR image generating unit generates the AR image in which a drawing attribute of a part of the image is changed so that the part of the image is displayed more emphasized than other parts of the image, based on the machine information and the environmental information.

2. The image generating apparatus according to claim 1.

6. The free viewpoint image generating unit is configured to be able to change a viewpoint in the free viewpoint image based on the machine information and the environmental information, or in response to a selection by an operator who operates the work machine.

3. The image generating apparatus according to claim 2.

7. the AR image generation unit acquires information regarding whether or not the work machine is performing a predetermined operation from the vehicle sensor provided on the work machine, from the three-dimensional measuring device provided on the work machine, or from the imaging device provided on the work machine, The AR image generating unit generates the AR image in which an area corresponding to the predetermined action is emphasized.

2. The image generating apparatus according to claim 1.

8. The AR image generation unit and the free viewpoint image generation unit, in response to an operation performed by the work machine, Switching the AR image to be superimposed on the captured image between the machine information AR image, the environmental information AR image, and both the machine information AR image and the environmental information AR image; By changing a drawing attribute of a part of the AR image, the AR image is generated in which the part of the AR image is emphasized more than other parts of the AR image, or or Changing the viewpoint of the free viewpoint image 3. The image generating apparatus according to claim 2.

9. The AR image generation unit and the free viewpoint image generation unit acquire information regarding a predetermined threshold in the machine information or the environmental information, and based on the threshold, Switching the AR image to be superimposed on the captured image between the machine information AR image, the environmental information AR image, and both the machine information AR image and the environmental information AR image; By changing a drawing attribute of a part of the AR image, the AR image is generated in which the part of the AR image is emphasized more than other parts of the AR image, or or Changing the viewpoint of the free viewpoint image 3. The image generating apparatus according to claim 2.

10. the AR image generation unit and the free viewpoint image generation unit acquire, as the machine information, information regarding a pressure of a cylinder provided in the work machine from the vehicle sensor; When the pressure exceeds the threshold, the free viewpoint image generation unit switches the viewpoint of the free viewpoint image so as to include at least the cylinder where the pressure exceeds the threshold, and the AR image generation unit changes drawing attributes of the region of the AR image of the cylinder where the pressure exceeds the threshold, thereby generating the AR image that is displayed more emphasized than other regions.

10. The image generating apparatus according to claim 9.

11. The AR image generation unit and the free viewpoint image generation unit acquire, as the environmental information, from the three-dimensional measuring device or the imaging device, information regarding the degree of unevenness of an arrival location where a loading vehicle carrying the soil excavated by the work machine is scheduled to arrive, When the degree of unevenness exceeds the threshold, the free viewpoint image generation unit switches the viewpoint of the free viewpoint image so as to include at least the arrival location, and the AR image generation unit changes drawing attributes of an area of ​​the arrival location in the AR image to generate the AR image that is displayed more emphasized than other parts.

10. The image generating apparatus according to claim 9.

12. the AR image generating unit and the free viewpoint image generating unit acquire, as the environmental information, from the three-dimensional measuring device or the imaging device, information regarding an expected weight of the soil on the loading vehicle when the soil excavated by the work machine is transferred to the loading vehicle; When the predicted weight exceeds the threshold value, the free viewpoint image generation unit switches the viewpoint of the free viewpoint image so as to include at least the soil on the loaded vehicle, and the AR image generation unit changes drawing attributes of an area of ​​the AR image that includes the soil on the loaded vehicle, thereby generating the AR image that is displayed more emphasized than other parts.

10. The image generating apparatus according to claim 9.

13. the AR image generation unit and the free viewpoint image generation unit acquire, as the environmental information, from the three-dimensional measuring device or the imaging device, information relating to the distance between the foreground in the traveling direction of the work machine and a slope existing in the traveling direction of the work machine; the AR image generation unit or the free viewpoint image generation unit calculates a risk of the work machine sliding down the slope based on the distance; When the risk level exceeds the threshold, the free viewpoint image generation unit switches the viewpoint of the free viewpoint image so as to include at least an end of the slope, and the AR image generation unit changes drawing attributes of an area of ​​the end of the slope in the AR image to generate the AR image that is displayed more emphasized than other parts.

10. The image generating apparatus according to claim 9.

14. the AR image generation unit and the free viewpoint image generation unit acquire, as the machine information, information regarding a movement range when the work machine turns, from the vehicle sensor; the AR image generation unit and the free viewpoint image generation unit acquire, as the environmental information, information on the positions of objects present around the work machine from the three-dimensional measuring device or the imaging device; the AR image generation unit or the free viewpoint image generation unit calculates a risk of a collision between the work machine and the object when the work machine turns, based on the movement range and the position of the object; When the risk level exceeds the threshold, the free viewpoint image generation unit switches the viewpoint of the free viewpoint image so as to include at least the movement range, and the AR image generation unit changes drawing attributes of an area of ​​the movement range where there is a possibility of the work machine colliding with the object, thereby generating the AR image that is displayed with emphasis more than other areas.

10. The image generating apparatus according to claim 9.

15. the AR image generation unit and the free viewpoint image generation unit acquire, as the environmental information, information relating to a distance between a front surface of the work machine and a downhill surface present in front of the work machine from the three-dimensional measuring device or the imaging device; the AR image generation unit or the free viewpoint image generation unit calculates a risk of the work machine excavating an area below the work machine based on the distance; When the degree of risk exceeds the threshold, the free viewpoint image generating unit switches the viewpoint of the free viewpoint image so as to include at least a side surface of the work machine and the downward surface, and the AR image generating unit changes a drawing attribute of an area of ​​the downward surface to generate the AR image that is displayed more emphasized than other parts.

10. The image generating apparatus according to claim 9.

16. An operation assistance system for assisting an operation of a work machine, comprising:

2. The image generating device according to claim 1 , a display device that displays the image generated by the image generating device; An operation support system comprising:

Images