Image display system, image display method, and program

Abstract

To provide an image display system or the like capable of grasping a position in a depth direction in a two dimensional image.SOLUTION: There are provided a coordinate acquisition unit that acquires a position of an object as three dimensional coordinates with a depth direction as a first axis, a conversion unit that converts the three dimensional coordinates into a two dimensional image with the first axis and a second axis orthogonal to the first axis as coordinate axes, and a display unit that displays the two dimensional image in a display form according to coordinates of a third axis orthogonal to the first axis and the second axis. The display part displays the two dimensional image in a display color corresponding to the coordinate of the third axis as a display form.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image display system, an image display method, and a program. [Background technology]

[0002] Patent Document 1 discloses a remote control device equipped with a scanner image display unit that displays three-dimensional distance data acquired by a three-dimensional scanner. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-43488 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the technology disclosed in Patent Document 1, it is not possible to grasp the position in the depth direction in the two-dimensional image only from the two-dimensional image displayed by the scanner image display unit.

[0005] Therefore, in one aspect, the present invention aims to provide an image display system and the like that allows a user to grasp a position in the depth direction in a two-dimensional image. [Means for solving the problem]

[0006] In one embodiment, a coordinate acquisition unit that acquires the position of an object as three-dimensional coordinates with a depth direction as a first axis; a conversion unit that converts the three-dimensional coordinates into a two-dimensional image having coordinate axes of the first axis and a second axis orthogonal to the first axis; a display unit that displays the two-dimensional image in a display format according to coordinates of a third axis perpendicular to the first axis and the second axis; An image display system is provided, comprising: [Effects of the Invention]

[0007] According to one aspect of the present invention, it is possible to grasp a position in the depth direction in a two-dimensional image. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating a configuration of an image display system according to an embodiment of the present invention. [Figure 2] FIG. 1 is a diagram showing the state of a work site. [Figure 3] FIG. 2 is a diagram illustrating an example of an image captured by a camera installed in the driver's seat of a work machine. [Figure 4] 10A and 10B are diagrams illustrating examples of images displayed by a display unit. [Figure 5] FIG. 2 is a diagram illustrating an example of a two-dimensional image generated by a display unit. [Figure 5A] FIG. 10 is a top view showing an example of the arrangement of objects. DETAILED DESCRIPTION OF THE INVENTION

[0009] FIG. 1 is a diagram showing the configuration of an image display system according to this embodiment.

[0010] As shown in Figure 1, the image display system 10 of this embodiment includes a coordinate acquisition unit 11 that acquires the position of an object as three-dimensional coordinates with the depth direction as the first axis, a conversion unit 12 that converts the acquired three-dimensional coordinates into a two-dimensional image with the first axis and a second axis perpendicular to the first axis as coordinate axes, and a display unit 13 that displays this two-dimensional image in a display form according to the coordinates of the third axis perpendicular to the first axis and the second axis.

[0011] The image display system 10 can be configured using one or more computers on which a predetermined program is installed.

[0012] Next, an example of the operation of the image display system 10 will be described.

[0013] The image display system 10 can be used, for example, to generate images to be presented to an operator remotely operating a work machine. Conventionally, when remotely operating a work machine such as heavy machinery, an image of the work site captured by a camera is transmitted to a monitor placed in front of the operator, and the operator operates the work machine while viewing the image. However, the image does not allow the operator to recognize positions in the depth direction, so the operator has to rely on intuition and skill, resulting in problems such as reduced productivity and difficulty in securing operators skilled in remote operation.

[0014] According to the image display system 10 of this embodiment, it is possible to provide the operator with information necessary for remote operation via images generated by the display unit 13. Furthermore, even when an operator is on board the work machine, the images generated by the display unit 13 can be used as effective information for operating the work machine.

[0015] FIG. 2 is a diagram showing the state of the work site, FIG. 3 is a diagram showing an example of an image captured by a camera installed in the driver's seat of the work machine, and FIG. 4 is a diagram showing an example of an image displayed on the display unit.

[0016] In this case, the coordinate acquisition unit 11 acquires point cloud data (three-dimensional coordinates) via a sensor 20, such as a 3D Lidar or stereo camera, attached to a backhoe 50 (FIG. 2) serving as a work machine. The conversion unit 12 converts the three-dimensional coordinates acquired by the sensor 20 into a two-dimensional image having a first axis and a second axis perpendicular to the first axis as coordinate axes. The display unit 13 displays this two-dimensional image in a display form according to the coordinates of a third axis perpendicular to the first and second axes.

[0017] Here, the direction of the first axis corresponds to the depth direction in Fig. 3 and the left-right direction in Fig. 4. The direction of the second axis corresponds to the height direction in Fig. 3 and the up-down direction in Fig. 4. The direction of the third axis corresponds to the width direction in Fig. 3 and the depth direction in Fig. 4.

[0018] As shown in FIG. 4, in this embodiment, the display unit 13 displays an image of the surrounding objects, including the bucket 51, viewed cross-sectionally from the side of the bucket 51. This allows the operator to easily recognize the position and perspective of the bucket 51 of the backhoe 50. In other words, it is difficult to determine the position of the bucket 51 in the depth direction (direction of the first axis) using only the image (FIG. 3) captured by the camera installed in the driver's seat of the backhoe 50. However, by viewing the image generated by the display unit 13, the operator can accurately recognize the position and perspective of the bucket 51 in the depth direction (direction of the first axis). This allows for accurate work using the bucket 51.

[0019] Furthermore, in this embodiment, the display unit 13 displays the two-dimensional image in a display format according to the coordinates of a third axis that is orthogonal to the first axis and the second axis.

[0020] Next, an example of a display form of a two-dimensional image on the display unit 13 will be described.

[0021] FIG. 5 is a diagram illustrating a two-dimensional image generated by the display unit, and FIG. 5A is a top view illustrating an example of the arrangement of objects.

[0022] 5 and 5A show an example in which a sand pile 71 and a wall-like obstacle 72 are placed at a work site. In the two-dimensional image shown in Fig. 5, the position in the depth direction (Y direction, third axis direction) in the two-dimensional image is expressed using, for example, display color. In the example of Fig. 5A, the point cloud data is partitioned into three regions 80A, 80B, and 80C according to its Y coordinate (third axis coordinate), and different display colors are assigned to the regions 80A, 80B, and 80C, respectively.

[0023] In this case, in the two-dimensional image shown in Fig. 5, sand pile 71 is positioned in area 80A and area 80B, and obstacle 72 is positioned in area 80C. Bucket 51 is also positioned in area 80B.

[0024] Therefore, the operator can recognize the positions of the sand pile 71, the obstacle 72, and the bucket 51 in the Y direction (the coordinate of the third axis) through the display colors of the lines corresponding to the outlines of these objects. This allows the operator to understand, for example, the positional relationship between the sand pile 71 and the obstacle 72 in the Y direction (the coordinate of the third axis). Furthermore, the position of the bucket 51 in the Y direction (the coordinate of the third axis) can also be understood from the display color, which effectively supports the operator's work. Furthermore, it is easy to avoid the bucket 51 coming into contact with the obstacle 72. This is particularly effective when remotely operating the backhoe 50 while achieving productivity similar to that of on-board operation, when moving the bucket 51 as close as possible to a wall, or when loading the dump truck 60 as safely as possible.

[0025] As a display format for the two-dimensional image, instead of assigning different display colors to the regions 80A, 80B, and 80C, the display color may be changed continuously in the Y direction (the coordinate of the third axis). For example, a gradation may be applied to the display color in accordance with the change in color temperature relative to the temperature.

[0026] In this embodiment, the display unit 13 can accept the specification of a range in the Y direction (the coordinate of the third axis) and display only objects within the specified range as a two-dimensional image. For example, while working without rotating the boom of the backhoe 50 to which the bucket 51 is attached, only the area 80B in FIG. 5A corresponding to the range of movement of the bucket 51 and the boom may be specified, and only the area 80B may be displayed as a two-dimensional image. Furthermore, when rotating the boom of the backhoe 50, the areas 80A to 80C, including the areas 80A and 80C, may be specified, and the entire areas 80A to 80C may be displayed as a two-dimensional image. This allows the operator to monitor the range in which the bucket 51 and the boom may move using a two-dimensional image.

[0027] Furthermore, the display unit 13 may acquire the operating state of the boom or the like of the backhoe 50 from the control device 55 (FIG. 1) of the backhoe 50. In this case, the range displayed as a two-dimensional image by the display unit 13 may be switched depending on the acquired operating state, for example, whether or not the boom is rotating.

[0028] In addition, restriction lines that restrict the height and depth directions are added to the two-dimensional image displayed on the display unit 13, and the operator can operate the machine while checking the restriction lines, thereby preventing the bucket 51 and boom from coming into contact with obstacles 72, etc.

[0029] In this way, the image display system 10 of this embodiment can be applied to the remote operation of a work machine. In this case, when operating the bucket 51 of the backhoe 50, for example, the operator can easily visually recognize the detailed distance information between the excavation location, loading location, and the bucket 51, thereby improving the operational stability, productivity, and safety.

[0030] In the image display system of the present disclosure, the directions of the first to third axes are arbitrary and are not limited to the directions shown in the above embodiments. Furthermore, the image display system of the present disclosure can be applied to any object and for any purpose.

[0031] Although the embodiments have been described in detail above, the present invention is not limited to the specific embodiments, and various modifications and changes are possible within the scope of the claims. In addition, it is also possible to combine all or a plurality of components of the above-described embodiments. [Explanation of symbols]

[0032] 10 Image display system 11 Coordinate acquisition section 12 Conversion unit 13 Display section

Claims

1. a coordinate acquisition unit that acquires the position of an object as three-dimensional coordinates with a depth direction as a first axis; a conversion unit that converts the three-dimensional coordinates into a two-dimensional image having coordinate axes of the first axis and a second axis orthogonal to the first axis; a display unit that displays the two-dimensional image in a display format according to coordinates of a third axis perpendicular to the first axis and the second axis; An image display system comprising:

2. The image display system according to claim 1 , wherein the display unit displays the two-dimensional image in a display color according to the coordinate of the third axis as the display format.

3. 2. The image display system according to claim 1, wherein the display unit receives a specification of a range of coordinates of the third axis, and displays only an object within the specified range as the two-dimensional image.

4. a coordinate acquisition step of acquiring the position of the object as three-dimensional coordinates with the depth direction as a first axis; a conversion step of converting the three-dimensional coordinates into a two-dimensional image having the first axis and a second axis orthogonal to the first axis as coordinate axes; a display step of displaying the two-dimensional image in a display format according to coordinates of a third axis perpendicular to the first axis and the second axis; An image display method comprising:

5. a coordinate acquisition step of acquiring the position of the object as three-dimensional coordinates with the depth direction as a first axis; a conversion step of converting the three-dimensional coordinates into a two-dimensional image having the first axis and a second axis orthogonal to the first axis as coordinate axes; a display step of displaying the two-dimensional image in a display format according to coordinates of a third axis perpendicular to the first axis and the second axis; A program that causes a computer to execute an image display method comprising:

Images