Display device and display program

Abstract

To provide a display device and a display program capable of displaying a work machine and the surroundings thereof on a display screen with excellent visibility.SOLUTION: A display device 20 displays a bird's-eye view 31 of the circumference of a work machine 1 and information indicating a position of an imaging device 10 present in the periphery of the work machine 1. The display device 20 includes a bird's-eye view display region 31 for displaying a bird's-eye view, a captured image display region 32, which is different from the bird's-eye view display region 31 and displays a captured image captured by the imaging device 10, and an acquisition unit 21 for acquiring a display range ROI which is displayed in an enlarged manner in the captured image. The display range ROI acquired by the acquisition unit 21 is displayed in an enlarged manner on the captured image display region 32.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a display device and a display program. [Background technology]

[0002] 2. Description of the Related Art In order to improve the workability of an operator of a work machine such as a crane, technology is being developed that captures images of the work machine and its surroundings with a camera and displays the images on a display screen located near the operator, for example. Patent Document 1 also discloses a technique in which a monitor is equipped with a photographing device, and a map including a monitored area and a forward area of ​​each monitor stationed in the monitored area are displayed on a display screen. Patent Document 1 also allows a monitor to be selected, and an image captured by the photographing device worn by the selected monitor is displayed small on, for example, the upper right part of the display screen. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-137514 A Summary of the Invention [Problem to be solved by the invention]

[0004] If one considers applying the above-mentioned technology described in Patent Document 1 to a technology for capturing images of a work machine and its surroundings with a camera and displaying the images on a display screen, the image captured by the camera would be displayed, for example, in the upper right corner of the display screen. In that case, however, the images of the work machine and its surroundings captured by the camera will be displayed small, making them difficult for the operator to see and resulting in poor visibility.

[0005] The present invention has been made in consideration of the above-mentioned points, and has an object to provide a display device and a display program that are capable of displaying a work machine and its surroundings with good visibility on a display screen. [Means for solving the problem]

[0006] According to one aspect of the display device of the present invention, A display device that displays an overhead view of the surroundings of a work machine and information indicating the positions of image capture devices present around the work machine, an overhead view display area for displaying the overhead view; a photographed image display area, separate from the overhead view display area, for displaying a photographed image photographed by the photographing device; an acquisition unit that acquires a display range to be enlarged and displayed in the captured image; Equipped with The display range acquired by the acquisition unit is enlarged and displayed in the captured image display area.

[0007] According to another aspect of the display device of the present invention, The display program causes the display device to execute the above display. Effect of the Invention

[0008] According to the present invention, it is possible to display the work machine and its surroundings on the display screen with good visibility. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a side view illustrating a configuration example of a crane. [Diagram 2] FIG. 2 is a block diagram showing a configuration of the imaging device. [Diagram 3] FIG. 1 is a block diagram illustrating a configuration of a display device. [Figure 4] FIG. 11 is a diagram illustrating an example of a display method on a display screen of a display device. [Diagram 5] FIG. 13 is a diagram illustrating an example of a display on a display screen when there are multiple workers. [Figure 6]FIG. 1A is a diagram showing a state in which a display range is specified on a captured image display area, and FIG. 1B is a diagram showing a state in which the display range is enlarged and displayed. [Figure 7] FIG. 13 is a diagram illustrating a method for enlarging a display range. [Figure 8] 11 is a diagram showing a photographing range corresponding to a display range displayed in a photographed image display area. FIG. [Figure 9] 13A and 13B are diagrams for explaining a method of displaying a shooting range corresponding to a display range on an overhead view display area. [Figure 10] FIG. 13 is a diagram showing a point of interest set on the bird's-eye view display area. [Figure 11] (a) is a diagram showing a state in which the point of interest is not within the shooting range corresponding to the display range, and (b) is a diagram showing a state in which the shooting range has been changed so that the point of interest is within the shooting range corresponding to the display range. [Figure 12] 13A and 13B are diagrams illustrating a method for changing a shooting range corresponding to a display range so that a point of interest approaches the center of the display range. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of a display device and a display program according to the present invention will be described with reference to the accompanying drawings. In the following, the working machine is a crane, and a crawler crane is used as an example of a crane, but the present invention is not limited to this and can be applied to any type of cranes, including other mobile cranes such as wheel cranes, truck cranes, rough terrain cranes, and all-terrain cranes, as well as tower cranes, overhead cranes, jib cranes, retractable cranes, stacker cranes, gantry cranes, and unloaders. The present invention is also applicable not only to cranes equipped with a load hook, but also to cranes that suspend attachments such as magnets and earth drill buckets. Of course, the present invention is not limited to cranes, and can also be applied to working machines such as hydraulic excavators and forklifts.

[0011] As shown in FIG. 1, the crane 1 is configured to include a self-propelled crawler-type lower traveling body 2 and an upper rotating body 3 that is mounted on the lower traveling body 2 so as to be rotatable. A boom 4 is attached so as to be able to rise and fall on the front side of the upper rotating body 3. A counterweight 5 is attached to the rear of the upper rotating body 3 to balance the weight of the boom 4 and the suspended load.

[0012] The boom 4 is raised and lowered by a boom winch (not shown) which winds up and unwinds a boom rope 6. A hoisting rope 8 which has a hook 7 at its tip is wound up and unwinded by a hoisting winch (not shown). A cab 9 is disposed on the right front side of the upper rotating body 3. The cab 9 is a control seat where an operator (not shown) sits and operates the crane 1.

[0013] The cab 9 is equipped with a display device 20, which will be described in detail later. In this embodiment, the display device 20 is provided in the cab 9 of the crane 1, but it may be a computer used for monitoring by a site manager or the like, or may be located in a management center or the like away from the site. It may be a mobile terminal such as a tablet carried by a worker or manager at the site, or, in the case of remote operation, it may be a monitor in a remote operation room.

[0014] Around the crane 1, a worker A photographs the crane 1 etc. with a photographing device 10 carried by him / her. As will be described later, there may be a plurality of workers A taking photographs. The photographing device 10 may be a camera, or may be a smartphone or a portable information terminal having a camera function.

[0015] In this embodiment, the photographing device 10 includes a control unit 11, a photographing unit 12, a GNSS (Global Navigation Satellite System) receiving unit 13, a sensor unit 14, and a communication unit 15, as shown in FIG. In addition, other components such as memory may be provided as necessary.

[0016] The control unit 11 includes a CPU (Central Processing Unit) and controls the operation of each part of the image capturing device 10 according to a control program. The photographing unit 12 has a camera function and can photograph moving images. The photographing unit 12 transmits image data of the moving image to the display device 20 via the communication unit 15 together with information such as the angle of view when the photographed image was photographed.

[0017] The GNSS receiver unit 13 is equipped with a receiver that receives radio waves transmitted from GPS (Global Positioning System) satellites, etc., and determines its own position, i.e., the position (latitude, longitude) of the imaging device 10, based on the information superimposed on the received radio waves. The sensor unit 14 includes, for example, a gyro sensor, a geomagnetic sensor, etc., and transmits data measured by the sensors to the control unit 11.

[0018] The control unit 11 calculates its own orientation, that is, the orientation (direction) of the image capturing device 10, based on the transmitted data. The control unit 11 then transmits information on the position and orientation of the image capturing device 10 to the display device 20 via the communication unit 15.

[0019] Next, the display device 20 according to this embodiment will be described. As shown in FIG. 3, the display device 20 includes a control unit 21 consisting of a CPU etc., a communication unit 22, a GNSS receiving unit 23, a sensor unit 24 including a gyro sensor etc., an input unit 25, a display unit 26 having a display screen 30, and a memory 27.

[0020] The control unit 11 is configured with a computer equipped with a CPU and the like, and controls the operation of each part of the display device 20 according to a control program. The control unit 11 also includes a display program, and controls the display unit 26 to display on a display screen 30, which will be described below, in accordance with the display program.

[0021] When the communication unit 22 receives information on the position and orientation of the image capturing device 10 from the image capturing device 10 , the communication unit 22 stores the information in the memory 27 . The GNSS receiving unit 23 determines its own position, that is, the position (latitude, longitude) of the display device 20 based on the information superimposed on the received radio waves. The sensor unit 24 transmits data measured by the gyro sensor and the like to the control unit 21. Then, the control unit 21 calculates its own orientation, i.e., the orientation (direction) of the display device 20, based on the transmitted data, similarly to the control unit 11 of the imaging device 10.

[0022] In this embodiment, the input unit 25 is configured as a touch panel, and an operator (user) can touch the display screen 30 of the display device 20 to input instructions to the display device 20. The input unit 25 may be one that inputs instructions by cursor operation or the like. The display unit 26 performs display on a display screen 30 under the control of the control unit 21. The display screen 30 is formed of, for example, a liquid crystal display.

[0023] The method of displaying on the display screen 30 of the display device 20 and the method of processing in the control unit 21 will be specifically described below. Fig. 4 is a diagram showing an example of how to display on the display screen of a display device. In Fig. 4, the crane 1 (work machine) is depicted in a simplified form as if it were a combination of rectangular parallelepipeds.

[0024] The display screen 30 of the display device 20 includes an overhead view display area 31 and a captured image display area 32 . In this embodiment, the upper right corner of the overhead view display area 31 is set as a captured image display area 32 separate from the overhead view display area 31. In this manner, the captured image display area 32 has a smaller area displayed on the display screen 30 than the overhead view display area 31.

[0025] In the overhead view display area 31, an overhead view of the surroundings of the crane 1 (work machine) is displayed. In this case, the overhead view may be, for example, a photographed image of the surroundings of the crane 1 (work machine) from above by attaching a camera to the hook 7 (see FIG. 1) of the crane 1. Also, the overhead view may be, for example, an around-view image composed of photographed images taken by a plurality of cameras attached to the crane 1 for photographing the surroundings.

[0026] The overhead view also includes symbols such as squares, circles, and dots to represent the crane 1 (work machine), worker A, and surrounding objects, as shown in FIG. Furthermore, the bird's-eye view also includes a combination of an actually photographed image and symbols such as squares, circles, and dots.

[0027] For example, in the bird's-eye view display area 31, the crane 1 is displayed near the center, and the crane 1 is displayed so that the forward direction of the crane 1 faces upward on the display screen 30. Additionally, worker A who is present around the crane 1 is also displayed in the overhead view display area 31. The position of worker A is also the position of the image capturing device 10.

[0028] In this case, as described above, the image of the actual worker A captured in the image actually taken of the crane 1 may be displayed as is in the overhead view display area 31. Furthermore, as shown in FIG. 4, when the crane 1 and worker A are displayed graphically, it is possible to configure the position and orientation of the aforementioned camera 10 to be displayed as the position and orientation of worker A.

[0029] In this case, the control unit 21 of the display device 20 calculates the relative position and orientation between the crane 1 and the camera device 10 based on the position and orientation of the camera device 10 transmitted from the camera device 10 and the position and orientation of the display device 20. Then, as described above, when the crane 1 is displayed in the overhead view display area 31 so that its forward direction faces upward on the display screen 30, the location of the camera 10 on the overhead view display area 31 and the direction in which it faces are calculated.

[0030] As another method for calculating the relative position and orientation between the crane 1 and the camera device 10, it is possible to analyze images captured by multiple cameras capturing images around the crane 1 to obtain the position and facial direction of photographer A, and then calculate the position of the camera device 10 on the overhead view display area 31 and the direction in which it is facing.

[0031] In the example of Figure 4, the position of the photographing device 10 calculated by the control unit 21 is displayed as a circle, and an illustration of the face of worker A is displayed within the circle so that worker A is facing the direction of the photographing device 10. By displaying in this manner, worker A standing around the crane 1 is displayed in a pseudo manner.

[0032] In this way, the overhead view display area 31 is configured to display information indicating the positions of the camera devices 10 present around the crane 1 (work machine). The information indicating the position of the camera devices 10 does not necessarily have to indicate both the position of the camera devices 10 and the method of photographing them, but may indicate either one of them. For example, it may indicate the direction in which the camera devices 10 located outside the overhead view are positioned relative to the work machine. The information indicating the position of the camera devices 10 may be displayed in an area other than the overhead view display area 31, as long as the positional relationship with the work machine is known. 4, the shooting range R of the image capturing device 10 is further displayed as an isosceles triangle (hereinafter simply referred to as a triangle) on the overhead view. The shooting range R of the image capturing device 10 may be displayed as a sector shape or the like.

[0033] In the example of FIG. 4, the center line of the triangle representing the imaging range R (see the dashed line in the figure) represents the imaging direction of the imaging device 10 and coincides with the direction of the face of the worker A. Moreover, the photographing range R (that is, the length of the base of the triangle) is calculated based on the information on the angle of view transmitted from the photographing device 10 as described above.

[0034] That is, if the angle of view is large, the length of the base of the triangle is displayed as being long, and if the angle of view is small, the length of the base of the triangle is displayed as being short. In this way, by configuring the shooting range R of the photographing device 10 to be displayed on the overhead view display area 31, a person who sees it (an operator) can easily intuitively grasp not only the shooting direction of the photographing device 10 but also what range is being photographed.

[0035] As shown in FIG. 5, when there are multiple camera devices 101, 102, ... around the crane 1 (i.e., when there are workers A1, A2, ...), the position, shooting direction and shooting range R of each camera device 101, 102, ... are calculated and displayed. In addition, when displaying actual photographed images of the crane 1 and worker A as an overhead view, it is also possible to configure the system to calculate and display the photographing direction and photographing range R of the photographing device 10 using a triangle or the like in the same manner as described above.

[0036] In the photographed image display area 32, a photographed image photographed by the photographing device 10 is displayed. 5, in this embodiment, when there are multiple camera devices 10 around the crane 1 (work machine), when the operator selects one of them on the display screen 30 by touching it or the like, the control unit 21 displays the image captured by the selected camera device 10 in the captured image display area 32. Therefore, the operator can select on the display screen 30 the camera device 10 capturing the image of the part of the crane 1 that he or she wishes to see, and easily display the captured image of the part that he or she wishes to see.

[0037] On the other hand, when the operator inputs and specifies a display range ROI on the photographed image display area 32, the control unit 21 acquires a display range ROI to be enlarged and displayed in the photographed image displayed in the photographed image display area 32. That is, in this embodiment, the control unit 21 functions as an acquisition unit in the present invention. Specifically, as shown in FIG. 6(a), the control unit 21 acquires information of the specified display range ROI (i.e., the pixel positions of the top, bottom, left and right ends of the display range ROI, etc.) from the captured image displayed in the captured image display area 32.

[0038] Then, the control unit 21 is adapted to enlarge and display the acquired display range ROI in the captured image display region 32, as shown in FIG. 6(b). At this time, it is possible to configure so that the display range ROI is enlarged and displayed in accordance with the captured image display area 32.

[0039] Specifically, as shown in FIG. 7, the control unit 21 calculates how many times the vertical width (number of pixels) Wa of the captured image display area 32 is compared to the vertical width wa of the display range ROI (vertical magnification ra), and calculates how many times the horizontal width Wb of the captured image display area 32 is compared to the horizontal width wb of the display range ROI (horizontal magnification rb).

[0040] Then, the control unit 21 enlarges the display area ROI by the smaller of the vertical magnification ra and the horizontal magnification rb, and displays it in the captured image display area 32 as shown in FIG. 6(b). By enlarging and displaying the display range ROI in this manner, it is possible to display the display range ROI in a greatly enlarged manner within the captured image display region 32.

[0041] As described above, according to the display device 20 of this embodiment, it is possible to greatly enlarge and display the display range ROI that the operator wants to see in the captured image display area 32 of the display screen 30. Therefore, it is possible to display the crane 1 (working machine) and its surroundings on the display screen 30 with good visibility.

[0042] It is also possible to configure the display area 32 so that the operator can pinch out on the captured image display area 32 to enlarge and display the display range ROI in the captured image display area 32 . In this case, the control unit 21 simultaneously performs the process of obtaining the display range ROI specified by the operator from the captured image displayed in the captured image display area 32, and the process of enlarging and displaying the obtained display range ROI in the captured image display area 32.

[0043] Furthermore, when the display range ROI is enlarged and displayed in the captured image display region 32, it may become difficult to know which part of the crane 1 (work machine) is actually being enlarged and displayed. Therefore, for example, as shown in FIG. 8, the control unit 21 can be configured to display an imaging range Rroi corresponding to the display range ROI on the overhead view display area 31.

[0044] In this case, for example, as shown in Figure 9 (a), the width (number of pixels) wbr from the right end of the captured image display area 32 to the right end of the display range ROI is calculated, and the width wbl from the left end of the captured image display area 32 to the left end of the display range ROI is calculated. Then, the ratios rr and rl of the width Wb in the left-right direction of the photographed image display region 32 to the widths wbr and wbl are calculated.

[0045] 9(b), when the length of the base of a triangle representing the shooting range R of the image capturing device 10 in the overhead view display area 31 is 1, a straight line is drawn from the right end of the base of the triangle to the position of ratio rr and the apex of the triangle (i.e., the circle representing the image capturing device 10). Similarly, a straight line is drawn from the left end of the base of the triangle to the position of ratio rl and the apex of the triangle. The triangular area between these two lines becomes the shooting range Rroi corresponding to the display range ROI.

[0046] In this way, by displaying the shooting range Rroi corresponding to the display range ROI on the bird's-eye view display area 31, when the operator enlarges the display range ROI in the captured image display area 32, it becomes possible to easily understand which part of the crane 1 (work machine) is being enlarged, thereby improving understandability.

[0047] As described above, in this embodiment, the video captured by the worker A around the crane 1 and transmitted from the imaging device 10 is displayed in the captured image display area 32 of the display device 20. When the operator designates a display range ROI in the photographed image displayed in the photographed image display area 32, the display range ROI is enlarged and displayed in the photographed image display area 32.

[0048] In this case, if Worker A moves while taking pictures, changes the shooting direction, or moves Crane 1, the point that the operator is focusing on within the display range ROI (hereinafter referred to as the focus point) may move within the display range ROI, or may move out of the display range ROI and no longer be visible within the display range ROI. However, if the operator has to follow the moving part of interest within the display range ROI, it may become difficult to see and may cause eye fatigue. Also, if the part of interest moves outside the display range ROI, it is necessary to re-specify the display range ROI or give instructions to operator A, which is troublesome.

[0049] Therefore, when the operator sets the area of ​​interest, i.e., the focus area, on the overhead view display area 31 of the display device 20, the control unit 21 of the display device 20 can be configured to automatically change the display range ROI to follow this. In that case, for example, when the operator touches a point of interest on the overhead view display area 31 to set a point of interest P on the overhead view display area 31 as shown in Fig. 10, the control unit 21 of the display device 20 acquires information about the point of interest P on the overhead view display area 31 (such as a pixel position on the display screen 30). That is, the control unit 21 is made to function as a point of interest acquisition unit.

[0050] Then, as shown in FIG. 11(a), if the point of interest P is not within the shooting range Rroi corresponding to the display range ROI described above, the point of interest P falls outside the display range ROI of the captured image display area 32 and is no longer reflected within the display range ROI. Therefore, when such a situation occurs, the shooting range Rroi is changed based on the point of interest P on the overhead view display area 31 and the position of the photographing device 10 (the position of the circle in the overhead view display area 31) so that the point of interest P falls within the shooting range Rroi corresponding to the display range ROI, as shown in Figure 11 (b).

[0051] By changing the shooting range Rroi in this manner, even if the position of at least one of the crane 1 (work machine) and the camera device 10 moves or the shooting direction of the camera device 10 changes, the control unit 21 can change the display range ROI based on the position of the point of interest P on the overhead view display area 31 and the position of the camera device 10 so that the point of interest P is displayed within the display range ROI, and the display range ROI can be enlarged and displayed in the captured image display area 32. Therefore, even if the position of the crane 1 (work machine) or the imaging device 10 moves or the imaging direction of the imaging device 10 changes, it is possible to prevent at least the point of interest P from moving out of the display range ROI and becoming invisible within the display range ROI.

[0052] Furthermore, when changing the display range ROI as described above, it is also possible to configure the display range ROI to be changed so that the attention point P approaches the center of the display range ROI. That is, when the positional relationship between the set point of interest P and the shooting range Rroi corresponding to the display range ROI on the overhead view display area 31 is, for example, as shown in FIG. 12(a), the shooting range Rroi is rotated and moved around the position of the photographing device 10 (the circular position in the overhead view display area 31) so that the center line of the triangle representing the shooting range Rroi (see the dotted line in the figure) passes through the point of interest P, as shown in FIG. 12(b).

[0053] Then, the display range ROI corresponding to the rotated and moved shooting range Rroi is enlarged and displayed in the shot image display region 32 of the display device 20. With this configuration, it is possible to change the display range ROI so that the attention point P approaches the center of the display range ROI. Therefore, the attention point P does not move around within the display range ROI, making the attention point P easier to see and improving comprehension.

[0054] It goes without saying that the present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit of the present invention. For example, while the above embodiment shows a case where only an overhead view of the work machine is displayed, it is also possible to configure the system to display a front view, side view, etc. of the work machine, and to perform each process based not only on the planar positional relationship between the work machine and the worker, etc., based on the overhead view, but also on the vertical positional relationship between the work machine and the worker, based on the front view, etc. [Explanation of symbols]

[0055] 1 Crane (working machine) 10. Imaging Device 101, 102 Multiple imaging devices 20 Display device 21 Control unit (acquisition unit, attention point acquisition unit) 31 Bird's-eye view display area 32 Photographed image display area P Points of interest R Shooting range Rroi Shooting range corresponding to the display range ROI Display Range

Claims

1. A display device that displays an overhead view of the surroundings of a work machine and information indicating the positions of image capture devices present around the work machine, an overhead view display area for displaying the overhead view; a photographed image display area, separate from the overhead view display area, for displaying a photographed image photographed by the photographing device; an acquisition unit that acquires a display range to be enlarged and displayed in the captured image; Equipped with the display range acquired by the acquisition unit is enlarged and displayed in the captured image display area. Display device.

2. a photographing range of the photographing device is displayed on the bird's-eye view display area; The display device according to claim 1 .

3. displaying the shooting range corresponding to the display range on the overhead view display area; The display device according to claim 2 .

4. when one of the plurality of photographing devices present around the work machine is selected, the photographed image captured by the selected photographing device is displayed in the photographed image display area. The display device according to claim 1 .

5. The acquisition unit acquires the display range designated on the captured image display area. The display device according to claim 1 .

6. a point of interest acquisition unit for acquiring a point of interest on the overhead view, when the position of at least one of the work machine and the photographing device has moved, or when the photographing direction of the photographing device has changed, the display range is changed based on the position of the attention point and the photographing device acquired by the attention point acquisition unit so that the attention point is displayed within the display range, and the display is enlarged and displayed in the photographed image display area. The display device according to claim 1 .

7. changing the display range so that the attention point approaches the center of the display range; The display device according to claim 6.

8. A display program for causing the display device according to claim 1 to execute the display according to claim 1.

Images