Display system

Abstract

A display system (1) comprises a computer (13) and a display unit (12a). The display unit (12a) displays an image of a work machine (4) and / or the surrounding environment as a main image (MP). The computer (13) displays, as a sub-image (SP) on the display unit (12a), an image of the work machine (4) and / or the surrounding environment as seen from a viewpoint different from that of the main image (MP), the sub-image (SP) being displayed outside the range of a gaze region (A1) in the main image (MP).

Description

Display system

[0001] The present invention relates to a display system that simulates a working machine in a virtual space.

[0002] For example, Patent Document 1 describes a remote operation support device used for simulating the remote operation of a working machine.

[0003] In the prior art as described above, for example, it was common to display an image viewed from one viewpoint such as inside the cab as the main image on one or more displays. Therefore, in order to display an image viewed from another viewpoint set outside the cab as a sub-image, it was necessary to increase the display or display the main image smaller. Therefore, there was a risk that the visibility of the main image would decrease.

[0004] Japanese Unexamined Patent Application Publication No. 2021-179105

[0005] Therefore, an object of the present invention is to provide a display system that can reduce the decrease in visibility of the main image that may occur when displaying images from a plurality of viewpoints.

[0006] The display system includes a computer and a display unit. The display unit displays an image of the working machine and / or the surrounding environment as the main image. The computer displays an image of the working machine and / or the surrounding environment viewed from a viewpoint different from the main image as a sub-image on the display unit outside the range of the attention area in the main image.

[0007] With the above display system, it is possible to reduce the decrease in visibility of the main image when displaying images from a plurality of viewpoints.

[0008] This is a block diagram of the display system 1. This is a diagram showing the work machine 4 displayed on the display system 1. This is a diagram showing a specific example of the operator device 10 of the display system 1. This is a diagram showing a specific example of the main image MP and sub-image SP displayed on the operator device 10. This is a diagram showing a modified version of the main image MP and sub-image SP displayed on the operator device 10. This is a diagram showing a specific example of the main image MP and sub-image SP when the gaze area A1 moves in response to the operation of the work machine 4. This is a diagram showing a specific example of the main image MP and sub-image SP when the viewpoint and gaze point of the main image MP are moved. This is a diagram showing a specific example of the main image MP and sub-image SP when the gaze point is set so that the gaze area A3 is located in the center of the main image MP. This is a diagram showing a specific example of the main image MP and sub-image SP when the direction of the gaze point of the main image MP is tilted upwards compared to Figure 8A. This is a diagram showing a specific example of the main image MP and sub-image SP when the direction of the gaze point of the main image MP is tilted downwards compared to Figure 8A. This is a diagram showing a specific example of the main image MP and sub-image SP when the gaze area A4 changes in response to the operation of the work machine 4. This figure shows specific examples of the main video MP and sub-video SP when the gaze area A5 changes in accordance with the rotational movement of the work machine 4.

[0009] The display system 1 will be described with reference to the drawings.

[0010] As shown in Figure 1, the display system 1 is used as a means to display various information in a simulator that simulates the operation of a work machine 4 (Figure 2) reproduced in a virtual space. The display system 1 is also used as a means to display various information in a remote control device that operates an actual work machine 40 located in a remote location. Thus, the display system 1 of this embodiment can be applied to either a simulator or a remote control device, but below we will mainly describe the case in which the display system 1 is used as an information display means for a simulator. That is, below we will mainly describe the case in which the display system 1 is used as a means to display various objects, including the work machine 4, in a virtual space. When the display system 1 is used as an information display means for a remote control device, the display system 1 is used to display images of the actual work machine 40 and its surroundings.

[0011] First, let's describe the work machine 4 that is reproduced in the virtual space. As shown in Figure 2, the work machine 4 is a machine that performs work, for example, a construction machine that performs construction work. The work machine 4 may be, for example, an excavator, a crane, a bulldozer, a wheel loader, or a dump truck (transport vehicle). Below, we will mainly describe the case where the work machine 4 is an excavator. The work machine 4 may operate based on information pre-set in the computer 13. The work machine 4 may also operate in response to manual operation by the user. The work machine 4 may travel, the upper slewing body 41b (described later) may rotate relative to the lower traveling body 41a (described later), and the posture of the attachment 43 (described later) may change. The work machine 4 may have a device that sprays washer fluid onto the outer surface of the windshield (not shown). The work machine 4 may have a device that wipes away rainwater and washer fluid etc. adhering to the outer surface of the windshield (not shown). The work machine 4 may have lights that illuminate, such as headlights capable of projecting light in front of the work machine 4 (not shown). The work machine 4 does not need to be operational; it simply needs to be placed in the virtual space.

[0012] The work machine 4 comprises a machine body 41 and an attachment 43.

[0013] The machine body 41 is the main body of the work machine 4. The machine body 41 comprises a lower traveling body 41a and an upper rotating body 41b.

[0014] The lower traveling body 41a is capable of traveling on a surface such as the ground. The lower traveling body 41a may be equipped with crawlers 41a1 or wheels (not shown). The lower traveling body 41a may have a lower working section for performing work. The lower working section may be a lower working device such as an attachment for performing work on an object to be worked on. The lower working device may include a dozer blade for leveling the ground, a clamp arm for holding down an automobile when dismantling an automobile, and outriggers.

[0015] The upper slewing body 41b is rotatable relative to the lower traveling body 41a. The upper slewing body 41b may be equipped with a cab 41b2. The cab 41b2 is a room in which an operator who operates the work machine 4 sits. The cab 41b2 may be fixed to the bottom (slewing frame) of the upper slewing body 41b. The cab 41b2 may be movable relative to the bottom of the upper slewing body 41b. In this case, the cab 41b2 may be able to move parallel to the top and bottom of the upper slewing body 41b, rotate in the top and bottom, or move parallel to the top and bottom and rotate. For example, the cab 41b2 may be an elevator cab, a link cab, or a tilt cab. The cab 41b2, which is an elevator cab, is able to move vertically relative to the bottom of the upper slewing body 41b. The driver's cab 41b2, which is a link cab, is installed at the bottom of the upper slewing body 41b via a link mechanism. The driver's cab 41b2, which is a link cab, is movable in the vertical and longitudinal directions relative to the bottom of the upper slewing body 41b by the operation of the link mechanism. The driver's cab 41b2, which is a tilt cab, is able to rotate (tilt) vertically around a pivot axis that extends laterally relative to the bottom of the upper slewing body 41b. The driver's cab 41b2 may be equipped with accessories attached to the main body of the driver's cab 41b2. These accessories may be, for example, guards to protect the top surface of the driver's cab 41b2 and the front window 41b2A (see Figure 4) from flying objects.

[0016] Attachment 43 is the upper working section that performs the work. Attachment 43 is rotatable in the vertical direction, that is, it can be raised and lowered relative to the upper slewing body 41b. Attachment 43 comprises a boom 43a, an arm 43b, and a tip attachment 43c. In the example shown in Figure 2, the work machine 4 has an upper working section and no lower working section, but is not limited to this. That is, the work machine 4 may have an upper working section and / or a lower working section.

[0017] The boom 43a is rotatably attached to the upper slewing body 41b. The arm 43b is rotatably attached to the boom 43a.

[0018] The tip attachment 43c is provided at the tip of the attachment 43. The tip attachment 43c is rotatably attached to the arm 43b. The tip attachment 43c may be a bucket used for scooping or excavating work objects. The tip attachment 43c may be a device for gripping work objects (grapple, crusher, etc.), a device for crushing work objects (breaker, etc.), or a lifting magnet for attracting metal work objects. The object to be worked on by the tip attachment 43c (work object) may be soil, stone, wood, metal, resin, waste, or structure (block, etc.).

[0019] Display system 1 is a system that displays the work machine 4 and / or surrounding environment reproduced in a virtual space as the main image MP (see Figure 4). Display system 1 has a display unit 12a, which is, for example, a display. The display unit 12a has the function of displaying the main image MP. In addition to the main image MP, the display unit 12a also has the function of displaying a secondary image SP (see Figure 4), which is an image of the work machine 4 and / or surrounding environment viewed from a different viewpoint than the main image MP. In this specification, "viewpoint" refers to the starting point of the line of sight, and represents the position of the observer's eyes when viewing the object, that is, where the object is being viewed from. The viewpoint is defined, for example, by coordinates in the virtual space.

[0020] In the display unit 12a, the secondary image SP is displayed outside the gaze area A1 (see Figure 4) of the main image MP. The gaze area A1 is the area that includes the operator's gaze point. In this specification, "gaze point" refers to the endpoint of the line of sight and indicates where the observer is looking. The gaze point is defined, for example, by coordinates in the virtual space. The gaze area A1 that includes such a gaze point is the area to which the operator primarily shifts their gaze. The "operator" is the user of the display system 1 and operates the work machine 4 reproduced in the virtual space. The display system 1 may include, for example, at least one of a personal computer, a tablet, and a smartphone.

[0021] As shown in Figure 1, the display system 1 comprises an operator device 10 and a computer 13. The display system 1 also comprises a server device 20 and an actual work machine 40.

[0022] The operator device 10 is a device operated by an operator. The operator device 10 is used when an operator operates the work machine 4 in a virtual space while using the display system 1. The operator device 10 may also be used to remotely control the actual work machine 40.

[0023] The operator device 10 includes an input device 11 and an output device 12.

[0024] The input device 11 is a device for inputting information into the computer 13. The input device 11 is operated by an operator. The input device 11 may be used to change the viewpoint of the main image MP (see Figure 4) and / or the secondary image SP (see Figure 4). The input device 11 may be used to change the viewpoint of the main image MP / or the secondary image SP. The input device 11 may be used to set the gaze area A1 (see Figure 4) in the main image MP. The input device 11 may be used to input setting information for automatically setting the gaze area A1 in the main image MP. The input device 11 may include at least one of a mouse, keyboard, touch panel, joystick, pedal, and gamepad. The input device 11 may be a touch panel provided on the display unit 12a. The input device 11 may be used for remotely controlling the actual work machine 40. The input device 11 may receive instructions to start and stop the operation of the work machine 4.

[0025] The input device 11 includes an operating device 11a. The operating device 11a is a device that receives input from an operator to operate the work machine 4 in the virtual space. The operating device 11a includes a travel lever 11a2 and an attachment operating lever 11a1.

[0026] The travel lever 11a2 is operated to move the work machine 4. As shown in Figure 3, the travel lever 11a2 is located in front of the seat 10a. The travel lever 11a2 has a pair of levers arranged side by side. A travel pedal is fixed to the lower part of the travel lever 11a2.

[0027] The attachment operating lever 11a1 is operated to control the operation of the attachment 43. The attachment operating lever 11a1 comprises a left operating lever 11a1A and a right operating lever 11a1B. The left operating lever 11a1A is located in front of the left frame of the seat 10a. The right operating lever 11a1B is located in front of the right frame of the seat 10a. The left operating lever 11a1A and the right operating lever 11a1B are configured to allow multiple operations, such as tilting them in the forward / backward and left / right directions, respectively. Functions for rotating the upper slewing body 41b of the work machine 4 (see Figure 2), operating the boom 43a, operating the arm 43b, and operating the tip attachment 43c are assigned to one of the above multiple operations. The operation assignments may be changeable. The input device 11 may also be operated to control the lower work section. For example, the input device 11 may have an operating device for controlling the clamp arm as a lower working unit. The operating device may be a device for raising and lowering the clamp arm relative to the lower traveling body 41a. The operating device may be a device for opening and closing the clamp arm relative to the lower traveling body 41a. For example, the operating device may be an operating pedal provided separately from the travel pedal. For example, the operating device may have an operating pedal for raising and lowering the clamp arm and an operating pedal for opening and closing the clamp arm.

[0028] As shown in Figure 1, the output device 12 outputs in response to commands input from the computer 13. The output device 12 includes a display unit 12a and a notification unit 12b.

[0029] As shown in Figure 3, the display unit 12a is a device for displaying information to the operator. The display unit 12a displays calculation results from the computer 13 (see Figure 1), etc. The display unit 12a displays the virtual space calculated by the computer 13. The display unit 12a is used to display the virtual space in which the work machine 4 is reproduced. The display unit 12a may also be used to explain how to operate the work machine 4. The following "displays" are displays that the computer 13 causes the display unit 12a to perform.

[0030] The display unit 12a includes a display unit 12a1. The display unit 12a1 is located in front of the seat 10a. In this embodiment, the display unit 12a1 has a standalone display, but is not limited to that. The display unit 12a1 may also have a touch panel.

[0031] The notification unit 12b (see Figure 1) is a device for notifying the operator. The output from the notification unit 12b is, for example, one or more of either an audio output or a vibration output. The notification unit 12b provides, for example, feedback on the operator's actions. The notification unit 12b notifies, for example, of the operations the operator should input. The notification unit 12b is, for example, a speaker. The output from the notification unit 12b may be displayed on the display unit 12a.

[0032] The seat 10a may be a motion seat that operates to provide feedback to the operator. The operation of the seat 10a for feedback is one or more of the following: a change in the tilt angle of the seating area, and vibration of the seating area. The seat 10a provides feedback to the operator by reproducing movements corresponding to the operation of the work machine 4 when it is operating in the virtual space. For example, the seat 10a receives a signal indicating the tilt of the work machine 4 and is controlled to achieve the same tilt. The seat 10a transmits feedback to the operator operating the input device 11 through vibration.

[0033] Computer 13 (see Figure 1) is a computer that performs signal input / output, calculations, and information storage. Each function of computer 13 is realized by the execution of a program stored in the storage unit 13a by the calculation unit 13b. Examples of the functions of computer 13 include the object processing unit 13b1, the gaze area determination unit 13b2, the sub-image state determination unit 13b3, the output control unit 13b4, the operation determination unit 13b5, and the remote control unit 13b6, which will be described later. Computer 13 and other devices (for example, the input device 11 and the output device 12) may be connected by wireless communication or by wired communication. Information is input to computer 13 from the input device 11. Computer 13 outputs information to the output device 12. Computer 13 may exchange information with the server device 20. The exchange between computer 13 and the server device 20 is carried out by communication means such as a mobile phone line, an optical line, a wireless LAN (Local Area Network), or a wired LAN.

[0034] As shown in Figure 2, the computer 13 reproduces the work machine 4 in a virtual space. The work machine 4 in the virtual space may be a reproduction of the entire actual work machine 40, or a reproduction of a part of the actual work machine 40. The computer 13 also reproduces the surrounding environment in the virtual space. In this specification, objects reproduced in the virtual space are collectively referred to as "objects." Objects include objects of the work machine 4 reproduced in the virtual space, and objects of the surrounding environment reproduced around the work machine 4 in the virtual space. Objects of the surrounding environment may include, for example, people, the ground, work objects, obstacles, transport vehicles, walls, slope shoulders, embankments, buildings, roads, and objects on roads such as guardrails.

[0035] The computer 13 comprises a storage unit 13a and an arithmetic unit 13b.

[0036] The storage unit 13a (see Figure 1) stores various types of information. The storage unit 13a stores the program for the display system 1. For example, the storage unit 13a stores information for determining the gaze area A1 (see Figure 4) in the main video MP. For example, the storage unit 13a stores information for determining the state of the secondary video SP (see Figure 4). For example, the storage unit 13a stores authentication information for login. The authentication information may also be stored in the storage unit 13a of the server device 20 (see Figure 1).

[0037] The arithmetic unit 13b (see Figure 1) performs various calculations and judgments. The arithmetic unit 13b comprises an object processing unit 13b1, a gaze area determination unit 13b2, a sub-image state determination unit 13b3, an output control unit 13b4, an operation determination unit 13b5, and a remote control unit 13b6. The object processing unit 13b1 processes various objects reproduced in the virtual space. The gaze area determination unit 13b2 determines the gaze area A1 in the main image MP (see Figure 4). The sub-image state determination unit 13b3 determines the state of the sub-image SP in the main image MP. As will be described in detail later, the state of the sub-image SP refers to the position, range, shape, etc. of the sub-image SP. The output control unit 13b4 determines the content to be output to the output device 12. The operation determination unit 13b5 determines the operation received by the input device 11. The remote control unit 13b6 controls the operation of the work machine 40 (see Figure 1) when the input device 11 is used to remotely control the work machine 40. The calculation unit 13b may have an authentication processing unit (not shown). The authentication processing unit performs a process to authenticate the user's login. The authentication processing unit identifies the user using user identification information. The user identification information may be stored in the storage unit 13a in association with a report. The user identification information may be, for example, a username and a user ID. The authentication process may be performed by the calculation unit 13b of the server device 20.

[0038] The server device 20 (see Figure 1) is a computer that operates in response to commands from the client device 19. The client device 19 consists of the operator device 10 (input device 11 and output device 12) and the computer 13. The server device 20 may store authentication information used by the arithmetic unit 13b to process user authentication. The server device 20 may also perform authentication processing to authenticate the user. The functions of the display system 1, such as the object processing unit 13b1, the gaze area determination unit 13b2, the sub-image state determination unit 13b3, the output control unit 13b4, the operation determination unit 13b5, and the remote control unit 13b6, may be realized by a combination of the client device 19 and the server device 20. At least one of the storage unit 13a and the arithmetic unit 13b in the computer 13 may be provided in the client device 19 and the server device 20, respectively. There may be one or more server devices 20. Note that the display system 1 does not need to include a server device 20. The input device 11 may be provided on the server device 20 (not shown). In this case, for example, various information such as setting information for automatically setting the gaze area A1 of the main video MP may be input from the input device 11 provided on the server device 20 to the computer 13 provided on the client device 19.

[0039] The actual work machine 40 (see Figure 1) is a real machine configured in the same way as the work machine 4 simulated by the display system 1. The display system 1 may include a remote control device for remotely controlling the actual work machine 40. The display system 1 may also be implemented in the remote control device. The remote control device includes an input device 11, a computer 13, and an output device 12. The input device 11 includes, for example, an operating device 11a including an operating lever and an operating pedal, and outputs an operation signal (for example, a lever operation signal or a pedal operation signal) in response to the operation of the operating device 11a. When remotely controlling, the computer 13 exchanges signals with the actual machine that is the target of remote control via a communication means. When remotely controlling, the computer 13 transmits signals (operation command signals) to the actual work machine 40 to operate the upper work section (corresponding to the attachment 43) and the traveling device (corresponding to the lower traveling body 41a), etc., based on the operation signals input from the input device 11. During remote operation, the computer 13 receives image data from a camera mounted on the actual work machine 40, as well as signals output by sensors mounted on the actual work machine 40, via communication means. The display unit 12a1 of the output device 12 outputs image data from the camera mounted on the actual work machine 40 (data acquired by the computer 13) during remote operation.

[0040] Since the display system 1 also incorporates a remote control device, the user can operate the work machine 4 in the virtual space using the remote control device used to remotely operate the actual work machine 40. Therefore, by operating the work machine 4 in the virtual space, the user can experience the feel of operating the actual work machine 40. As a result, for example, the user's proficiency in operating the remote control device is promoted. Also, for example, it becomes easy to anticipate what kind of situation will occur when the actual work machine 40 at the work site is operated with the remote control device.

[0041] The display system 1 (mainly the computer 13) is configured to perform the following processing related to simulation and / or remote control. The simulation program causes the computer 13 to execute the following processing. The display system 1 implements a method in which the following processing is performed.

[0042] The object processing unit 13b1 of the computer 13 has an object processing function that processes objects. As described above, the objects are models of the work machine 4 and the surrounding environment. The objects may be three-dimensional models or two-dimensional models. Each function of the display system 1 is equivalent to a step in the display program and display method. For example, the object processing function that processes objects is equivalent to an object processing step. The object processing function and object processing step include the process of displaying the objects on the display unit 12a1.

[0043] Computer 13 operates either the work machine 4 or the surrounding environment, or both. The work machine 4 may operate in response to operator input or automatically. For example, the work machine 4 may be operated by a machine control system (MC). Specifically, a work plan is set in computer 13. The operator then operates, for example, only some elements of the attachment 43 (see Figure 2) (for example, only the boom 43a). In this case, computer 13 automatically controls the elements not operated by the operator (for example, the arm 43b and the end attachment 43c) so that the work machine 4 works according to the work plan. In this case, computer 13 controls the operation of the work machine 4 based on its posture (the same applies in the case of automatic operation). As a result, the work machine 4 works according to the work plan. Alternatively, for example, the work machine 4 may be operated by automatic operation. In this case, computer 13 controls the operation of the work machine 4 so that the work machine 4 works automatically according to the work plan.

[0044] The computer 13 operates the object according to the input received by the operation device 11a (see FIG. 3). For example, the computer 13 may operate the lower traveling body 41a (see FIG. 2) according to the operation of the traveling lever 11a2 (see FIG. 3) to make the work machine 4 travel. The computer 13 may, for example, turn the upper slewing body 41b (see FIG. 2) according to the left-right direction operation of the left operation lever 11a1A (see FIG. 3). The computer 13 may, for example, rotate the arm 43b according to the front-rear direction operation of the left operation lever 11a1A. The computer 13 may, for example, rotate the boom 43a (see FIG. 2) according to the front-rear direction operation of the right operation lever 11a1B (see FIG. 3). The computer 13 may, for example, rotate the tip attachment 43c (see FIG. 2) according to the left-right direction operation of the right operation lever 11a1B.

[0045] For example, the computer 13 changes the operating speed of the operation target according to the operation amount of the input device 11. For example, the computer 13 changes the operating speed of the attachment 43 according to the operation amount of the attachment operation lever 11a1. For example, the computer 13 changes the operating speed of the work machine 4 according to the set engine speed. The computer 13, for example, changes the operating speed of the basic attachment 43 and the traveling speed of the work machine 4 according to the engine speed.

[0046] (Display content of the display unit) As shown in Figure 4, the work machine 4 operated by the input device 11 and objects of the surrounding environment (e.g., the work object) are displayed on the display unit 12a (specifically, the display unit 12a1). The objects representing the surrounding environment are, for example, one or more of the following: people, the ground, the work object, obstacles, transport vehicles, walls, and slope shoulders. The main image MP representing the work machine 4 and the surrounding environment is displayed across the entire surface of the display unit 12a. In addition, a secondary image SP is displayed on the display unit 12a superimposed on the main image MP. There may be one or more secondary images SP. The secondary images SP are displayed based on the state determined by the secondary image state determination unit 13b3 of the computer 13. The state of the secondary image SP may be information indicating the position and range of the secondary image SP in the main image MP. The state of the secondary image SP may be information indicating the shape. The state of the secondary image SP may also be information indicating whether it is displayed or not (presence or absence of the secondary image). In the example in Figure 4, there is only one screen, but this is not limited to one; there may be multiple screens. Also, for example, in the example in Figure 4, there are sub-video SPs with different widths, but all sub-video SPs may have the same width.

[0047] (Regarding the main image) The position of the viewpoint (starting point of the line of sight) of the main image MP is not particularly limited. For example, the viewpoint of the main image MP may be inside the operator's cab 41b2 of the work machine 4. In this case, the viewpoint of the main image MP may be set above the operator's cab 41b2, corresponding to the eye position of the operator seated in the seat of the operator's cab 41b2, or it may be set at another position inside the operator's cab 41b2. Also, the viewpoint of the main image MP may be outside the operator's cab 41b2. In this case, the viewpoint of the main image MP can be set at an appropriate position outside the operator's cab 41b2, such as above the operator's cab 41b2, to the side of the operator's cab 41b2, or behind the operator's cab 41b2. The viewpoint of the main image MP may be moved. The movement of the viewpoint of the main image MP may be performed automatically, or it may be performed in response to an operation received by the input device 11.

[0048] The position of the fixation point (the end point of the line of sight) of the main image MP is not particularly limited. For example, the fixation point of the main image MP may be set on the working machine 4 or a specific part of the working machine 4 (e.g., the attachment 43), may be set on the work target of the working machine 4, or may be set in the working range of the working machine 4. For example, the main image MP may be a top view image with the upper part of the working machine 4 as the viewpoint and the working machine 4 as the fixation point. The main image MP may be a side view image with the side of the working machine 4 as the viewpoint and the working machine 4 as the fixation point. The main image MP may be a perspective view image with the diagonally upper part of the working machine 4 as the viewpoint and the working machine 4 as the fixation point. When remotely operating the actual working machine 40, the main image MP is an image acquired by an imaging device (not shown) provided on the actual working machine 40. When overlaying the sub-image, the main image outside the range of the fixation area may be displayed as it is, may not be displayed by trimming the image, or may be filled with a single color. Also, the display unit shape of the main image and the sub-image is not limited to a rectangle, and may be a polygon, a circle, an ellipse, etc.

[0049] The fixation point of the main image MP may be changed. The change of the fixation point of the main image MP may be performed automatically or may be performed according to an operation received by the input device 11. For example, the fixation point of the main image MP may be changed according to the vertical movement of the cab 41b2, which is an elevator cab, with respect to the bottom of the upper swing body 41b. The fixation point of the main image MP may be changed according to the vertical and longitudinal movement of the cab 41b2, which is a link cab, with respect to the bottom of the upper swing body 41b due to the operation of the link mechanism. The fixation point of the main image MP may be changed according to the vertical rotation of the cab 41b2, which is a tilt cab, about a rotation axis extending in the horizontal direction with respect to the bottom of the upper swing body 41b. The fixation point of the main image MP may be changed according to the rotation of the upper swing body 41b. The fixation point of the main image MP may be changed according to the travel of the lower traveling body 41a. Also, the main image MP may be enlarged or reduced.

[0050] (Regarding the secondary image) Similar to the main image MP, the position of the viewpoint (starting point of the line of sight) of the secondary image SP is not particularly limited. The example of setting the viewpoint for the main image MP described above can also be applied to the viewpoint of the secondary image SP. The viewpoint of the secondary image SP may be moved. The movement of the viewpoint of the secondary image SP may be performed automatically, or it may be performed in response to an operation received by the input device 11.

[0051] Similar to the main video MP, the position of the point of fixation (endpoint of the line of sight) of the secondary video SP is not particularly limited. The example of setting the point of fixation for the main video MP described above can also be applied to the point of fixation for the secondary video SP. The point of fixation for the secondary video SP may be changed. The point of fixation for the secondary video SP may be changed automatically or in response to an operation received by the input device 11. In addition, the secondary video SP may be displayed enlarged or reduced. When the actual work machine 40 is remotely operated, the secondary video SP is an image acquired by an imaging device (not shown) provided on the actual work machine 40, separate from the imaging device that captures the main video MP.

[0052] For example, in the example shown in Figure 4, six sub-images SP, consisting of the first sub-image SP1 to the sixth sub-image SP6, are superimposed on the main image MP. Specifically, the first sub-image SP1 is an image with the viewpoint to the left of the work machine 4 and the point of focus in front of the work machine 4. The first sub-image SP1 also includes a transport vehicle as part of the surrounding environment. The second sub-image SP2 is a top view of the transport vehicle. The third sub-image SP3 is an oblique view including the transport vehicle and the work machine 4. The fourth sub-image SP4 is an image with the viewpoint to the right of the work machine 4 and the point of focus in front of the work machine 4. The fourth sub-image SP4 also includes the work object as part of the surrounding environment. The fifth sub-image SP5 is an oblique view including the work object and the work machine 4. The sixth sub-image SP6 is a side view of the work machine 4. Thus, in this embodiment, the computer 13 displays an image of the work machine 4 as seen from outside the machine on the display unit 12a as a secondary image SP. The secondary image SP may include one or both of the work object and the work machine 4.

[0053] The secondary image SP may not include either the work object or the work machine 4. For example, as shown in Figure 5, the secondary image SP may be an image showing the input status of the input device 11. Specifically, the secondary image SP may be an image showing the operating status of the operating device 11a. The secondary image SP may display the operating status of one or both of the attachment operating lever 11a1 and the travel lever 11a2. The secondary image SP may display the operating status of both the left operating lever 11a1A and the right operating lever 11a1B. In the example in Figure 5, the secondary image SP is a gauge schematically showing the operating status of the left operating lever 11a1A and the right operating lever 11a1B. The gauge includes a white arrow indicating the direction of operation and a black dot indicating the current direction of operation.

[0054] The display position of the secondary image SP, that is, the display position of the secondary image SP relative to the gaze area A1 of the main image MP, may be determined based on the positional relationship between the viewpoint of the main image MP and the viewpoint of the secondary image SP. Specifically, an example of setting the display position of the secondary image SP1 is shown with reference to Figure 4. As described above, the first secondary image SP1 is an image viewed from a viewpoint set to the left side of the work machine 4. In this case, the viewpoint of the first secondary image SP1 is located to the left of the viewpoint of the main image MP located in the operator's cab 41b2 in the virtual space, looking in the direction of the gaze point of the main image MP (forward). Therefore, the display position of the first secondary image SP1 is placed to the left of the gaze area A1.

[0055] Although not shown in the diagram, for example, if the point of focus of the main video MP shifts to the left in the example of Figure 4, and the main video MP is changed to an image that looks to the left of the work machine 4, the secondary video SP, which is positioned to the right of the main video MP, may be changed to an image related to the front of the work machine 4. Similarly, in the same case, the secondary video SP, which is positioned to the left of the main video MP, may be changed to an image related to the rear of the work machine 4. Thus, the display position of the secondary video SP may be set in accordance with the positional relationship between the viewpoint of the main video MP and the viewpoint of the secondary video SP. The content of the secondary video SP may also be information such as operating procedures, work instructions, and machine tilt. The secondary video SP may also be displayed in a position corresponding to the driver's seat monitor control unit (cluster) or ICT construction monitor of the actual work machine 40. The secondary video SP may also be displayed while avoiding the mirror (not shown) of the work machine 4.

[0056] (Regarding the gaze area) As described above, the main video MP has a gaze area A1 which includes the point of focus. The gaze area A1 is set to determine the area in which the secondary video SP is displayed. That is, by setting the gaze area A1 within the main video MP, the display area of ​​the secondary video SP is determined to be outside the range of the set gaze area A1. The range of the gaze area A1 is determined by the gaze area determination unit 13b2 of the computer 13. Thus, in this embodiment, the gaze area A1 is determined by the computer 13, but it may be set in advance. The gaze area A1 may be set arbitrarily. The computer 13 may arbitrarily set the gaze area A1 in response to an operation received by the input device 11. The computer 13 may periodically determine (update) the gaze area A1. The computer 13 may determine the gaze area A1 for each event. An event that determines the gaze area A1 is, for example, the operation of the work machine 4. More specifically, the events that determine the gaze area A1 are one or more of the following: the movement of the lower traveling body 41a, the rotation of the upper rotating body 41b, and a change in the posture of the attachment 43. Alternatively, the events that determine the gaze area A1 may also be the input of an operation to operate the work machine 4 to the input device 11. The gaze area A1 may be determined based on the gaze point of the main image MP. The gaze area A1 may be a specific area fixed on the display unit 12a. The gaze area A1 may be determined according to the content of the main image MP. For example, as shown in Figure 4, the gaze area A1 may be the range of the main image MP viewed from a viewpoint inside the operator's cab 41b2 of the work machine 4, displayed through the front window 41b2A of the operator's cab 41b2. The gaze area A1 may be determined arbitrarily. The gaze area A1 may be determined based on the operation of the attachment 43. The gaze area A1 may be determined according to the operation of the work machine 4 in the main image MP. More specifically, the gaze area A1 may be enlarged according to the display range of the attachment 43 of the work machine 4.

[0057] As shown in Figure 6, the gaze region A1 may be determined to overlap with the operable range of the attachment 43 of the work machine 4 in the main image MP, which is the range in which the attachment 43 of the work machine 4 can operate. Note that the overlap referred to here is a concept that includes partial overlap, where only a part, not the entire operable range, overlaps with the gaze region A1 (the same applies hereinafter). In the example of Figure 6, the viewpoint of the main image MP is inside the driver's cab 41b2. On the other hand, the attachment 43 is located to the right of the driver's cab 41b2. From this, at least a part of the operable range of the attachment 43 is included in the right part of the main image MP. Therefore, the gaze region A1 is set in the right part of the main image MP. As a result, each sub-image SP is set outside the range of the gaze region A1 in the main image MP, that is, to the left of the gaze region A1. In the example of Figure 6, the sub-image SP is placed in the left part of the main image MP, superimposed on the main image MP.

[0058] Conversely, if the viewpoint of the main image MP is on the opposite side of the attachment 43 from the driver's cab 41b2, that is, to the right of the driver's cab 41b2, then at least a portion of the operable range of the attachment 43 is included in the left portion of the main image MP. Therefore, the gaze area A1 is set to the left portion of the main image MP. As a result, the secondary image SP is positioned superimposed on the main image MP in the right portion of the main image MP, which is outside the gaze area A1.

[0059] As described above, the gaze area A1 is set to overlap with the range in which the upper work unit, such as the attachment 43, can operate in the main image MP, but is not limited to this. For example, although not shown, the gaze area A1 may overlap with the range in which the lower work unit, such as the attachment provided on the lower traveling body 41a, can operate. Also, the gaze area A1 may overlap with the range in which the lower traveling body 41a can travel in the main image MP. Furthermore, the gaze area A1 may overlap with the range in which the upper rotating body 41b can rotate in the main image MP.

[0060] (Changes in the state of the secondary image) The state of the secondary image SP may change in accordance with the deformation of the gaze area A1. The state of the secondary image SP is a concept that includes one or more of the following: the position of the secondary image SP in the main image MP, the range of the secondary image SP, the shape of the secondary image SP, and the presence or absence of the secondary image SP. The presence or absence of the secondary image SP indicates whether or not the secondary image SP is displayed.

[0061] The state of the secondary image SP may change according to the state of the main image MP. The state of the main image MP is a concept that includes one or more of the following: the position of the viewpoint of the main image MP, the direction of the point of fixation of the main image MP (the direction from the viewpoint toward the point of fixation), and the state of scaling of the main image MP. For example, Figure 7 shows a state in which the viewpoint of the main image MP is moved downward from the state in Figure 4, while the direction of the point of fixation is tilted upward, that is, the viewpoint and point of fixation are changed so that the viewer is looking diagonally upward from a relatively low position inside the driver's cab 41b2. As the viewpoint of the main image MP is moved, the fixation area changes from fixation area A1 (Figure 4) to fixation area A2 (Figure 7). As shown in Figure 7, the fixation area A2 after the change is roughly trapezoidal. In other words, in the main image MP, the area outside the fixation area A2 is wider at the top and narrower at the bottom. Therefore, the layout of the secondary image SP is changed to match the modified fixation area A4. For example, on the left side of the gaze area A2, the vertically oriented first secondary image SP1 moves below the horizontally oriented second and third secondary images SP2 and SP3. Similarly, on the left side of the gaze area A2, the vertically oriented fourth secondary image SP4 moves below the horizontally oriented fifth and sixth secondary images SP5 and SP6.

[0062] Figures 8A to 8C show the changes in the gaze region A3 and secondary image SP that can occur when the gaze point direction of the main image MP is tilted vertically. Specifically, in the example in Figure 8A, the gaze point direction is set so that the gaze region A3a is located in the center of the main image MP. In this case, the secondary image SP displays the seventh secondary image SP7, which is located above the gaze region A3a, and the eighth secondary image SP8, which is located below the gaze region A3a. In contrast, in the example in Figure 8B, the gaze point direction of the main image MP is tilted upward compared to the example in Figure 8A. As a result of this change in the gaze point, the gaze region in the main image MP changes from gaze region A3a (Figure 8A) to gaze region A3b (Figure 8B). The changed gaze region A3b is located above the gaze region A3a before the change. As a result of this change from gaze region A3a to gaze region A3b, the seventh gaze region SP7 moves from a position above gaze region A3a (Figure 8A) to a position below gaze region A3b (Figure 8B). Also, in the example of Figure 8C, the direction of the gaze point of the main image MP is tilted downwards compared to the example of Figure 8A. As a result of this change in the gaze point, the gaze region in the main image MP changes from gaze region A3a (Figure 8A) to gaze region A3c (Figure 8C). The gaze region A3c after the change is located below the gaze region A3a before the change. As a result of this change from gaze region A3a to gaze region A3c, the eighth secondary image SP8 moves from a position below gaze region A3a (Figure 8A) to a position above gaze region A3c. In this way, the computer 13 moves the gaze area A3 according to the direction of the gaze point of the main image MP, and also changes the state of the secondary image SP. This improves the visibility of the area that the operator wants to see in the main image MP.

[0063] The state of the secondary image SP may change in accordance with the operation of the work machine 4. For example, the operation of the work machine 4 that changes the state of the secondary image SP may be the operation of the work machine 4 in the main image MP. More specifically, the operation of the work machine 4 that changes the state of the secondary image SP may be the operation of the attachment 43 of the work machine 4 in the main image MP. For example, in the example of Figure 9, compared to the example of Figure 4, the attachment 43 of the work machine 4 is activated and moves closer to the operator's cab 41b2. As a result, the gaze area in the main image MP changes from gaze area A1 (Figure 4) to gaze area A4 (Figure 9), which has a wider upper width to the right than gaze area A1. As a result, the fourth to sixth secondary images SP4 to SP6, which were displayed in the right part of the main image MP (to the right of gaze area A1), are hidden. In this way, the state of the secondary image SP may change as a result of the operation of the work machine 4. The secondary image SP may be hidden when displaying other information (such as explanatory text).

[0064] Furthermore, the state of the secondary image SP may change in response to the input device 11 receiving an operation to operate the work machine 4. For example, an operation that changes the state of the secondary image SP may be one or more of the following: a rotation operation of the work machine 4, a travel operation of the work machine 4, and a movement or rotation operation of the driver's cab 41b2. An operation that changes the state of the secondary image SP may also be an operation that changes the imaging direction of the imaging device (for example, an imaging device for the main image MP) mounted on the actual work machine 40. For example, in the example in Figure 10, the upper rotating body 41b of the work machine 4 is rotated to the left compared to the example in Figure 4. As a result of the input device 11 receiving an operation to rotate the upper rotating body 41b to the left, the gaze area in the main image MP changes from gaze area A1 (Figure 4) to gaze area A5 (Figure 10), which is biased to the right of gaze area A1. Furthermore, the fourth to sixth sub-images SP4 to SP6, which were positioned on the right side of the main image MP in Figure 4 before rotation, move to the left side of the main image MP after rotation as shown in Figure 10, avoiding the gaze area A5 which has shifted to the right. Thus, the state of the sub-images SP may change as a result of the operation to operate the work machine 4. The amount of movement of the gaze area A5 may be determined based on the amount of operation to the input device 11. In addition, the direction of movement of the gaze area A5 may be the direction in which the upper rotating body 41b rotates, or it may be the direction in which the lower traveling body 41a travels.

[0065] (Summary) [Configuration 1] The display system 1 comprises a computer 13 and a display unit 12a. The display unit 12a displays an image of the work machine 4 and / or the surrounding environment (e.g., transport vehicles and work objects, etc.) as the main image MP. The computer 13 displays an image of the work machine 4 and / or the surrounding environment from a different viewpoint than the main image MP as a secondary image SP on the display unit 12a, outside the range of the gaze area A1 of the main image MP.

[0066] In the above configuration 1, the image of the work machine 4 and / or the surrounding environment is displayed as the main image MP, while the image of the work machine 4 and / or the surrounding environment viewed from a different viewpoint than the main image MP is displayed as a secondary image SP outside the range of the gaze area A1 determined within the main image MP. In this way, since the secondary image SP is not superimposed on the gaze area A1 in the main image MP, good visibility within the gaze area A1 in the main image MP can be ensured, while increasing the amount of information that can be displayed by displaying the secondary image SP outside the gaze area A1.

[0067] [Configuration 2] As shown in Figure 4, in the display system 1, the viewpoint of the main video MP is set inside the operator's cab 41b2 of the work machine 4.

[0068] In the above configuration 2, the image viewed from inside the driver's cab 41b2 can be used as the main image MP.

[0069] [Configuration 3] As shown in Figure 4, in the display system 1, the gaze area A1 is set to the area of ​​the front window 41b2A of the driver's cab 41b2 within the main video MP.

[0070] In the above configuration 3, the area visible from inside the driver's cab 41b2 through the front window 41b2A is defined as the gaze area A1, and the secondary image SP can be displayed outside this area. Therefore, the reduction in the visibility of the image within the range of the front window 41b2A can be reduced.

[0071] [Configuration 4] In the display system 1, the work machine 4 has an upper work section (e.g., an attachment 43) and / or a lower work section (e.g., a clamp arm provided on the lower traveling body 41a), and the gaze area A1 overlaps with the range in which the upper work section or the lower work section can operate in the main image MP.

[0072] In the above configuration 4, the gaze area A1 overlaps with the range in which the upper or lower work unit in the main video MP can operate, thus reducing the decrease in visibility of the video of the work of the work machine 4.

[0073] [Configuration 5] As shown in Figure 6, the work machine 4 has an attachment 43 for performing work, and the gaze area A1 overlaps with the range in which the attachment 43 can operate in the main image MP.

[0074] In the above configuration 5, the gaze area A1 overlaps with the operating range of the attachment 43 in the main video MP, thus reducing the decrease in visibility of the video of the attachment 43 of the work machine 4.

[0075] [Configuration 6] As shown in Figure 4, in the display system 1, the computer 13 displays the image of the work machine 4 as seen from outside the work machine 4 as a sub-image SP (for example, the third, fifth, and sixth sub-images SP3, SP5, SP6) on the display unit 12a.

[0076] In the above configuration 6, the main video MP displays the image of the work machine 4 as seen from the outside as the secondary video SP. This allows the viewer to check the status of the work machine 4 as seen from the outside simultaneously with the gaze area A1 in the main video MP.

[0077] [Configuration 7] In the display system 1, the work machine 4 has a lower traveling body 41a as a traveling device, and the gaze area A1 overlaps with the range in which the lower traveling body 41a can travel in the main image MP.

[0078] In the above configuration 7, the gaze area A1 overlaps with the range in which the lower traveling body 41a can travel in the main video MP, thus reducing the decrease in visibility of the video of the work of the work machine 4.

[0079] [Configuration 8] In the display system 1, the work machine 4 has a rotatable upper rotating body, and the gaze area A1 overlaps with the range in which the upper rotating body 41b can rotate in the main image MP.

[0080] In the above configuration 8, the gaze area A1 overlaps with the range in which the upper rotating body 41b in the main image MP can rotate, thus reducing the decrease in visibility of the work of the work machine 4.

[0081] [Configuration 9] As shown in Figure 4, in the display system 1, the computer 13 sets the display position of the secondary image SP relative to the gaze area A1 in the main image MP to a position corresponding to the positional relationship between the viewpoint of the main image MP and the viewpoint of the secondary image SP.

[0082] In the above configuration 9, the secondary video SP can be displayed at a position corresponding to the positional relationship of each viewpoint of the main video MP and the secondary video SP.

[0083] [Configuration 10] The display system 1 is equipped with an input device 11 that accepts operations from an operator, and the computer 13 sets the gaze area A1 according to the operations accepted by the input device 11.

[0084] In the above configuration 10, the gaze area A1 is set according to the operation performed by the operator, so the operator can include the area in which they want to ensure visibility in the gaze area A1.

[0085] [Configuration 11] As shown in Figures 7, 8A, 8B, and 8C, in the display system 1, the computer 13 changes one or more of the following in relation to the main image MP: the position of the viewpoint of the main image MP, the direction of the point of focus of the main image MP, and the state of scaling of the main image MP: the position of the sub-image SP in the main image MP, the range of the sub-image SP, the shape of the sub-image SP, and the presence or absence of the sub-image SP.

[0086] In the above configuration 11, the state of the secondary video SP can be changed in accordance with the change in the state of the main video MP.

[0087] [Configuration 12] As shown in the example in Figure 9, in the display system 1, the computer 13 changes one or more of the following in response to the operation of the work machine 4: the position of the secondary image SP in the main image MP, the range of the secondary image SP, the shape of the secondary image SP, and the presence or absence of the secondary image SP.

[0088] In the above configuration 12, the state of the secondary image SP can be changed in accordance with the operation of the work machine 4.

[0089] [Configuration 13] As shown in the example in Figure 10, the display system 1 is equipped with an input device 11 that accepts operations from an operator, and the computer 13 changes one or more of the following in response to the input device 11 receiving an operation to operate the work machine 4: the position of the sub-image SP in the main image MP, the range of the sub-image SP, the shape of the sub-image SP, and the presence or absence of the sub-image SP.

[0090] In the above configuration 13, the state of the secondary video SP can be changed in accordance with the operation of operating the work machine 4.

[0091] [Configuration 14] As shown in Figure 3, the display system 1 includes an input device 11 that receives operations from an operator to operate a work machine 4 in a virtual space. The input device 11 is used to remotely operate the actual work machine 40, which is an existing work machine 4.

[0092] In the above configuration 14, the input device 11 for operating the work machine 4 in the virtual space can also be used as a device for remotely controlling the actual work machine 40. Therefore, the operator can remotely control the actual work machine 40 without changing the feel of operation.

[0093] (Modifications) The above embodiments may be modified in various ways. For example, various examples (including modifications) of the above embodiments may be combined in various ways. For example, the connections of each component shown in Figure 1, etc., may be changed. For example, the number of components in the above embodiments (including modifications) may be changed, and some components may be omitted. This "component" includes components (specifically, objects) in the virtual space (the same applies hereinafter). For example, the arrangement of components may be changed. For example, the inclusion relationships of components may be changed in various ways. For example, something described as a subordinate component included in a higher-level component may not be included in this higher-level component, but may be included in other components. For example, something described as multiple different members or parts may be treated as a single member or part. For example, something described as a single member or part may be divided and provided as multiple different members or parts.

Claims

1. A display system comprising a computer and a display unit that displays images of a work machine and / or the surrounding environment as a main image, wherein the computer displays images of the work machine and / or the surrounding environment from a viewpoint different from the main image as a secondary image on the display unit, outside the viewing area of ​​the main image.

2. A display system according to claim 1, wherein the viewpoint of the main image is set inside the operator's cabin of the work machine.

3. A display system according to claim 3, wherein the viewing area is set to the area of ​​the front window of the driver's cab within the main image.

4. A display system according to claim 1, wherein the work machine has an upper work section and / or a lower work section, and the viewing area overlaps with the range in which the upper work section or the lower work section can be operated in the main image.

5. A display system according to claim 1, wherein the work machine has an attachment for performing work, and the viewing area overlaps with the range in which the attachment can operate in the main image.

6. A display system according to claim 1, wherein the computer displays an image of the work machine as seen from outside the work machine as the sub-image on the display unit.

7. A display system according to claim 6, wherein the work machine has a traveling device, and the viewing area overlaps with the range in which the traveling device can travel in the main image.

8. A display system according to claim 6, wherein the work machine has a rotatable upper rotating body, and the viewing area overlaps with the range in which the upper rotating body can rotate in the main image.

9. A display system according to claim 1, wherein the computer sets the display position of the sub-image relative to the gaze area in the main image to a position corresponding to the positional relationship between the viewpoint of the main image and the viewpoint of the sub-image.

10. A display system according to claim 1, comprising an input device for receiving operations by an operator, wherein the computer sets the gaze area in accordance with the operations received by the input device.

11. A display system according to claim 1, wherein the computer changes one or more of the following in the main image: the position of the viewpoint of the main image, the direction of the point of focus of the main image, and the state of scaling of the main image: the position of the sub-image in the main image, the range of the sub-image, the shape of the sub-image, and the presence or absence of the sub-image.

12. A display system according to claim 1, wherein the computer changes one or more of the following in accordance with the operation of the work machine: the position of the secondary image in the main image, the range of the secondary image, the shape of the secondary image, and the presence or absence of the secondary image.

13. A display system according to claim 1, comprising an input device for receiving operations by an operator, wherein the computer changes one or more of the following in response to the input device receiving an operation to operate the work machine: the position of the secondary image in the main image, the range of the secondary image, the shape of the secondary image, and the presence or absence of the secondary image.

14. A display system according to claim 1, comprising an input device that receives an operation by an operator to operate the work machine in the virtual space, wherein the input device is used to remotely operate an actual work machine which is an existing work machine.

Images