Display system

Abstract

This invention provides a display system that can mitigate the reduction in visibility of the main image when displaying images from multiple viewpoints. [Solution] The display system 1 comprises a computer 13 and a display unit 12a. The display unit 12a displays the work machine 4 and / or the surrounding environment (e.g., transport vehicles and work objects, etc.) as the main image MP. The computer 13 displays the work machine 4 and / or the surrounding environment from a different viewpoint than the main image MP as a secondary image SP on the display unit 12a in an area outside the gaze area A1 of the main image MP.

Description

Technical Field

[0006] , , ,

[0005] , , , ,

[0001] The present invention relates to a display system that simulates a working machine in a virtual space.

Background Art

[0002] For example, Patent Document 1 describes a remote operation support device used for simulating remote operation of a working machine.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the prior art as described above, for example, it was common to display one viewpoint from inside the cab or the like as the main image on one or more displays. Therefore, in order to display an image from another viewpoint, such as outside the cab, as a sub-image, it was necessary to increase the display or display the main image smaller. Therefore, there was a risk that the visibility of the main image would be reduced.

[0005] Therefore, an object of the present invention is to provide a display system capable of reducing a decrease in visibility of a main image when displaying images from a plurality of viewpoints.

Means for Solving the Problems

[0006] The display system includes a computer and a display unit. The display unit displays a working machine and / or the surrounding environment as the main image. The computer displays the working machine and / or the surrounding environment from a viewpoint different from the main image as a sub-image on the display unit in an area outside the range of the fixation area in the main image.

Effects of the Invention

[0007] The above display system can mitigate the reduction in visibility of the main image when displaying images from multiple viewpoints. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram of display system 1. [Figure 2] This figure shows the work machine 4 displayed on the display system 1 shown in Figure 1. [Figure 3] This figure shows a specific example of the operator device 10 of the display system 1 shown in Figure 1. [Figure 4] Figure 3 shows specific examples of the main video MP and secondary video SP displayed in the operator's cab 41b2 from inside the operator's cab 41b2. [Figure 5] Figure 3 shows a modified example of the main video MP and sub-video SP displayed in the operator's cab 41b2 on the operator device 10. [Figure 6] Figure 2 shows specific examples of the main video MP and sub-video SP, where the operating range of the work machine 4 shown in Figure 2 is the gaze area A1. [Figure 7] This figure shows a specific example of moving the viewpoint of the main video (MP) shown in Figure 4 upwards. [Figure 8A] This diagram shows a specific example of changing the state of the secondary image SP1 according to the viewpoint direction of the main image MP. [Figure 8B] Figure 8A shows a specific example of what happens when the viewpoint of the main video (MP) is moved upwards. [Figure 8C] Figure 8A shows a specific example of what happens when the viewpoint of the main video (MP) is moved downwards. [Figure 9] This figure shows a specific example of changing the state of the secondary image SP1 in accordance with the operation of the work machine 4 shown in Figure 2. [Figure 10] This figure shows a specific example of how the state of the secondary image SP1 changes in response to an operation that changes the viewpoint of the main image MP shown in Figure 4. [Modes for carrying out the invention]

[0009] The display system 1 will be described with reference to the drawings.

[0010] As shown in Figure 1, the display system 1 is applied to a simulator that performs a simulation simulating a work machine 4 in a virtual space. The display system 1 is also applied to a remote control device that operates the actual work machine 40 at a remote location. In this embodiment, the display system 1 has a configuration that includes the simulator and remote control device described above. The following description will mainly focus on the work machine 4 in the virtual space, but it can also be applied to the actual work machine 40.

[0011] First, let's describe the work machine 4 that is simulated in the virtual space. As shown in Figure 2, the work machine 4 is a machine that performs work, for example, a construction machine that performs construction work, and may be an excavator, a crane, a bulldozer, a wheel loader, or a dump truck (transport vehicle). Below, we will mainly describe the case where the work machine 4 is an excavator. The work machine 4 may be operated. The work machine 4 may be operated based on information pre-set in the computer 13. The work machine 4 may be operated in response to manual operation by the user. The work machine 4 may travel, the upper slewing body 41b (described later) may rotate relative to the lower traveling body 41a (described later), and the posture of the attachment 43 may change. The work machine 4 may spray washer fluid onto the outer surface of the windshield (not shown). The work machine 4 may wipe away rainwater and washer fluid etc. adhering to the outer surface of the windshield (not shown). The work machine 4 may have lights (for example, headlights capable of illuminating the area in front of the work machine 4) turned on (not shown). The work machine 4 does not have to be operational (it may simply be placed in the virtual space). The work machine 4 comprises a machine body 41 and an attachment 43 (attachment object).

[0012] The machine body 41 is the main body of the work machine 4. The machine body 41 comprises a lower traveling body 41a and an upper rotating body 41b.

[0013] The lower traveling body 41a is capable of traveling on a traveling surface (such as the ground). The lower traveling body 41a may include crawlers 41a1 or may include wheels (not shown). The lower traveling body 41a may have a working part (lower working part) for performing work. The lower working part may be a lower working device such as an attachment for performing work on a work object. The lower working device includes a dozer blade for leveling the ground, a clamp arm for holding down an automobile during automobile disassembly, and outriggers.

[0014] The upper slewing body 41b is capable of slewing with respect to the lower traveling body 41a. The upper slewing body 41b may include a cab 41b2. The cab 41b2 is a part where an operator operates the machine in the actual working machine 4. The cab 41b2 may be fixed to the bottom (slewing frame) of the upper slewing body 41b. The cab 41b2 may be movable with respect to the bottom of the upper slewing body 41b. In this case, the cab 41b2 may be movable in a direction parallel to the vertical direction with respect to the bottom of the upper slewing body 41b, may be rotatable (rotationally movable) in the vertical direction, or may be movable in a direction parallel to the vertical direction and rotatable. For example, the cab 41b2 may be an elevator cab, a link cab, or a tilt cab. The cab 41b2 that is an elevator cab is movable in the vertical direction with respect to the bottom of the upper slewing body 41b. The cab 41b2 that is a link cab is installed on the bottom of the upper slewing body 41b via a link mechanism. The cab 41b2 that is a link cab is movable in the vertical direction and the front-rear direction with respect to the bottom of the upper slewing body 41b when the link mechanism operates. The cab 41b2 that is a tilt cab is rotatable (tiltable) in the vertical direction about a rotation axis extending in the lateral direction with respect to the bottom of the upper slewing body 41b. The cab 41b2 may include accessories attached to the main body of the cab 41b2. This accessory may be, for example, a guard for protecting the upper surface of the cab 41b2, the front window 41b2A (see FIG. 4), etc. from flying objects.

[0015] The attachment 43 is the part (upper working part) for performing operations. The attachment 43 is an upper working device for performing operations. The attachment 43 is capable of pitching (rotating in the vertical direction) with respect to the upper swing body 41b. The attachment 43 includes a boom 43a, an arm 43b, and a tip attachment 43c. In the example of FIG. 2, the working machine 4 has an upper working part and does not have a lower working part, but is not limited thereto. The working machine 4 may have an upper working part and / or a lower working part.

[0016] The boom 43a is rotatably attached to the upper swing body 41b. The arm 43b is rotatably attached to the boom 43a.

[0017] The tip attachment 43c is provided at the tip of the attachment 43. The tip attachment 43c is rotatably attached to the arm 43b. The tip attachment 43c may be a bucket for scooping up a work object or performing excavation. The tip attachment 43c may be a device (such as a grapple or crusher) for clamping a work object, or a device (such as a breaker) for crushing a work object, or a lifting magnet for adsorbing a metal work object. The object (work object) to be worked on by the tip attachment 43c may be earth and sand, stone, wood, metal, resin, waste, or a structure (such as a block).

[0018] Display System 1 is a system that simulates the work machine 4 and its surrounding environment in a virtual space. Display System 1 is a system that displays the work machine 4 and / or its surrounding environment simulated in the virtual space as the main image MP (see Figure 4) on the display unit 12a. Display System 1 is a system that displays the work machine 4 and / or its surrounding environment from a different viewpoint than the main image MP as the secondary image SP (see Figure 4) on the display unit 12a. In the following description, "viewpoint" means the point of gaze, indicating which direction is being viewed from a specific viewpoint. On the display unit 12a, the secondary image SP is displayed in an area outside the gaze area A1 (see Figure 4) of the main image MP. The main image MP, secondary image SP, and gaze area A1 will be described in detail later. The "operator" is a user of Display System 1 who operates the work machine 4 simulated in the virtual space. Display System 1 may include, for example, at least one of a personal computer, a tablet, and a smartphone.

[0019] As shown in Figure 1, the display system 1 comprises an operator device 10 and a computer 13. The display system 1 may also comprise a server device 20 and an actual work machine 40.

[0020] The operator device 10 is a device operated by an operator. The operator device 10 is a device used by the operator to utilize the display system 1. The operator device 10 is a device used by the operator to operate the work machine 4 in a virtual space. The operator device 10 is a device used to display the work machine 4 and / or the surrounding environment simulated in the virtual space to the operator. The operator device 10 may also be used to remotely control the actual work machine 40. The operator device 10 comprises an input device 11 and an output device 12.

[0021] The input device 11 is a device for inputting information into the computer 13. The input device 11 is operated by an operator. The input device 11 may be used to change the viewpoint of the main image MP (see Figure 4) and / or the secondary image SP (see Figure 4). Here, the viewpoint refers to the position of focus (e.g., coordinates in virtual space). The input device 11 may be used to change the viewpoint (direction from the viewpoint) of the main image MP / or secondary image SP. The input device 11 may be used to set the focus area A1 (see Figure 4) in the main image MP. The input device 11 may be used to input setting information for automatically setting the focus area A1 in the main image MP. The input device 11 may include at least one of a mouse, keyboard, touch panel, joystick, pedal, and gamepad. The input device 11 may be a touch panel provided on the display unit 12a (e.g., a display), which will be described later. The input device 11 may be used for remote control of a real machine (a machine corresponding to the work machine 4). The input device 11 may receive instructions to start and stop the operation of the work machine 4. The input device 11 includes an operating device 11a.

[0022] The operating device 11a is a device that receives operations from an operator to operate the work machine 4 in the virtual space. The operating device 11a comprises a travel lever 11a2 and an attachment operating lever 11a1.

[0023] The travel lever 11a2 is operated to move the work machine 4. As shown in Figure 3, the travel lever 11a2 is located in front of the seat 10a. A pair of travel levers 11a2 are arranged side by side. The travel lever 11a2 also serves as the travel pedal. This travel pedal is fixed to the lower part of the travel lever 11a2.

[0024] The attachment operating lever 11a1 is operated to control the posture of the attachment 43. The attachment operating lever 11a1 comprises a left operating lever 11a1A and a right operating lever 11a1B. The left operating lever 11a1A is located in front of the left frame of the seat 10a. The right operating lever 11a1B is located in front of the right frame of the seat 10a. The left operating lever 11a1A and the right operating lever 11a1B are configured to allow multiple operations, such as tilting in the forward / backward direction and the left / right direction, respectively. Functions for rotating the upper slewing body 41b of the work machine 4 (see Figure 2), operating the boom 43a, operating the arm 43b, and operating the tip attachment 43c are assigned to one of the above multiple operations. The operation assignments may be changeable. The input unit 11 may also be operated to control the lower work unit. Specifically, for example, the input unit 11 may have an operating device for controlling the clamp arm as a lower working unit. The operating device may be a device for raising and lowering (rotating vertically) the clamp arm relative to the lower traveling body 41a. The operating device may be a device for opening and closing the clamp arm relative to the lower traveling body 41a. For example, the operating device may be an operating pedal provided separately from the travel pedal. Specifically, for example, the operating device may have an operating pedal for raising and lowering the clamp arm and an operating pedal for opening and closing the clamp arm.

[0025] As shown in Figure 1, the output device 12 outputs in response to commands input from the computer 13. The output device 12 includes a display unit 12a and a notification unit 12b.

[0026] As shown in Figure 3, the display unit 12a is a device for displaying information to the operator. The display unit 12a displays calculation results from the computer 13 (see Figure 1), etc. The display unit 12a displays the virtual space calculated by the computer 13. The display unit 12a is used to display the virtual space in which the work machine 4 is simulated. The display unit 12a may also be used to explain how to operate the work machine 4. The following "display" refers to the display that the computer 13 causes the display unit 12a to perform. For example, the display unit 12a includes a display unit 12a1.

[0027] The display unit 12a1 is located in front of the seat 10a. In this embodiment, the display unit 12a1 has a standalone display, but is not limited to that. The display unit 12a1 may also have a touch panel.

[0028] The notification unit 12b (see Figure 1) is a device for notifying the operator. The output of the notification unit 12b is, for example, one or more of either an audio output or a vibration output. The output of the notification unit 12b is, for example, feedback on the operator's actions. The output of the notification unit 12b is, for example, an operation that the operator should input. The output of the notification unit 12b may be displayed on the display unit 12a. For example, the notification unit 12b is a speaker.

[0029] Furthermore, for example, the seat 10a may be a motion seat that provides feedback to the operator through its operation. The feedback operation of the seat 10a consists of at least one of a change in the tilt angle of the seating area and vibration. When the work machine 4 operates in the virtual space, the seat 10a reproduces and provides feedback of the movement corresponding to the operation. For example, the seat 10a receives a signal indicating the tilt of the work machine 4 and is controlled to achieve the same tilt. The seat 10a transmits feedback to the operator operating the input device 11 through vibration.

[0030] Computer 13 (see Figure 1) is a computer that performs signal input / output, calculations, and information storage. Computer 13 simulates the work machine 4 and the surrounding environment in a virtual space. Each function of computer 13 is realized by the execution of a program stored in the memory unit 13a by the calculation unit 13b. Examples of the functions of computer 13 include the object processing unit 13b1, the gaze area determination unit 13b2, the sub-image state determination unit 13b3, the output control unit 13b4, the operation determination unit 13b5, and the remote control unit 13b6, which will be described later. Computer 13 and other devices (e.g., input device 11, output device 12) may be connected by wireless communication or by wired communication. Information output by the input device 11 is input to computer 13. Computer 13 outputs information to the output device 12. Computer 13 may exchange information with the server device 20. Communication between the computer 13 and the server device 20 is carried out by communication means such as a mobile phone line, fiber optic line, wireless LAN (Local Area Network), or wired LAN. The computer 13 comprises a storage unit 13a and an arithmetic unit 13b.

[0031] The memory unit 13a (see Figure 1) stores various types of information. The memory unit 13a stores the program for the display system 1. For example, the memory unit 13a stores information for determining the gaze area A1 (see Figure 4) in the main video MP (see Figure 4). For example, the memory unit 13a stores information for determining the state of the secondary video SP (see Figure 4). For example, the memory unit 13a stores authentication information for login. The authentication information may also be stored in the memory unit 13a of the server device 20 (see Figure 1).

[0032] The calculation unit 13b (see Figure 1) performs calculations (processing such as calculation and determination). The calculation unit 13b comprises an object processing unit 13b1, a gaze area determination unit 13b2, a sub-image state determination unit 13b3, an output control unit 13b4, an operation determination unit 13b5, and a remote control unit 13b6. The object processing unit 13b1 processes object 30 (described later). The gaze area determination unit 13b2 determines the gaze area A1 in the main image MP (see Figure 4) (described later). The sub-image state determination unit 13b3 determines the state of the sub-image SP (see Figure 4) in the main image MP (described later). The output control unit 13b4 determines the content to be output to the output device 12. The operation determination unit 13b5 determines the operation received by the input device 11 (described later). The remote control unit 13b6 controls the remote operation of the actual work machine 4 (actual work machine 40 (see Figure 1)) by the input device 11. The calculation unit 13b may have an authentication processing unit (not shown). The authentication processing unit performs a process to authenticate the user's login. The authentication processing unit identifies the user using user identification information. The user identification information may be stored in the storage unit 13a in association with a report. The user identification information may be, for example, a username and a user ID. The authentication process may be performed in the calculation unit 13b of the server device 20.

[0033] The server device 20 (see Figure 1) is a computer that operates in response to commands from the client device 19. The client device 19 consists of the operator device 10 (input device 11 and output device 12) and a computer 13. The server device 20 may store authentication information used by the arithmetic unit 13b to process user authentication. The server device 20 may also perform authentication processing to authenticate the user. The functions of the display system 1 (for example, the functions of the object processing unit 13b1, the gaze area determination unit 13b2, the sub-image state determination unit 13b3, the output control unit 13b4, the operation determination unit 13b5, and the remote control unit 13b6) may be implemented by the client device 19 and the server device 20. The computer 13 (at least one of the storage unit 13a and the arithmetic unit 13b) may be provided in both the client device 19 and the server device 20. Multiple server devices 20 may be provided. Note that the display system 1 does not necessarily have to include a server device 20. The input device 11 may be provided on the server device 20 (not shown). In this case, for example, information (for example, setting information for automatically setting the gaze area A1 (see Figure 4) in the main video MP (see Figure 4)) may be input from the input device 11 provided on the server device 20 to the computer 13 provided on the client device 19.

[0034] The actual work machine 40 (see Figure 1) is a real machine configured in the same way as the work machine 4 simulated by the display system 1. The display system 1 may include a remote control device for remotely controlling the actual work machine 40. The display system 1 may also be implemented in the remote control device. The remote control device includes an input device 11, a computer 13, and an output device 12. The input device 11 includes an operating device 11a (e.g., an operating lever, an operating pedal) and outputs operating signals (lever operating signals, pedal operating signals) in response to the operation of the operating device 11a. When remotely controlling, the computer 13 exchanges signals with the actual machine to be remotely controlled via a communication means. When remotely controlling, the computer 13 transmits signals (operation command signals) to the actual work machine 40 to operate the upper work device (corresponding to attachment 43 (see Figure 2)) and the traveling device (corresponding to lower traveling body 41a (see Figure 2)), etc., based on the operating signals input from the input device 11. During remote operation, the computer 13 receives image data from a camera mounted on the actual work machine 40, as well as signals output by sensors mounted on the actual work machine 40, via communication means. During remote operation, the output device 12 (specifically, the display unit 12a1) outputs image data from the camera mounted on the actual work machine 40 (data acquired by the computer 13).

[0035] Since the display system 1 is equipped with a remote control device, the user can operate the work machine 4 in the virtual space using the remote control device used to remotely operate the actual work machine 40. Therefore, by operating the work machine 4 in the virtual space, the user can experience the feel of operating the actual work machine 40. As a result, for example, the user's proficiency in operating the remote control device is promoted. In addition, for example, it becomes easy to make assumptions (such as what kind of situation would occur) about what would happen if the actual work machine 40 at the actual work site (corresponding to the work site) were operated with the remote control device.

[0036] (Operation) Display system 1 (primarily computer 13) is configured to perform the following operations (simulation and / or display of remote control devices). The simulation program causes computer 13 to perform the following operations. Display system 1 implements a method for performing the following operations.

[0037] As shown in Figure 2, the display system 1 simulates the work machine 4 in a virtual space. The object 30 of the work machine 4 may simulate the entire actual work machine 40 in the virtual space, or it may simulate a part of the actual work machine 40. The display system 1 simulates the surrounding environment in a virtual space. The surrounding environment consists of objects 30 such as people, the ground, the work object, obstacles, transport vehicles, walls, slope shoulders, embankments, buildings, roads, and roads such as guardrails. The display system 1 has an object processing function for controlling such objects 30.

[0038] (Object processing function) Computer 13 (specifically, object processing unit 13b1) processes object 30 (object processing function, object processing step). Object 30 is a model of the work machine 4 and the surrounding environment. Object 30 may be a three-dimensional model or a two-dimensional model. Note that each operation (function) of the display system 1 may be referred to as a "step" in the display program and display method. For example, the processing of object 30 may be referred to as an "object processing step". The processing of object 30 and the object processing step include the process of displaying object 30 on the display unit 12a1.

[0039] Computer 13 operates either the work machine 4 or the surrounding environment, or both. The work machine 4 may operate in response to operator input or automatically. For example, the work machine 4 may be operated by a machine control system (MC). Specifically, a work plan is set in computer 13. The operator then operates only some elements of the attachment 43 (see Figure 2), for example, only the boom 43a. In this case, computer 13 automatically controls the elements not operated by the operator (for example, the arm 43b, the end attachment 43c) so that the work machine 4 works according to the work plan. In this case, computer 13 controls the operation of the work machine 4 based on its posture (the same applies in the case of automatic operation). As a result, the work machine 4 works according to the work plan. Alternatively, for example, the work machine 4 may be operated by automatic operation. In this case, computer 13 controls the operation of the work machine 4 so that the work machine 4 works automatically according to the work plan.

[0040] (Regarding the movement of object 30 in response to the operation) Computer 13 operates object 30 in response to input received by operating device 11a (see Figure 3). Computer 13 may, for example, operate the lower traveling body 41a (see Figure 2) to move the work machine 4 by operating the traveling lever 11a2 (see Figure 3). Computer 13 may, for example, rotate the upper rotating body 41b (see Figure 2) by operating the left operating lever 11a1A (see Figure 3) in the left-right direction. Computer 13 may, for example, rotate the arm 43b by operating the left operating lever 11a1A in the front-back direction. Computer 13 may, for example, rotate the boom 43a (see Figure 2) by operating the right operating lever 11a1B (see Figure 3) in the front-back direction. Computer 13 may, for example, rotate the tip attachment 43c (see Figure 2) by operating the right operating lever 11a1B in the left-right direction.

[0041] For example, the computer 13 changes the operating speed of the object operated by the input device 11 according to the amount of operation of the input device 11. For example, the computer 13 changes the operating speed of the attachment 43 according to the amount of operation of the attachment operating lever 11a1. For example, the computer 13 changes the operating speed of the work machine 4 according to the set engine speed. For example, the computer 13 changes the operating speed of the basic attachment 43 and the travel speed of the work machine 4 according to the engine speed.

[0042] (Regarding the content displayed on display unit 12a) As shown in Figure 4, the work machine 4 operated by the input device 11 and objects 30 of the surrounding environment (e.g., the work object) are displayed on the display unit 12a (specifically, the display unit 12a1). The objects 30 representing the surrounding environment are, for example, one or more of the following: a person, the ground, the work object, an obstacle, a transport vehicle, a wall, and a slope. The main image MP representing the work machine 4 and the surrounding environment is displayed across the entire surface of the display unit 12a. In addition, a sub-image SP is displayed superimposed on the main image MP. The sub-image SP may be a single image or multiple images. The state of the sub-image SP is determined by the computer 13 (more specifically, the sub-image state determination unit 13b3) (sub-image state determination processing function, sub-image state determination processing step). The state of the sub-image SP may be information indicating its position and range in the main image MP. The state of the sub-image SP may be information indicating its shape. The state of the sub-image SP may also be information indicating whether it is displayed or hidden (presence or absence of the sub-image). Note that in the example in Figure 4, there is only one screen, but this is not limited to one; there may be multiple screens. Also, for example, in the example in Figure 4, there are sub-video SPs with different widths, but all sub-video SPs may have the same width.

[0043] (Regarding the main video MP) The viewpoint of the main image MP is not limited to a specific position. For example, the viewpoint of the main image MP may be inside the operator's cab 41b2 of the work machine 4. Alternatively, it may be outside the operator's cab 41b2. For example, the viewpoint of the main image MP may be above the operator's cab 41b2. Alternatively, it may be to the side of the operator's cab 41b2. Alternatively, it may be behind the operator's cab 41b2. The viewpoint of the main image MP may be moved. The movement of the viewpoint of the main image MP may be performed automatically or in response to an operation received by the input device 11. The viewpoint of the main image MP is not limited to a specific position. For example, the viewpoint of the main image MP may be the work machine 4 or a specific part of the work machine 4 (e.g., an attachment 43), the work object of the work machine 4, or the working range of the work machine 4. For example, the main video MP may be a top view with the viewpoint above the work machine 4 and the work machine 4 as the viewpoint. The main video MP may be a side view with the viewpoint to the side of the work machine 4 and the work machine 4 as the viewpoint. The main video MP may be an oblique view with the viewpoint diagonally above the work machine 4 and the work machine 4 as the viewpoint. When the actual work machine 40 is remotely operated, the main video MP will be the image acquired by an imaging device (not shown) installed on the actual work machine 40. When overlaying a secondary image, the main image outside the area of ​​focus may be displayed as is, cropped and not displayed, or filled with a single color. Furthermore, the display area shapes for both the main and secondary images are not limited to rectangles; they may also be polygons, circles, ellipses, etc.

[0044] The viewpoint direction of the main video MP may be changed. The viewpoint of the main video MP may be changed automatically or in response to an operation received by the input device 11. For example, the viewpoint of the main video MP may be changed by the elevator cab, which is the driver's cab 41b2, moving vertically relative to the bottom of the upper slewing body 41b. Alternatively, for example, the viewpoint of the main video MP may be changed by the link mechanism of the link cab, which is the driver's cab 41b2, being activated, causing it to move vertically and longitudinally relative to the bottom of the upper slewing body 41b. Alternatively, for example, the viewpoint of the main video MP may be changed by the tilt cab, which is the driver's cab 41b2, rotating vertically around a pivot axis extending laterally relative to the bottom of the upper slewing body 41b. Alternatively, for example, the viewpoint of the main video MP may be changed by the rotation of the upper slewing body 41b. Alternatively, for example, the viewpoint of the main video MP may be changed by the movement of the lower traveling body 41a. Furthermore, the main video (MP) may be displayed enlarged or reduced in size.

[0045] (Regarding the supplementary video SP) Similar to the main image MP, the secondary image SP is not limited to a specific viewpoint position. The example of the main image MP described above can be applied to the secondary image SP. The viewpoint of the secondary image SP may be moved. The movement of the viewpoint of the secondary image SP may be performed automatically or in response to an operation received by the input device 11. Similar to the main image MP, the viewpoint position of the secondary image SP is not limited. The example of the main image MP described above can be applied to the secondary image SP. The direction of the viewpoint of the secondary image SP may be changed. The change of the viewpoint of the secondary image SP may be performed automatically or in response to an operation received by the input device 11. In addition, the secondary image SP may be displayed enlarged or reduced. Note that when the actual work machine 40 is remotely operated, the secondary image SP is an image acquired by an imaging device (not shown) installed on the actual work machine 40, which is separate from the imaging device that captures the main image MP.

[0046] (Regarding the content of the supplementary video SP) For example, in the example shown in Figure 4, six sub-images SP (sub-images SP1 to SP6) are superimposed on the main image MP. Specifically, sub-image SP1 is an image with the viewpoint from the left side of the work machine 4, looking forward. Sub-image SP1 also includes a transport vehicle as part of the surrounding environment. Sub-image SP2 is a top view of the transport vehicle. Sub-image SP3 is a perspective view including the transport vehicle and the work machine 4. Sub-image SP4 is an image with the viewpoint from the right side of the work machine 4, looking forward. Sub-image SP5 shows the work object as part of the surrounding environment. Sub-image SP2 is a perspective view including the work object and the work machine 4. Sub-image SP6 is a side view of the work machine 4. Sub-images SP may include one or both of the work object and the work machine 4. Thus, in this embodiment, the computer 13 displays the work machine 4 as seen from outside the work machine 4 as a sub-image SP on the display unit 12a.

[0047] Furthermore, the secondary image SP may not include either the work object or the work machine 4. For example, as shown in Figure 5, the secondary image SP may be an image showing the input status of the input device 11. Specifically, the secondary image SP may be an image showing the operating status of the operating device 11a. The secondary image SP may display the operating status of one or both of the attachment operating lever 11a1 and the travel lever 11a2. The secondary image SP may display the operating status of both attachment operating levers 11a1 (left operating lever 11a1A and right operating lever 11a1B). Specifically, the secondary image SP schematically shows the operating status of the left operating lever 11a1A and the right operating lever 11a1B in a gauge-like manner.

[0048] Furthermore, the display position of the secondary image SP may be determined based on its positional relationship with the main image MP. For example, the display position of the secondary image SP relative to the gaze area A1 (described later) in the main image MP may be a display position corresponding to the position or direction of the viewpoint of the secondary image SP relative to the viewpoint of the main image MP. Specifically, an example of the secondary image SP1 is shown with reference to Figure 4. As described above, the viewpoint of the secondary image SP1 is located to the left of the work machine 4. In the virtual space, the viewpoint of the secondary image SP1 is located to the left when viewed from the viewpoint of the main image MP (inside the driver's cab 41b2) in the direction of the viewpoint of the main image MP. Therefore, the display position of the secondary image SP1 is located to the left of the gaze area A1.

[0049] Although not shown in the diagram, for example, if the viewpoint of the main video MP is facing left, as in the example in Figure 4, the secondary video SP, which is positioned to the right of the main video MP, may be changed to an image related to the front of the work machine 4. If the viewpoint of the main video MP is to the left of the work machine 4, the secondary video SP, which is positioned to the left of the main video MP, may be changed to an image related to the rear of the work machine 4. Thus, the display position of the secondary video SP may correspond to the position or direction of the viewpoint of the secondary video SP relative to the viewpoint of the main video MP. The content of the secondary video SP may also include information such as operating procedures, work instructions, and machine tilt. Furthermore, the secondary video SP may be displayed in a position corresponding to the driver's seat monitor control unit (cluster) or ICT construction monitor of the actual work machine 40. In addition, the secondary video SP may be displayed while avoiding the mirror (not shown) of the work machine 4.

[0050] (Regarding the area of ​​focus A1) A gaze area A1 is set in the main video MP. The gaze area A1 is used to determine the area in which the secondary video SP is displayed. The secondary video SP is displayed outside the range of the gaze area A1 in the main video MP. The range of the gaze area A1 is determined by the computer 13 (more specifically, the gaze area determination unit 13b2) (gaze area determination processing function, gaze area determination processing step). In this embodiment, the gaze area A1 is determined by the computer 13, but it may be set in advance. The gaze area A1 may be set arbitrarily. The computer 13 may set the gaze area arbitrarily in response to an operation received by the input device 11. The computer 13 may determine the gaze area A1 periodically. The computer 13 may determine the gaze area A1 for each event. An event that determines the gaze area A1 is, for example, the operation of the work machine 4. More specifically, an event that determines the gaze area A1 is the movement of the lower traveling body 41a. More specifically, the event that determines the gaze area A1 is the rotation of the upper rotating body 41b. More specifically, the event that determines the gaze area A1 is a change in the posture of the attachment 43. More specifically, the event that determines the gaze area A1 is the acceptance of an operation by the input device 11 to operate the work machine 4. The gaze area A1 may be determined based on the viewpoint of the main image MP. For example, the gaze area A1 may be a specific area fixed on the display unit 12a. The gaze area A1 may also be determined according to the content of the main image MP. For example, as shown in Figure 4, the gaze area A1 may be the range displayed through the front window 41b2A of the operator's cab 41b2 in the main image MP, which is the viewpoint from inside the operator's cab 41b2 of the work machine 4. The gaze area A1 may also be determined arbitrarily. The gaze area A1 may also be determined based on the operation of the attachment 43. For example, the gaze area A1 may also be determined according to the operation of the work machine 4 in the main image MP. More specifically, the gaze area A1 may be enlarged depending on the display range of the attachment 43 of the work machine 4.

[0051] Furthermore, as shown in Figure 6, for example, the gaze area A1 may be determined to include the operable range of the attachment 43 of the work machine 4 in the main image MP. In the example in Figure 6, the viewpoint of the main image MP is above the cab 41b2. In this case, the operable range of the attachment 43 is included in the right portion of the main image MP. Therefore, the gaze area A1 is set in the right portion of the main image MP. Each sub-image SP is set outside the range of the gaze area A1 in the main image MP. In the example in Figure 6, the sub-image SP is positioned on the left portion of the main image MP, superimposed on the main image MP. For example, if the viewpoint of the main image MP is on the opposite side of the cab 41b2 from the attachment 43, the operable range of the attachment 43 is included in the left portion of the main image MP. Therefore, the gaze area A1 will be set in the left portion of the main image MP. The secondary image SP is positioned superimposed on the main image MP in the right portion of the main image MP, which is outside the range of the gaze area A1. Thus, the gaze area A1 includes, but is not limited to, a portion of the range in the main image MP where the upper work unit, such as the attachment 43, can operate. For example, although not shown, the gaze area A1 may include a portion of the range in which the lower work unit, such as an attachment provided on the lower traveling body 41, can operate. The gaze area A1 may also include the range in the main image MP where the lower traveling body 41a can travel. Furthermore, the gaze area A1 may include the range in the main image MP where the upper rotating body 41b can rotate.

[0052] (Changes in the status of the secondary video SP) The state of the secondary image SP may change in accordance with the deformation of the gaze area A1. For example, the state of the secondary image SP may be its position in the main image MP. Alternatively, for example, the state of the secondary image SP may be its range. Alternatively, for example, the state of the secondary image SP may be its shape. Alternatively, for example, the state of the secondary image SP may be its presence or absence. The presence or absence of the secondary image SP indicates whether or not to display the secondary image SP.

[0053] The state of the secondary image SP may change according to the state of the main image MP. For example, the state of the main image MP is the viewpoint position of the main image MP. Also, for example, the state of the main image MP is the direction of the viewpoint of the main image MP. Also, for example, the state of the main image MP is the state of the main image MP's scaling. For example, in the example in Figure 7, the main image MP shows a state where the viewpoint has been moved upward from the example in Figure 4. Due to the movement of the viewpoint of the main image MP, the gaze area A1 is deformed into the gaze area A2. Specifically, the deformed gaze area A2 is roughly trapezoidal, and in the main image MP, the space outside the gaze area A2 is wider at the top and narrower at the bottom. Then, in line with the deformed gaze area A4, the vertically elongated secondary images SP (secondary images SP3 and SP6) are moved upward compared to the horizontally elongated secondary images SP (secondary images SP1, SP2, SP4 and SP5).

[0054] Furthermore, for example, in the examples in Figures 8A and 8C, the viewpoint direction of the main video MP is moved vertically. Specifically, in the example in Figure 8A, the gaze area A3a is positioned in the center of the main video MP. The secondary video SP7 is positioned above the gaze area A3a. The secondary video SP8 is positioned below the gaze area A3a. In the example in Figure 8B, the viewpoint of the main video MP is moved upward compared to the example in Figure 8A. As the main video MP moves, the gaze area A3a in the main video MP moves upward and changes to the gaze area A3b. As the gaze area moves from A3a to A3b, the secondary video SP7, which was above the gaze area A3a, moves below the gaze area A3b. In this way, by moving the secondary video SP, the visibility of the main video MP in the direction that the operator wants to see (direction of viewpoint movement) can be improved. Furthermore, as shown in Figure 8C, when the viewpoint of the main image MP is moved downward compared to the example in Figure 8A, the gaze region A3a in the main image MP moves downward and changes to the gaze region A3c. The secondary image SP8 is then moved above the gaze region A3c. In this way, the computer 13 moves the gaze region A according to the direction of the viewpoint of the main image MP and changes the state of the secondary images SP7 and SP8. As a result of the change in the state of the main image MP, the state of the secondary image SP may also change.

[0055] Furthermore, the state of the secondary image SP may change in accordance with the operation of the work machine 4. For example, the operation of the work machine 4 that changes the state of the secondary image SP may be the operation of the work machine 4 in the main image MP. More specifically, the operation of the work machine 4 that changes the state of the secondary image SP may be, for example, the operation of the attachment 43 of the work machine 4 in the main image MP. For example, in the example of Figure 9, compared to the example of Figure 4, the attachment 43 of the work machine 4 is activated and moves closer to the operator's cab 41b2. As a result, the gaze area A1 (see Figure 4) is expanded to the right of the main image MP and transformed into the gaze area A4. As a result, the secondary images SP4 to SP6 that were displayed on the right side of the main image MP (to the right of the gaze area A1) are hidden. In this way, the state of the secondary image SP may change as a result of the operation of the work machine 4. Note that the secondary image SP may be hidden when displaying other information (such as explanatory text).

[0056] Furthermore, the state of the secondary image SP may change in response to the input device 11 receiving an operation to operate the work machine 4. For example, an operation that changes the state of the secondary image SP is a rotation operation of the work machine 4. Another example is a travel operation of the work machine 4. Yet another example is a movement or rotation operation of the driver's cab 41b2. Yet another example is a change in the imaging direction of the imaging device (for example, the imaging device for the main image MP) mounted on the actual work machine 40. For example, in the example in Figure 10, the main image MP shows the state in which the work machine 4 has been rotated to the left from the example in Figure 4. As a result of the operation to rotate the work machine 4, the gaze area A1 (see Figure 4) has been transformed into a gaze area A5 that has been moved from the center to the right. In the example in Figure 4, the secondary images SP (secondary images SP4 to SP6), which were positioned to the right of the main image MP, have moved to the left in the example in Figure 10 to match the deformed gaze area A5. Thus, the state of the secondary images SP may change as a result of operating the work machine 4. The amount of movement of the gaze area A5 may be determined based on the amount of operation to the input device 11. Furthermore, the direction of viewpoint movement may be the direction in which the upper rotating body 41b rotates, or it may be the direction in which the lower traveling body 41a travels.

[0057] (Effects of the first invention) [Configuration 1] The display system 1 comprises a computer 13 and a display unit 12a. The display unit 12a displays the work machine 4 and / or the surrounding environment (e.g., transport vehicles and work objects, etc.) as the main image MP. The computer 13 displays the work machine 4 and / or the surrounding environment from a different viewpoint than the main image MP as a secondary image SP on the display unit 12a in an area outside the gaze area A1 of the main image MP.

[0058] In the above [Configuration 1], the work machine 4 and / or the surrounding environment are displayed as the main image MP, while the secondary image SP of the work machine 4 and / or the surrounding environment is displayed outside the gaze area A1 determined by the main image MP. Therefore, since the secondary image SP is not superimposed on the gaze area A1 in the main image MP, the reduction in visibility of the main image MP within the gaze area A1 is reduced, while the amount of information displayed can be improved by displaying the secondary image SP outside the gaze area A1.

[0059] (Effects of the second invention) [Configuration 2] As shown in Figure 4, in the display system 1, the viewpoint of the main video MP is the viewpoint from inside the operator's cab 41b2 of the work machine 4.

[0060] In the above [Configuration 2], the main video MP can be a view from inside the driver's cab 41b2.

[0061] (Effects of the third invention) [Configuration 3] As shown in Figure 4, in the display system 1, the gaze area A1 is the area of ​​the front window 41b2A of the driver's cab 41b2 in the main video MP.

[0062] In the above [Configuration 3], the image within the range of the front window 41b2A as seen from inside the driver's cab 41b2 is designated as the gaze area A1, and a secondary image SP can be displayed outside that range. Therefore, the reduction in visibility of the image within the range of the front window 41b2A can be mitigated.

[0063] (Effects of the fourth invention) [Configuration 4] In the display system 1, the work machine 10 has an upper work section (e.g., attachment 43) and / or a lower work section (e.g., a clamp arm provided on the lower traveling body 41), and the gaze area A1 includes a portion of the range in which the upper work section or the lower work section can operate in the main image MP.

[0064] In the above configuration [4], the gaze area A1 includes a portion of the range in which the upper or lower work unit in the main video MP can operate, thereby reducing the decrease in visibility of the video of the work of the work machine 4.

[0065] (Effects of the fifth invention) [Configuration 5] As shown in Figure 6, the work machine 4 has an attachment 43 for performing work, and the gaze area A1 includes the range in the main image MP where the attachment 43 can operate.

[0066] In the above configuration [5], the gaze area A1 includes the operating range of the attachment 43 in the main video MP, thus reducing the decrease in visibility of the video of the attachment 43 of the work machine 4.

[0067] (Effects of the sixth invention) [Configuration 6] As shown in Figure 4, in the display system 1, the computer 13 displays the work machine 4 as seen from the outside of the work machine 4 as a secondary image SP (for example, secondary images SP3, SP5, and SP6) on the display unit 12a.

[0068] In the above [Configuration 6], the work machine 4 as seen from the outside is displayed as a secondary image SP in the main image MP. This allows the status of the work machine 4 as seen from the outside to be checked simultaneously with the gaze area A1 in the main image MP.

[0069] (Effects of the seventh invention) [Configuration 7] In the display system 1, the work machine 10 has a lower traveling body 41a as a traveling device, and the gaze area A1 includes the range in which the lower traveling body 41a can travel in the main image MP.

[0070] In the above configuration [7], the gaze area A1 includes the range in which the lower traveling body 41a can travel in the main video MP, thus reducing the decrease in visibility of the video of the work of the work machine 4.

[0071] (Effects of the 8th Invention) [Configuration 8] In the display system 1, the work machine 10 has a rotatable upper slewing body, and the gaze area A1 includes the range in which the upper slewing body 41b can rotate in the main image MP.

[0072] In the above configuration [8], the gaze area A1 includes the range in which the upper rotating body 41b in the main video MP can rotate, thus reducing the decrease in visibility of the video of the work of the work machine 4.

[0073] (Effects of the 9th Invention) [Configuration 9] As shown in Figure 4, in the display system 1, the computer 13 sets the display position of the secondary image SP relative to the gaze area A1 in the main image MP to a display position corresponding to the position or direction of the viewpoint of the secondary image SP (secondary image SP1 and secondary image SP4) relative to the viewpoint of the main image MP.

[0074] In the above [Configuration 9], the position of the secondary video SP can be displayed so as to match the correspondence between the viewpoint or direction of the main video MP and the secondary video SP.

[0075] (Effects of the 10th Invention) [Configuration 10] The display system 1 is equipped with an input device 11 that accepts operations from an operator, and the computer 13 arbitrarily sets the gaze area A1 according to the operations accepted by the input device 11.

[0076] In the above [Configuration 10], the gaze area A1 can be set arbitrarily, so the area where visibility is to be ensured can be set as desired.

[0077] (Effects of the 11th Invention) [Configuration 11] As shown in the examples in Figures 7, 8A, 8B, and 8C, in the display system 1, the computer 13 changes one or more of the following in relation to the main image MP: the position of the viewpoint of the main image MP, the direction of the viewpoint of the main image MP, and the state of scaling of the main image MP: the position of the secondary image SP in the main image MP, the range of the secondary image SP, the shape of the secondary image SP, and the presence or absence of the secondary image SP.

[0078] In the above [Configuration 11], the state of the secondary video SP can be changed in accordance with the change in the state of the main video MP.

[0079] (Effects of the 12th Invention) [Configuration 12] As shown in the example in Figure 9, in the display system 1, the computer 13 changes one or more of the following in response to the operation of the work machine 4: the position of the secondary image SP in the main image MP, the range of the secondary image SP, the shape of the secondary image SP, and the presence or absence of the secondary image SP.

[0080] In the above configuration

[12] , the state of the secondary video SP can be changed in accordance with the operation of the work machine 4.

[0081] (Effects of the 13th Invention) [Configuration 13] As shown in the example in Figure 10, the display system 1 is equipped with an input device 11 that accepts operations from an operator, and the computer 13 changes one or more of the following in response to the input device 11 receiving an operation to operate the work machine 4: the position of the secondary image SP in the main image MP, the range of the secondary image SP, the shape of the secondary image SP, and the presence or absence of the secondary image SP.

[0082] In the above configuration

[13] , the state of the secondary video SP can be changed in accordance with the operation of operating the work machine 4.

[0083] (Effects of the 14th Invention) [Configuration 14] As shown in Figure 3, the display system 1 includes an input device 11 that receives operations from an operator to operate the work machine 4 in the virtual space. The input device 11 is used to remotely control the actual work machine 4 (actual work machine 40).

[0084] In the above [Configuration 14], the input device 11 for operating the work machine 4 in the virtual space can also be used as a device for remotely controlling the actual work machine 40. Therefore, the operator can remotely control the actual work machine 40 without changing the feel of operation.

[0085] (modified version) The above embodiments may be modified in various ways. For example, various examples (including modified versions) of the above embodiments may be combined in various ways. For example, the connections of each component shown in Figure 1, etc., may be changed. For example, the number of components (including modified versions) of the above embodiments may be changed, and some components may not be provided. This "component" includes components in the virtual space (specifically, object 30) (the same applies hereinafter). For example, the arrangement of components may be changed. For example, the inclusion relationships of components may be changed in various ways. For example, something described as a subordinate component included in a higher-level component may not be included in this higher-level component, but may be included in other components. For example, something described as multiple different members or parts may be treated as a single member or part. For example, something described as a single member or part may be divided and provided as multiple different members or parts. [Explanation of Symbols]

[0086] 1: Display System 4: Work Machinery 10: Operator device 11: Input device 11a: Operating device 12: Output device 12a: Display section 13: Calculator 13a: Storage section 13b: Arithmetic section 13b1: Object Processing Unit 13b2: Gaze area determination part 13b3: Sub-image state determination unit 13b4: Output control unit 13b5: Operation judgment section 13b6: Remote control unit 19: Client device 20: Server device 30: Object 40: Actual work machine 41b2: Driver's cab 41b2A: Front window 43: Attachment MP: Main Image SP: Sub-image

Claims

1. A computer and, A display unit that displays the work machine and / or the surrounding environment as the main image, Equipped with, The aforementioned computer is The work machine and / or the surrounding environment from a different viewpoint than the main image are displayed on the display unit as a secondary image in an area outside the viewing area of ​​the main image. Display system.

2. A display system according to claim 1, The viewpoint of the main video is from inside the operator's cabin of the aforementioned work machine. Display system.

3. A display system according to claim 3, The aforementioned viewing area is the area of ​​the front window of the driver's cab in the main image. Display system.

4. A display system according to claim 1, The aforementioned work machine has an upper work section and / or a lower work section, The aforementioned viewing area includes a portion of the range in which the upper or lower work unit can operate in the main image. Display system.

5. A display system according to claim 1, The aforementioned work machine has an attachment for performing work, The gaze area includes the range in the main image in which the attachment can be operated. Display system.

6. A display system according to claim 1, The computer displays the working machine as seen from the outside of the working machine as the secondary image on the display unit. Display system.

7. A display system according to claim 6, The aforementioned work machine has a traveling device, The aforementioned viewing area includes the range in which the traveling device can travel in the main image. Display system.

8. A display system according to claim 6, The aforementioned work machine has a rotatable upper rotating body, The aforementioned viewing area includes the range in which the upper rotating body can rotate in the main image. Display system.

9. A display system according to claim 1, The computer sets the display position of the secondary image relative to the gaze area in the main image to a display position corresponding to the position or direction of the viewpoint of the secondary image relative to the viewpoint of the main image. Display system.

10. A display system according to claim 1, Equipped with an input device that accepts operations from an operator, The computer sets the gaze area arbitrarily in response to the operation received by the input device. Display system.

11. A display system according to claim 1, The computer changes one or more of the following in relation to the main image: the position of the viewpoint of the main image, the direction of the viewpoint of the main image, and the state of scaling the main image: the position of the secondary image within the main image, the range of the secondary image, the shape of the secondary image, and the presence or absence of the secondary image. Display system.

12. A display system according to claim 1, The computer changes one or more of the following in response to the operation of the work machine: the position of the secondary image in the main image, the range of the secondary image, the shape of the secondary image, and the presence or absence of the secondary image. Display system.

13. A display system according to claim 1, Equipped with an input device that accepts operations from an operator, The computer changes one or more of the following in response to the input device receiving an operation to operate the work machine: the position of the secondary image in the main image, the range of the secondary image, the shape of the secondary image, and the presence or absence of the secondary image. Display system.

14. A display system according to claim 1, The system includes an input device that receives operations from an operator to operate the work machine in the virtual space, The input device is used to remotely control the actual machine of the work machine. Display system.

Images