Display system, remote driving device and display control method

Abstract

An acquisition unit acquires a captured image captured by an imaging device provided to a work machine. A rotation determination unit determines whether or not a rotating body of the work machine is rotating. When the rotating body is determined to be rotating, a display control unit outputs to a display device a display signal causing the display device to display an image in which a partial area of the captured image on the opposite side in the rotation direction is blocked.

Description

TITLE OF INVENTION: DISPLAY SYSTEM, REMOTE DRIVING DEVICE AND DISPLAY CONTROL METHOD 5 FIELD

[0001] The present disclosure relates to a remote driving device and a display control method. The present application claims priority on Japanese Patent Application No. 10   2022-175051, filed in Japan on October 31, 2022, the content of which is incorporated herein by reference. BACKGROUND

[0002] Technology for aiding the remote operation of a work machine by an operator by 15 displaying, on a display device provided at a remote location, video images from an imaging device provided on the work machine is known, for example Japanese Unexamined Patent Application, First Publication No. H9-218713 (Patent Document 1).

[0003] Any discussion of documents, acts, materials, devices, articles or the like which 20 has been included in the present specification is not to be taken as an admission that any or all of these matters form part of the prior art base or were common general knowledge in the field relevant to the present disclosure as it existed before the priority date of each of the appended claims. SUMMARY OF INVENTION 25

[0004] 2023372861   20 Aug 2026 In order to reduce the gap between remote operation and actual machine operation, it is desirable to construct, in the remote driving device, an environment close to that when an operator actually boards a work machine. Therefore, the video images from the imaging device provided on the work machine are required to be displayed on a 5 large screen. However, when an entire screen occupying much of the field of view of an operator moves in unison due to rotation of the work machine, there is a possibility that the operator will experience motion sickness. One or more embodiments of the present disclosure may be suitable for 10 providing a display system, a remote driving device, and a display control method that can prevent motion sickness in an operator. One or more embodiments of the present disclosure address or ameliorate at least one disadvantage or shortcoming of prior techniques, or at least provide a useful alternative thereto. 15

[0005] Some embodiments of the present disclosure relate to a display system used for operating a work machine comprising a rotating body that rotates about a center of rotation. The display system comprises: a display device; and a display control device that outputs a display signal for displaying an image on the display device. The display 20 device is provided so as to span regions that are in an operator's peripheral vision when the operator, seated on a driving seat, looks at a center of the display device. The display control device comprises: an acquisition unit that acquires an environmental image showing a direction faced by the rotating body in a peripheral environment of the work machine, a rotation determination unit that determines whether or not the rotating body is 25 rotating, and a display control unit that outputs, to the display device, a display signal for 2023372861   20 Aug 2026 displaying an image in which a region of the environmental image that is in the operator's peripheral vision when displayed on the display device and that is on a side opposite to a rotation direction is blocked, for preventing motion sickness induced by perceiving large movement in the operator's peripheral vision, when the rotating body is determined to be 5 rotating. The term ‘comprising’ as used in this specification means ‘consisting at least in part of’. When interpreting each statement in this specification that includes the term ‘comprising’, features other than that or those prefaced by the term may also be present. Related terms such as ‘comprise’ and ‘comprises’ are to be interpreted in the same 10 manner. Some embodiments of the present disclosure relate to a display control method in a display device used for operating a work machine comprising a rotating body that rotates about a center of rotation. The display device is provided so as to span regions that are in an operator's peripheral vision when the operator, seated on a driving seat, 15 looks at a center of the display device. The display control method comprises: acquiring an environmental image showing a direction faced by the rotating body in the peripheral environment of the work machine; determining whether or not the rotating body is rotating; and outputting, to the display device, a display signal for displaying an image in which a region of the environmental image that is in the operator's peripheral vision 20 when displayed on the display device and that is on a side opposite to a rotation direction is blocked, for preventing motion sickness induced by perceiving large movement in the operator's peripheral vision, when the rotating body is determined to be rotating. Some embodiments of the present disclosure relate to a remote driving device provided outside a work machine comprising a rotating body that rotates about a center 25 of rotation. The remote driving device comprises: a display device; and a display control 2023372861   20 Aug 2026 device that outputs a display signal for displaying an image on the display device. The display device is provided so as to span regions that are in an operator's peripheral vision when the operator, seated on a driving seat, looks at a center of the display device. The display control device comprises: an acquisition unit that acquires an environmental 5 image showing a direction faced by the rotating body in a peripheral environment of the work machine, a rotation determination unit that determines whether or not the rotating body is rotating, and a display control unit that outputs, to the display device, a display signal for displaying an image in which a region of the environmental image that is in the operator's peripheral vision when displayed on the display device and that is on a side 10 opposite to a rotation direction is blocked, for preventing motion sickness induced by perceiving large movement in the operator's peripheral vision, when the rotating body is determined to be rotating. Disclosed herein is a display system that is a display system used for operating a work machine comprised a rotating body that rotates about a center of rotation, the 15 display system comprising a display device and a display control device that outputs a display signal for displaying an image on the display device, wherein the display control device comprises an acquisition unit that acquires an environmental image showing a direction faced by the rotating body in a peripheral environment of the work machine, a rotation determination unit that determines whether or not the rotating body is rotating, 20 and a display control unit that outputs, to the display device, a display signal for displaying an image in which a side of the environmental image opposite to a rotation direction is blocked when the rotating body is determined to be rotating.

[0006] One or more embodiments of the present disclosure may be suitable for 25 providing a display system that can prevent motion sickness in an operator. 2023372861   20 Aug 2026 BRIEF DESCRIPTION OF DRAWINGS

[0007] [FIG. 1] A schematic view illustrating the configuration of a remote operation system according to a first embodiment. 5            [FIG. 2] An external view of a work machine according to the first embodiment. [FIG. 3] A view illustrating the configuration of an imaging device according to the first embodiment. [FIG. 4] A schematic block diagram illustrating the configuration of a control device of a remote driving device according to the first embodiment. 10           [FIG. 5] A flow chart indicating a display control method due to a control device according to the first embodiment. [FIG. 6] A view illustrating the difference in a display on a display device before and after initiating left rotation according to the first embodiment. DETAILED DESCRIPTION 15

[0008] <First Embodiment> <<Remote driving system>> FIG. 1 is a schematic view illustrating the configuration of a remote operation system according to a first embodiment. 20           The remote operation system 1 comprises a work machine 100 that is actuated by remote operation, and a remote driving device 500 for carrying out the remote operation. The work machine 100 runs at a work site (for example, a mine or a quarry). The remote driving device 500 is provided at a geographical location (for example, in a city or within the work site) removed from the work machine 100. The work machine 25   100 and the remote driving device 500 are connected by communication means such as 2023372861   20 Aug 2026 the internet. The remote operation system 1 is a system for operating the work machine 100 by using the remote driving device 500.

[0009] 5           The work machine 100 is actuated in accordance with operation signals received from the remote driving device 500. The remote driving device 500 transmits operation signals input by operations of an operator to the work machine 100.

[0010] 10 <<Configuration of work machine 100>> FIG. 2 is an external view of the work machine 100 according to the first embodiment. The work machine 100 according to the first embodiment is a hydraulic shovel. The work machine 100 according to another embodiment may be a rotatable work 15 machine other than a hydraulic shovel, for example, a crane, an electric shovel, etc. The work machine 100 comprises a traveling body 110, a rotating body 120, and work equipment 130.

[0011] The traveling body 110 supports the work machine 100 so as to be capable of 20 traveling. The traveling body 110 comprises two endless tracks 111 provided on the left and right, and two travel motors 112 for driving the respective endless tracks 111. The traveling body 110 is an example of a support. The rotating body 120 is supported on the traveling body 110 so as to be capable of rotating about a center of rotation. 25           The work equipment 130 is driven hydraulically. The work equipment 130 2023372861   20 Aug 2026 according to another embodiment may be driven electrically. The work equipment 130 is supported on the front portion of the rotating body 120 so as to be capable of being driven in the vertical direction. In this case, the portion of the rotating body 120 to which the work equipment 130 is attached is referred to as the front portion. Additionally, 5 regarding the rotating body 120, with reference to the front portion, the portion on the opposite side is referred to as the rear portion, the portion on the left side is referred to as the left portion, and the portion on the right side is referred to as the right portion.

[0012] <<Configuration of work equipment 130>> 10           The work equipment 130 comprises a boom 131, an arm 132, and a bucket 133. The work equipment 130 is driven by extending and retracting a boom cylinder 131C, an arm cylinder 132C, and a bucket cylinder 133C. A boom angle sensor 131S, an arm angle sensor 132S, and a bucket angle sensor 133S are respectively mounted on the boom 131, the arm 132, and the bucket 133. 15

[0013] A proximal end of the boom 131 is attached to the rotating body 120 by a pin. The arm 132 couples the boom 131 with the bucket 133. The proximal end of the arm 132 is attached to the distal end of the boom 131 by a pin. The bucket 133 comprises a blade for excavating dirt, etc. and a container for 20 containing the excavated dirt. The proximal end of the bucket 133 is attached to the distal end of the arm 132 by a pin.

[0014] The boom cylinder 131C is, for example, a hydraulic cylinder for driving the boom 131. The proximal end of the boom cylinder 131C is attached to the rotating body 25   120. The distal end of the boom cylinder 131C is attached to the boom 131. 2023372861   20 Aug 2026 The arm cylinder 132C is, for example, a hydraulic cylinder for driving the arm 132. The proximal end of the arm cylinder 132C is attached to the rotating body 120. The distal end of the arm cylinder 132C is attached to the arm 132. 5           The bucket cylinder 133C is, for example, a hydraulic cylinder for driving the bucket 133. The proximal end of the bucket cylinder 133C is attached to the boom 131. The distal end of the bucket cylinder 133C is attached to the bucket 133.

[0015] The boom angle sensor 131S is attached to the boom 131 and detects the 10 inclination angle of the boom 131. The arm angle sensor 132S is attached to the arm 132 and detects the inclination angle of the arm 132. The bucket angle sensor 133S is attached to the bucket 133 and detects the inclination angle of the bucket 133. 15           The boom angle sensor 131S, the arm angle sensor 132S, and the bucket angle sensor 133S according to the first embodiment detect the inclination angles with respect to the horizontal plane. The angle sensors according to other embodiments are not limited thereto, and may detect inclination angles with respect to other reference planes. For example, in another embodiment, the angle sensors may detect relative rotation angles by 20 means of potentiometers provided at the proximal ends of the boom 131, the arm 132, and the bucket 133, or may detect inclination angles by measuring the cylinder lengths of the boom cylinder 131C, the arm cylinder 132C, and the bucket cylinder 133C, and converting the cylinder lengths to angles.

[0016] 25 <<Configuration of rotating body 120>> 2023372861   20 Aug 2026 The rotating body 120 comprises an engine 121, a hydraulic pump 122, a control valve 123, a rotation motor 124, an angular velocity sensor 125, a driving cabin 126, imaging devices 127, and a control device 128. The engine 121 is an engine for driving the hydraulic pump 122. The engine 121 5 is an example of a drive power source. The hydraulic pump 122 is a variable-capacity pump driven by the engine 121. The hydraulic pump 122 supplies hydraulic fluid to the respective actuators (the boom cylinder 131C, the arm cylinder 132C, the bucket cylinder 133C, the travel motor 112, and the rotation motor 124) via the control valve 123. 10            The control valve 123 controls the flow rates of the hydraulic fluid supplied from the hydraulic pump 122. The rotation motor 124 is driven by the hydraulic fluid supplied from the hydraulic pump 122 via the control valve 123 to rotate the rotating body 120. The rotation motor 124 comprises the angular velocity sensor 125 for measuring the rotation 15 velocity. The angular velocity sensor 125 may be constituted, for example, by a rotary encoder or a gyrosensor. The angular velocity sensor 125 may, for example, be an inertial measurement unit (IMU).

[0017] The rotating body 120 comprises a driving cabin 126. The driving cabin 126 is 20 provided on the left side of the work equipment 130. The driving cabin 126 comprises a front camera 127F, a left-side camera 127L, a right-side camera 127R, an up-side camera 127U, and a down-side camera 127D. Hereinafter, the front camera 127F, the left-side camera 127L, the right-side camera 127R, the up-side camera 127U, and the down-side camera 127D will also be referred to collectively as imaging devices 127. Captured 25 images from the imaging devices 127 are an example of environmental images showing a 2023372861   20 Aug 2026 direction faced by the rotating body 120 in a peripheral environment of the work machine 100.

[0018] FIG. 3 is a view illustrating the configuration of imaging devices 127 according 5 to the first embodiment. In the first embodiment, the imaging devices 127 are installed on front and upper portions inside the driving cabin 126. The The front camera 127F captures images looking in front of the driving cabin 126. The left-side camera 127L captures images in the left forward direction of the driving cabin 126. The right-side camera 127R captures images in the right forward direction of the driving cabin 126. The 10 up-side camera 127U captures images in the upward forward direction of the driving cabin 126. The down-side camera 127D captures images in the downward forward direction of the driving cabin 126. Examples of the imaging devices 127 include, for example, imaging devices using CCD (Charge Coupled Device) sensors and CMOS (Complementary Metal Oxide Semiconductor) sensors. The work machine 100 according 15 to another embodiment may not comprise a driving cabin 126 in the case in which it is to be driven by remote operation or by unmanned driving. In this case, the imaging devices 127 are installed at the front portion of the rotating body 120. Additionally, in another embodiment, the imaging devices 127 may be installed at a location other than the front portion of the rotating body 120, for example, on a side portion of the rotating body, etc. 20

[0019] The control device 128 transmits measurement data from the various sensors (the angular velocity sensor 125, the imaging devices 127, the boom angle sensor 131S, the arm angle sensor 132S, and the bucket angle sensor 133S) provided on the work machine 100 to the remote driving device 500. The control device 128 may comprise an 25 encoder for encoding the images captured by the imaging devices 127. Additionally, the 2023372861   20 Aug 2026 control device 128 receives operation signals from the remote driving device 500, and drives the work equipment 130, the rotating body 120, or the traveling body 110 based on the received operation signals.

[0020] 5 <<Remote driving device 500>> The remote driving device 500 comprises a driving seat 510, a display device 520, an operation device 530, and a control device 540.

[0021] The display device 520 is disposed in front of the driving seat 510. The display 10 device 520 is positioned to face an operator when the operator is seated on the driving seat 510. The display device 520, as illustrated in FIG. 1, is constituted by a front display unit 520F, a left-side display unit 520L, a right-side display unit 520R, an up-side display unit 520U, and a down-side display unit 520D, which are arrayed. The left-side display unit 520L is provided on the left side of the front display unit 520F. The right-side 15 display unit 520R is provided on the right side of the front display unit 520F. The up-side display unit 520U is provided on the upper side of the front display unit 520F. The downside display unit 520D is provided on the lower side of the front display unit 520F. The display device 520 is arranged to occupy much of the field of view of the operator when the operator is seated on the driving seat 510. When the operator is seated on the driving 20 seat 510 and looks at the front display unit 50F (when the front display unit 520F is in the effective field of view), the left-side display unit 520L, the right-side display unit 520R, the up-side display unit 520U, and the down-side display unit 520D are all located in the operator's peripheral vision.

[0022] 25 In other embodiments, the number of displays constituting the display device 2023372861   20 Aug 2026 520 is not limited to the above. For example, the display device 520 may be constituted by a single display. Alternatively, the display device 520 may project images onto a curved surface or a spherical surface by means of a projector, etc. In this case also, the display device 520 is provided so as to span regions that are in the operator's peripheral 5 vision when the operator, seated on the driving seat 510, looks at the center of the display device 520.

[0023] The operation device 530 is disposed near the driving seat 510. The operation device 530 is located within a range allowing operation by an operator when the operator 10 is seated on the driving seat 510. The operation device 530 comprises, for example, an electric lever and an electric pedal. Due to the operator operating the electric lever and the electric pedal, the operation device 530 outputs operation signals of the boom 131, the arm 132, and the bucket 133, rotation operation signals of the rotating body 120, and traveling operation signals of the traveling body 110. 15

[0024] The control device 540 displays captured images received from the work machine 100 on the display device 520. Specifically, the control device 540 displays captured images from the front camera 127F on the front display unit 520F, displays captured images from the left-side camera 127L on the left-side display unit 520L, 20 displays captured images from the right-side camera 127R on the right-side display unit 520R, displays captured images from the up-side camera 127U on the up-side display unit 520U, and displays captured images from the down-side camera 127D on the downside display unit 520D. The control device 540 is an example of a display control device. Additionally, the control device 540 transmits, to the work machine 100, the 25 operation signals input to the operation device 530. 2023372861   20 Aug 2026

[0025] <<Control device 540 of remote driving device 500>> FIG. 4 is a schematic block diagram illustrating the configuration of the control device of the remote driving device according to the first embodiment. 5           The control device 540 is a computer comprising a processor 910, a main memory 930, storage 950, and an interface 960. The storage 950 stores a program. The processor 910 reads the program from the storage 950, loads it in the main memory 930, and executes processing in accordance with the program.

[0026] 10           Examples of the storage 950 include an HDD (Hard Disk Drive), an SSD (Solid State Drive), a magnetic disk, a magneto-optic disk, a CD-ROM (Compact Disc ReadOnly Memory), a DVD-ROM (Digital Versatile Disc Read-Only Memory), a semiconductor memory, etc. The storage 950 may be internal media directly connected to a common communication line of the control device 540, or may be external media 15 connected to the control device 540 via an interface 960. The storage 950 is a non-transitory, tangible storage medium. In other embodiments, the control device 540 may comprise, in addition to the above-mentioned features or instead of the above-mentioned features, a custom LSI (Large-Scale Integrated Circuit) such as a PLD (Programmable Logic Device) or a semi- 20 custom LSI such as an ASIC (Application-Specific Integrated Circuit). Examples of PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), and FPGA (Field-Programmable Gate Array). In this case, some or all of the functions realized by the processor 910 may be realized by said integrated circuit. 2023372861   20 Aug 2026 The processor 910, by executing the program, comprises a reception unit 911, a decoder 912, a rotation determination unit 913, a bucket position identification unit 914, a display control unit 915, and a transmission unit 916.

[0028] 5           The reception unit 911 receives measurement data from the work machine 100. The received measurement data includes measurement data from the angular velocity sensor 125, captured images from the front camera 127F, captured images from the leftside camera 127L, captured images from the right-side camera 127R, captured images from the up-side camera 127U, captured images from the down-side camera 127D, 10 measurement data from the boom angle sensor 131S, measurement data from the arm angle sensor 132S, and measurement data from the bucket angle sensor 133S. The reception unit 911 is an example of an acquisition unit.

[0029] The decoder 912 decodes the captured images received by the reception unit 15   911. The rotation determination unit 913 determines whether or not the rotating body 120 on the work machine 100 is rotating based on the measurement data from the angular velocity sensor 125 received by the reception unit 911. The rotation determination unit 913, in the case in which the rotating body 120 is determined to be rotating, identifies the 20 rotation direction. Specifically, if the absolute value of the measurement data of the angular velocity sensor 125 exceeds a prescribed threshold value, the rotation determination unit 913 determines that the rotating body 120 is rotating. Additionally, if the sign of the measurement data of the angular velocity sensor 125 is positive, the rotation determination unit 913 determines that the rotating body 120 is rotating to the 25 left, and if the sign of the measurement data of the angular velocity sensor 125 is 2023372861   20 Aug 2026 negative, the rotation determination unit 913 determines that the rotating body 120 is rotating to the right. In other embodiments, the rotation determination unit 913 may determine that the rotating body 120 is rotating when the operation amount of the operation device 530 exceeds a prescribed threshold value. 5

[0030] The bucket position identification unit 914 identifies the position of a blade tip of the bucket 133, with reference to the rotating body 120, based on the measurement data from the boom angle sensor 131S, the measurement data from the arm angle sensor 132S, and the measurement data from the bucket angle sensor 133S received by the 10 reception unit 911. Specifically, the bucket position identification unit 914 acquires the vertical direction component and the horizontal direction component of the axis of the boom 131 based on the measurement data from the boom angle sensor 131S and the known length of the boom 131 (the distance from the pin at the proximal end to the pin at the distal 15 end). The bucket position identification unit 914 determines the vertical direction component and the horizontal direction component of the axis of the arm 132 based on the measurement data from the arm angle sensor 132S and the known length of the arm 132. The bucket position identification unit 914 determines the vertical direction component and the horizontal direction component of the axis of the bucket 133 from the 20 proximal end to the blade tip based on the measurement data from the bucket angle sensor 133S and the known shape of the bucket 133. The bucket position identification unit 914 identifies, as the position of the blade tip of the bucket 133, a position separated from the position of the proximal end of the boom 131 on the work machine 100 by the sum of the vertical direction components and 25 the horizontal direction components of the axes of the boom 131, the arm 132, and the 2023372861   20 Aug 2026 bucket. The bucket position identification unit 914 may identify the position of the proximal end of the bucket 133 instead of the position of the blade tip of the bucket 133.

[0031] The display control unit 915 outputs, to corresponding display units in the 5 display device, display signals of captured images decoded by the decoder 912. The display control unit 915 determines whether or not each captured image requires to be displayed based on the determination results from the rotation determination unit 913 and the position of the blade tip of the bucket 133 identified by the bucket position identification unit 914. 10

[0032] The transmission unit 916 transmits, to the work machine 100, the operation signals input to the operation device 530.

[0033] <<Display control method of control device 540>> 15           The display control method due to the control device 540 according to the first embodiment will be explained with reference to the drawings. FIG. 5 is a flow chart indicating a display control method due to the control device 540 according to the first embodiment. The control device 540 repeatedly executes the process below each time measurement data is received from the work machine 100. 20

[0034] First, the reception unit 911 receives measurement data from the work machine 100 (step S1). The decoder 912 decodes the respective captured images received by the reception unit 911 (step S2). Additionally, the bucket position identification unit 914 identifies the position of the blade tip of the bucket 133 based on the measurement data 25 from the boom angle sensor 131S, the arm angle sensor 132S, and the bucket angle 2023372861   20 Aug 2026 sensor 133S (step S3).

[0035] Next, the rotation determination unit 913 determines whether or not the rotating body 120 of the work machine 100 is rotating based on the measurement data from the 5 angular velocity sensor 125 received by the reception unit 911 (step S4).

[0036] If the rotation determination unit 913 has not determined that the rotating body 120 is rotating (step S4: NO), then the display control unit 915 outputs, to the display device 520, display signals for displaying all of the captured images that have been 10 decoded on corresponding display units (step S5). That is, the display control unit 915 makes the front display unit 520F display the captured images from the front camera 127F, makes the left-side display unit 520L display the captured images from the left-side camera 127L, makes the right-side display unit 520R display the captured images from the right-side camera 127R, makes the up-side display unit 520U display the captured 15 images from the up-side camera 127U, and makes the down-side display unit 520D display the captured images from the down-side camera 127D.

[0037] On the other hand, if the rotation determination unit 913 has determined that the rotating body 120 is rotating (step S4: YES), then the rotation determination unit 913 20 identifies the rotation direction based on the measurement data from the angular velocity sensor 125 (step S6). If the rotation direction is to the left (step S6: Left), then the display control unit 915 determines that the captured images from the right-side camera 127R showing the region on the side opposite to the rotation direction are not to be displayed (step S7). On the other hand, if the rotation direction is to the right (step S6: Right), then 25 the display control unit 915 determines that the captured images from the left-side 2023372861   20 Aug 2026 camera 127L showing the region on the side opposite to the rotation direction are not to be displayed (step S8).

[0038] Next, the display control unit 915 determines whether or not the height of the 5 blade tip of the bucket 133 (the height from the bottom surface of the traveling body 110 to the proximal end of the bucket 133) is lower than a prescribed threshold value (step S9). If the height of the blade tip of the bucket 133 is lower than the prescribed threshold value (step S9: YES), then the display control unit 915 determines that the display control unit 915 will not display captured images from the up-side camera 127U (step 10   S10).

[0039] Furthermore, the display control unit 915 outputs, to the display device 520, display signals for displaying, on the corresponding display units, captured images, among the captured images that have been decoded, other than those that have been 15 determined will not be displayed (step S11).

[0040] <<Functions and effects>> Thus, the control device 540 according to the first embodiment comprises a reception unit 911, a rotation determination unit 913, and a display control unit 915. The 20 reception unit 911 acquires captured images captured by the imaging devices 127. The rotation determination unit 913 determines whether or not the rotating body 120 is rotating. If the rotating body 120 is determined to be rotating, the display control unit 915 outputs, to the display device 520, display signals for displaying images in which partial regions of the captured images on the side opposite to the rotation direction are blocked. 25 FIG. 6 is a view illustrating the difference in the display on the display device 520 before 2023372861   20 Aug 2026 and after initiating left rotation according to the first embodiment.

[0041] In a rotation operation, in order to perform correct operations, an operator is required to look at the captured image in the rotation direction and the captured image in 5 the frontward direction in which the bucket 133 appears. On the other hand, the direction opposite to the rotation direction is not closely viewed by the operator. However, even when it is not closely viewed, if there is a large movement in a portion appearing in the field of view, there is a risk that motion sickness will be induced. Therefore, as illustrated in FIG. 6, when rotating in which the entire captured image changes greatly, the control 10 device 540 according to the first embodiment can prevent the operator from experiencing motion sickness by blocking captured images that are unimportant, while displaying captured images that are important.

[0042] Additionally, the display device 520 of the remote driving device 500 according 15 to the first embodiment is provided so as to span regions that are in the operator's peripheral vision (for example, a range from approximately 30 degrees to 100 degrees from the line of sight) when the operator, seated on the driving seat 510, looks at the center of the display device 520. As a result thereof, the remote driving device 500 can construct an environment close to that when the operator is actually on board the work 20 machine and can reduce the gap between remote operation and actual machine operation. On the other hand, since the display device 520 occupies much of the field of view of the operator, there is a possibility that the operator will experience motion sickness if the entire screen moves in unison. In particular, motion sickness tends to be induced by perceiving large movements in one's peripheral vision. Therefore, the control device 540 25 according to the first embodiment outputs display signals for displaying images in which 2023372861   20 Aug 2026 regions of the captured images that are in the operator's peripheral vision when displayed on the display device 520 and that are on a side opposite to the rotation direction are blocked. As a result thereof, motion sickness can be prevented by preventing motion from occurring in the operator's peripheral vision during rotation. 5

[0043] Additionally, the respective display units in the display device 520 according to the first embodiment display the captured images captured by the corresponding imaging devices 127. In this case, when it is determined that the rotating body 120 is rotating to the right, the control device 540 makes the front display unit 520F display the captured 10 images from the front camera 127F and makes the right-side display unit 520R display the captured images from the right-side camera 127R without making the left-side display unit 520L display the captured images from the left-side camera 127L. Additionally, when it is determined that the rotating body 120 is rotating to the left, the control device 540 makes the front display unit 520F display the captured images from 15 the front camera 127F and makes the left-side display unit 520L display the captured images from the left-side camera 127L without making the right-side display unit 520R display the captured images from the right-side camera 127R. Additionally, when it is determined that the rotating body 120 is not rotating, the control device 540 makes the right-side display unit 520R display the captured images from the right-side camera 20   127R, makes the front display unit 520F display the captured images from the front camera 127F, and makes the left-side display unit 520L display the captured images from the left-side camera 127L. Thus, the control device 540 can prevent motion from occurring in the operator's peripheral vision during rotation, without performing any particular image processing, by determining whether or not any of the captured images 2023372861   20 Aug 2026 Additionally, when it is determined that the rotating body 120 is rotating, the control device 540 according to the first embodiment does not make the up-side display unit 520U display the captured images from the up-side camera 127U, and when it is 5 determined that the rotating body 120 is not rotating, the control device 540 makes the up-side display unit 520U display the captured images from the up-side camera 127U. As a result thereof, the images appearing in the operator's peripheral vision can be further reduced, thereby further lowering the possibility that motion sickness will occur. In particular, when it is determined that the rotating body 120 is rotating, the control device 10   540 according to the first embodiment determines whether or not to display the captured images from the up-side camera 127U based on the position of the work equipment 130. For example, when the bucket 133 is located at a high position, the captured images from the up-side camera 127U become important for checking that the bucket 133 will not come into contact with obstacles located above. By determining whether or not the 15 captured images from the upward camera are important depending on the height of the bucket 133, the control device 540 according to the first embodiment can display the captured images when they are important without displaying them when they are not important.

[0045] 20           There are cases in which the up-side camera 127U is always important regardless of the height of the blade tip of the bucket 133, such as in cases in which there are power lines above. In this case, the captured images from the up-side camera 127U may always be displayed regardless of the height of the blade tip of the bucket 133 by inputting instructions, in advance, to the control device 540 that the up-side camera 127U 2023372861   20 Aug 2026

[0046] <<Other embodiments>> While an embodiment has been explained in detail with reference to the drawings above, the specific configuration is not limited to that described above, and 5 various design changes, etc. are possible. That is, in another embodiment, the order of the above-mentioned processes can be changed as appropriate. Additionally, some of the processes may be executed in parallel. The control device 540 in the above-mentioned embodiment may be constituted by a single computer, or the configuration of the control device 540 may be arranged by 10 being divided between multiple computers, and the multiple computers may cooperate with each other to function as the control device 540. In this case, some of the computers constituting the control device 540 may be installed in the work machine 100, and the other computers may be provided outside the work machine 100.

[0047] 15           For example, in the remote operation system 1 according to another embodiment, the rotation determination unit 913 may be provided in the control device 128 in the work machine 100. In this case, the control device 128 may be configured so that, when it is determined that rotation is occurring, the captured images from the camera capturing images on the side opposite to the rotation direction are not decoded or 20 are not transmitted.

[0048] Additionally, for example, in a remote operation system 1 according to another embodiment, the bucket position identification unit 914 may be provided in the control device 128 in the work machine 100. In this case, the control device 128 may transmit 25 the position of the blade tip of the bucket 133 to the remote driving device 500 instead of 2023372861   20 Aug 2026 the measurement data from the boom angle sensor 131S, the arm angle sensor 132S, and the bucket angle sensor 133S. Additionally, the control device 128 may be configured so as to determine whether or not the height of the blade tip of the bucket 133 is lower than a prescribed threshold value, and so that, when it is determined that the height is equal to 5 or greater than the threshold value, the captured images from the up-side camera 127U are not decoded or are not transmitted.

[0049] The control device 540 according to the above-mentioned embodiment determines whether or not rotation is occurring based on received measurement data 10 from the angular velocity sensor 125. However, there is no limitation thereto. For example, the control device 540 according to another embodiment may determine whether or not rotation is occurring based on operation signals form the operation device 530. In this case, there is a lag caused by communication between operation of the operation device 530 and when the rotation actually starts. For this reason, the control 15 device 540 may determine that the rotating body 120 is rotating after a certain period of time has elapsed from a timing at which the value of the operation signal became large. Additionally, the control device 540 according to another embodiment may determine whether or not the rotating body 120 is rotating by analyzing captured images that have been received. 20

[0050] Additionally, the work machine 100 according to the above-mentioned embodiment comprises multiple cameras. However, there is no limitation thereto. For example, the work machine 100 according to another embodiment may comprise a single wide-angle camera as the imaging device 127, and the captured images from said wide- 25 angle camera may be divided and allocated to the multiple display units. 2023372861   20 Aug 2026 Additionally, the display device 520 in the remote driving device 500 according to the above-mentioned embodiment displays captured images from the image capture devices 127. However, there is no limitation thereto. For example, the display device 520 5 according to another embodiment may display images of the peripheral environment of the work machine 100 in the direction faced by the work machine 100 drawn as a result of simulations of the behavior of the work machine 100. Specifically, the control device 540 arranges three-dimensional models of the work machine 100 and the terrain at a work site in virtual space, and simulates the 10 orientation and the position of the work machine 100 based on measurement data received from the work machine 100. The control device 540 can generate an environmental image in the direction faced by the rotating body 120 in the peripheral environment of the work machine 100 by setting the viewpoint of a rendering camera to be that faced by (for example, in front of the driving cabin) the model of the work 15 machine 100 in the virtual space, setting the line-of-sight direction straight forward from the work machine 100, and rendering the three-dimensional models of the work machine 100 and the terrain of the work site. Additionally, the control device 540 may display the shapes of the traveling body 110 and the work machine 130, drawn with lines or by being filled in, based on the 20 orientation of the three-dimensional model of the work machine 100, so as to be overlaid on captured images or environmental images.

[0052] Additionally, the control device 540 according to the above-mentioned embodiment determines whether or not to display the captured images from the up-side 25 camera 127U based on the height of the blade tip of the bucket 133 during rotation of the 2023372861   20 Aug 2026 rotating body 120. However, there is no limitation thereto. For example, the control device 540 according to another embodiment may determine that, during rotation of the rotating body 120, the captured images from the up-side camera 127U are never to be displayed, or the captured images from the up-side camera 127U are always to be 5 displayed. In this case, the display control unit 915 in the control device 540 determines whether or not the respective captured images require to be displayed without using the position of the blade tip of the bucket 133 identified by the bucket position identification unit 914. Additionally, the control device 540 according to another embodiment may determine that the captured images from the down-side camera 127D are not to be 10 displayed during rotation of the rotating body 120.

[0053] Additionally, the display device 520 in the remote driving device 500 according to the above-mentioned embodiment comprises multiple display units. However, there is no limitation thereto. For example, the display device 520 according to another 15 embodiment may be constituted by a single display. Alternatively, the display device 520 may project images onto a curved surface or a spherical surface by means of a projector, etc. In this case also, the display device 520 is provided so as to span regions that are in the operator's peripheral vision when the operator, seated on the driving seat 510, looks at the center of the display device 520. The control device 540 generates a single combined 20 image by combining the respective captured images from the imaging devices 127 and displays the combined image on the display device 520. When it is determined that the rotating body 120 is rotating, the control device 540 may mask, with black coloring, etc., a partial region of the combined image on the side opposite to the rotation direction, particularly a region located in the peripheral vision of the operator. 25

[0054] 2023372861   20 Aug 2026 Additionally, the blocking of the partial region in the image in the abovementioned embodiment is implemented by not displaying the partial region. However, there is no limitation thereto. For example, in another embodiment, the partial image may be blocked by largely reducing the brightness (blacking out), largely increasing the 5 brightness (whiting out), largely reducing the contrast, or blurring the image in the partial region, thereby reducing the image variation in the region.

[0055] According to the embodiments described above, the display system can prevent motion sickness in an operator. 10 REFERENCE SIGNS LIST

[0056] 1 Remote operation system 100 Work machine 110 Traveling body 15 111 Endless track 112 Travel motor 120 Rotating body 121 Engine 122 Hydraulic pump 20 123 Control valve 124 Rotation motor 125 Angular velocity sensor 126 Driving cabin 127 Imaging device 25 127D Down-side camera 2023372861   20 Aug 2026 127F Front camera 127L Left-side camera 127R Right-side camera 127U Up-side camera 5 128 Control device 130 Work equipment 131 Boom 131C Boom cylinder 131S Boom angle sensor 10 132 Arm 132C Arm cylinder 132S Arm angle sensor 133 Bucket 133C Bucket cylinder 15 133S Bucket angle sensor 500 Remote driving device 510 Driving seat 520 Display device 520D Down-side display unit 20 520F Front display unit 520L Left-side display unit 520R Right-side display unit 520U Up-side display unit 530 Operation device 25 540 Control device 2023372861   20 Aug 2026 910    Processor 911     Reception unit 912    Decoder 913 Rotation determination unit 5 914    Bucket position identification unit 915    Display control unit 916 Transmission unit 930   Main memory 950    Storage 10 960 Interface

Claims

1. A display system used for operating a work machine comprising a rotating body that rotates about a center of rotation, the display system comprising:a display device; and5            a display control device that outputs a display signal for displaying an image onthe display device;wherein the display device is provided so as to span regions that are in an operator's peripheral vision when the operator, seated on a driving seat, looks at a center of the display device, and10           wherein the display control device comprises:an acquisition unit that acquires an environmental image showing a direction faced by the rotating body in a peripheral environment of the work machine,a rotation determination unit that determines whether or not the rotating body is rotating, and15            a display control unit that outputs, to the display device, a display signal fordisplaying an image in which a region of the environmental image that is in the operator's peripheral vision when displayed on the display device and that is on a side opposite to a rotation direction is blocked, for preventing motion sickness induced by perceiving large movement in the operator's peripheral vision, when the rotating body is determined to be20 rotating.

2. The display system according to claim 1, wherein:the work machine comprises an imaging device for imaging a peripheral environment of the work machine; and2023372861   20 Aug 2026environmental image.

3. The display system according to claim 2,wherein the work machine comprises, as the imaging device:5           a front camera that captures images in front of the work machine,a left-side camera that captures images on a left side of the work machine, anda right-side camera that captures images on a right side of the work machine;wherein the display device comprises:a front display unit that is provided in front of the driving seat and that displays10 the captured images from the front camera, which are environmental images,a left-side display unit that is provided on a left side of the front display unit and at a position in the operator's peripheral vision when the operator seated in the driving seat looks at the front display unit, and that displays the captured images from the leftside camera, which are environmental images, and15            a right-side display unit that is provided on a right side of the front display unitand at a position in the operator's peripheral vision when the operator seated in the driving seat looks at the front display unit, and that displays the captured images from the right-side camera, which are environmental images; andwherein the display control unit:20           when the swinging body is determined to be rotating to the right, makes thefront display unit display the captured images from the front camera and makes the rightside display unit display the captured images from the right-side camera, without making the left-side display unit display the captured images from the left-side camera,when the swinging body is determined to be rotating to the left, makes the front2023372861   20 Aug 2026display unit display the captured images from the left-side camera, without making the right-side display unit display the captured images from the right-side camera, andwhen the swinging body is determined to not be rotating, makes the right-side display unit display the captured images from the right-side camera, makes the front5 display unit display the captured images from the front camera, and makes the left-side display unit display the captured images from the left-side camera.

4. The display system according to claim 3,wherein the work machine further comprises an up-side camera that captures10 images from the front upper side of the work machine;wherein the display device further comprises an up-side display unit that is provided on an upper side of the front display unit and at a position in the operator's peripheral vision when the operator seated in the driving seat looks at the front display unit, and that displays the captured images from the up-side camera; and15            wherein the display control unit:does not make the up-side display unit display the captured images from the upside camera when the rotating body is determined to be rotating, andmakes the up-side display unit display the captured images from the up-side camera when the rotating body is determined to not be rotating.20

5. The display system according to claim 3,wherein the work machine further comprises work equipment provided on the rotating body, and an up-side camera that captures images from the front upper side of the work machine;25            wherein the display device further comprises an up-side display unit that is2023372861   20 Aug 2026provided on an upper side of the front display unit and at a position in the operator's peripheral vision when the operator seated in the driving seat looks at the front display unit, and that displays the captured images from the up-side camera; andwherein the display control unit:5           makes the up-side display unit display the captured images from the up-sidecamera when the rotating body is determined to not be rotating, anddetermines whether or not to display the captured images from the up-side camera on the up-side display unit based on a position of the work equipment when the rotating body is determined to be rotating.10

6. A display control method in a display device used for operating a work machine comprising a rotating body that rotates about a center of rotation, and the display device being provided so as to span regions that are in an operator's peripheral vision when the operator, seated on a driving seat, looks at a center of the display device,15 the display control method comprising:acquiring an environmental image showing a direction faced by the rotating body in the peripheral environment of the work machine;determining whether or not the rotating body is rotating; andoutputting, to the display device, a display signal for displaying an image in20 which a region of the environmental image that is in the operator's peripheral vision when displayed on the display device and that is on a side opposite to a rotation direction is blocked, for preventing motion sickness induced by perceiving large movement in the operator's peripheral vision, when the rotating body is determined to be rotating.25

7. 2023372861   20 Aug 2026A remote driving device provided outside a work machine comprising a rotating body that rotates about a center of rotation, the remote driving device comprising:a display device; anda display control device that outputs a display signal for displaying an image on5 the display device;wherein the display device is provided so as to span regions that are in an operator's peripheral vision when the operator, seated on a driving seat, looks at a center of the display device, andwherein the display control device comprises:10           an acquisition unit that acquires an environmental image showing a directionfaced by the rotating body in a peripheral environment of the work machine,a rotation determination unit that determines whether or not the rotating body is rotating, anda display control unit that outputs, to the display device, a display signal for15 displaying an image in which a region of the environmental image that is in the operator's peripheral vision when displayed on the display device and that is on a side opposite to a rotation direction is blocked, for preventing motion sickness induced by perceiving large movement in the operator's peripheral vision, when the rotating body is determined to be rotating.20

8. The display system according to claim 1,wherein the work machine comprises a work equipment provided with the rotating body, andwherein the display control unit outputs the display signal for displaying the2023372861   20 Aug 2026opposite to the rotation direction is blocked, while displaying both an image in the rotation direction and an image in the frontward direction in which the work machine appears, included in the environmental image, when the rotating body is determined to be rotating.5

9. The display system according to claim 1, wherein the display control unit outputs the display signal for displaying the image, in which an amount of change of the image based on the rotation is reduced in the region on the side opposite to the rotation direction.10

10. The display system according to claim 5,wherein the work machine includes a boom, an arm, and a bucket, andwherein when the rotating body is determined to be rotating:the display control unit does not display the captured images captured by the up-15 side camera on the up-side display unit, in a case where a height of a blade tip of the bucket is lower than a predetermined threshold, andthe display control unit displays the captured images captured by the up-side camera on the up-side display unit, in a case where the height of the blade tip of the bucket is higher than or equal to the predetermined threshold.20

Images