Remote machine operation system, visual presentation device, visual presentation method and program

Abstract

A remote machine operation system (100) according to the present disclosure comprises: an image-capture unit (26) that captures images of a remotely operated remote machine (2); and a visual presentation device (1) that receives video captured by the image-capture unit (26) and displays the received video so as to present the video to an operator who operates the remote machine (2). The visual presentation device (1) comprises: a communication situation monitoring unit (11) that detects the communication situation for reception of the video; and a presented video control unit (12) that determines a center visual field area and a peripheral visual field area for a display area in which the video is to be displayed and, on the basis of the communication situation detected by the communication situation monitoring unit (11), determines video parameters related to the transmission rates for center visual field video that is to be displayed in the center visual field area and peripheral visual field video that is to be displayed in the peripheral visual field area.

Description

[Technical Field]

[0001] The present disclosure relates to a remote machine operation system, a visual presentation device, a visual presentation method, and a program that present an image of a remote machine. [Background technology]

[0002] BACKGROUND ART Conventionally, there are devices that present images to an observer of an observation target. Patent Document 1 discloses a stereoscopic image observation device that presents images to both eyes of an observer to provide a stereoscopic image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 9-80353 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 is based on the premise that the video is transmitted directly from the camera that captures the video to the video presentation device, allowing the video data to be used without worrying about capacity. On the other hand, in order for an operator in a remote location away from the camera to observe the video, the video must be transmitted using a communication line. If the technology described in Patent Document 1 is applied to a system in which an operator in a remote location observes the video, stable communication over a wide bandwidth at all times is required, which poses the problem of not being able to present video that is appropriate for the communication conditions.

[0005] The present disclosure has been made in consideration of the above, and aims to provide a remote machine operation system that can present an image suitable for the communication situation. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the object, a remote machine operation system according to the present disclosure includes a camera that captures an image of a remotely operated remote machine, and a visual presentation device that receives an image captured by the camera and displays the received image to present the image to an operator operating the remote machine. The visual presentation device includes a communication status monitoring unit that detects a communication status in receiving the image, and a presentation image control unit that determines a central visual field area and a peripheral visual field area in a display area in which the image is displayed, and determines image parameters related to the transmission rates of the central visual field image to be displayed in the central visual field area and the peripheral visual field image to be displayed in the peripheral visual field area based on the communication status detected by the communication status monitoring unit and the operation status of the remote machine. The central and peripheral vision images are each part of the same image, and the peripheral vision image is the part of the image that is outside the central vision image. do. [Effects of the Invention]

[0007] The remote machine operation system according to the present disclosure has the effect of being able to present an image suited to the communication situation. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating a configuration example of a remote machine operation system according to a first embodiment. [Figure 2] Diagram explaining human visual characteristics [Figure 3] FIG. 1 shows an example of display areas of the image according to the first embodiment, which correspond to the central visual field and the peripheral visual field, respectively. [Figure 4] 10 is a flowchart showing an example of a video control process procedure in the visual presentation device according to the first embodiment. [Figure 5] FIG. 10 is a diagram showing an example of video setting information according to the first embodiment; [Figure 6] FIG. 1 is a diagram showing an example of the configuration of a computer system that realizes the visual presentation device according to the first embodiment. [Figure 7] FIG. 10 is a diagram illustrating a configuration example of a remote machine operation system according to a second embodiment. [Figure 8] FIG. 10 is a diagram illustrating a configuration example of a remote machine operation system according to a third embodiment. [Figure 9] FIG. 13 is a diagram showing an example of video setting information according to the third embodiment. [Figure 10] FIG. 10 is a diagram illustrating a configuration example of a remote machine operation system according to a fourth embodiment. [Figure 11] FIG. 13 is a diagram illustrating a configuration example of a remote machine operation system according to a fifth embodiment. [Figure 12] FIG. 13 is a diagram showing an example of an object detected by the device surroundings situation monitoring device according to the fifth embodiment. [Figure 13] FIG. 13 is a diagram illustrating a configuration example of a remote machine operation system according to a sixth embodiment. [Figure 14] FIG. 13 is a diagram illustrating a configuration example of a remote machine operation system according to a seventh embodiment. [Figure 15] FIG. 13 is a diagram showing a configuration example of a remote machine operation system according to an eighth embodiment. [Figure 16] FIG. 13 is a diagram showing an example of an image displayed by the visual presentation device of the eighth embodiment when the image is not enlarged or reduced. [Figure 17] FIG. 13 is a diagram showing an example of an image displayed by the visual presentation device according to the eighth embodiment when the image is enlarged and reduced. [Figure 18] FIG. 13 is a diagram illustrating a configuration example of a remote machine operation system according to a ninth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, a remote machine operation system, a visual presentation device, a visual presentation method, and a program according to embodiments will be described in detail with reference to the accompanying drawings.

[0010] Embodiment 1 1 is a diagram illustrating a configuration example of a remote machine operation system according to a first embodiment. The remote machine operation system 100 of this embodiment includes a visual presentation device 1 and a remote machine 2. The remote machine 2 is a machine that can be remotely operated by an operator located at a location distant from the remote machine 2. The visual presentation device 1 presents an image acquired by the remote machine 2 to the operator.

[0011] The remote machine 2 may be, for example, a work machine that performs work such as construction, assembly, transportation, packaging, or assistance, or it may be a mobile machine such as a vehicle or drone (unmanned aerial vehicle, underwater drone, or space drone), or it may be a machine that performs both work and movement.

[0012] 1, the remote machine 2 includes a camera unit 21, a communication unit 22, a drive unit 23, a drive control unit 24, and a situation acquisition device 25. As shown in the lower part of FIG. 1, the remote machine 2 is, for example, a construction machine, but as described above, is not limited to this.

[0013] The camera unit 21 includes a photographing unit 26, an image processing unit 27, and a communication unit 28. The photographing unit 26 is a camera that photographs the remotely operated remote machine 2, and specifically, is, for example, a monocular camera. The communication unit 28 communicates with the visual presentation device 1. The image processing unit 27 adjusts the resolution, frame rate, etc. of the image photographed by the photographing unit 26 based on instructions from the visual presentation device 1 received via the communication unit 28, and transmits the adjusted image to the visual presentation device 1 via the communication unit 28.

[0014] The communication unit 22 communicates with the visual presentation device 1. The communication unit 22 receives, for example, control information for controlling the remote machine 2 from the visual presentation device 1, and outputs the received control information to the drive control unit 24. The drive control unit 24 controls the drive unit 23 based on the control information. The drive unit 23 is a variety of actuators, and includes, for example, motors for operating the arms and manipulators of the remote machine 2, and motors for moving the remote machine 2.

[0015] The situation acquisition device 25 is, for example, an acceleration sensor, a force sensor, an arm, or the like, and may also be a camera attached to an arm, manipulator, or the like. It detects the operating status of the remote machine 2 and outputs the detection result to the communication unit 22. For example, the communication unit 22 transmits the detection result detected by the situation acquisition device 25 to the visual presentation device 1. The detection result acquired by the situation acquisition device 25 is used to determine an operating mode indicating the operating status of the remote machine 2, as described below. The operating mode indicating the operating status can be expressed as a combination of the work mode and the movement speed of the remote machine 2. The work mode is determined depending on the type of work status, such as whether the operator focuses on a specific area or performs work while grasping a wider range. If the operating mode is manually set by the operator, the situation acquisition device 25 may not be provided. Also, as described below, if the operating mode is determined from video acquired by the camera unit 21, the situation acquisition device 25 may not be provided.

[0016] In Figure 1, the remote machine 2 is equipped with both a communication unit 22 and a communication unit 28 within the camera unit 21, but it is also possible that the communication unit 28 within the camera unit 21 is not provided, and the image captured by the camera unit 21 is transmitted to the visual presentation device 1 via the communication unit 22, and instructions from the visual presentation device 1 to the camera unit 21 are received via the communication unit 22.

[0017] 1, the camera unit 21 is provided in the remote machine 2, but the camera unit 21 may be provided separately from the remote machine 2. Alternatively, a plurality of camera units 21 may be provided, and the visual presentation device 1 may select which camera unit 21 to present to the operator an image captured by the camera unit 21, or the operator may select which camera unit 21 to present to the operator an image captured by the camera unit 21.

[0018] As shown in FIG. 1, the visual presentation device 1 receives video captured by a camera unit 26 that captures a remote machine 2, and displays the received video to present the video to an operator operating the remote machine 2. Specifically, the visual presentation device 1 communicates with the remote machine 2 to acquire video captured at a remote location where the remote machine 2 is installed and remotely controls the remote machine 2. The communication line between the remote machine 2 and the visual presentation device 1 may be a wireless line, a wired line, or a combination of wireless and wired lines. For example, a communication line such as 5G (5th Generation: 5th generation mobile communication system) or Beyond 5G, which realizes high-capacity, low-latency transmission, may be used, or other communication lines may be used. Furthermore, the communication line may be changed when the location of the remote machine 2 or the visual presentation device 1 is changed. For example, the communication line used may be switched as the remote machine 2 moves.

[0019] The visual presentation device 1 receives video from the camera unit 21 of the remote machine 2 via a communication line and presents the received video to the operator by displaying it on a presentation unit 10 shown in FIG. 1, such as a monitor. However, the visual presentation device 1 is not always able to communicate with the camera unit 21 in a stable broadband communication environment. In this embodiment, the video transmission rate (transmission rate), which is the amount of video data per unit time, is controlled according to the communication situation, taking into account the characteristics of human vision. This makes it possible to provide the operator with video that is suited to the communication situation. Note that video that is suited to the communication situation is, for example, video that reduces the communication bandwidth while also taking into account the characteristics of human vision to suppress a decrease in the operator's visibility.

[0020] FIG. 2 is a diagram illustrating human visual characteristics. As shown in FIG. 2, in general, high-definition images are required in the area corresponding to the central visual field 5, which corresponds to the vicinity of the center of the visual field 4 of the human eye 3, and the peripheral visual field 6 outside the central visual field 5 is less affected by the resolution than the central visual field 5. In other words, as long as the image corresponding to the central visual field 5 is kept at a high resolution, the operator can perceive the image as natural even if the resolution of the image corresponding to the peripheral visual field 6 is lower than that of the image corresponding to the central visual field 5. Therefore, by lowering the resolution of the image corresponding to the peripheral visual field 6 below that of the image corresponding to the central visual field 5, it is possible to present a natural image to the operator while suppressing the bandwidth used for transmitting the image.

[0021] Furthermore, the size of the central visual field 5 changes depending on the situation. For example, the size of the central visual field 5 changes depending on the work being done by the person and the speed at which the person is moving. Therefore, when presenting an image to the operator, it is desirable to set the range of the image corresponding to the central visual field 5 depending on the work being done by the person and the speed at which the person is moving. For this reason, in this embodiment, a mode depending on the operating situation, such as the work being done by the remote machine 2 and the speed at which the remote machine 2 is moving, is defined as an operating mode, and the size of the image corresponding to the central visual field 5 is set depending on the operating mode.

[0022] FIG. 3 is a diagram showing an example of an image display area according to the present embodiment, corresponding to the central visual field and the peripheral visual field. As shown in FIG. 3, in this embodiment, the image display area 201 presented by the visual display device 1, i.e., displayed by the visual display device 1, is divided into a central visual field area 202 in which a central visual field image corresponding to the central visual field 5 is displayed, and a peripheral visual field area 203 in which a peripheral visual field image corresponding to the peripheral visual field 6 is displayed. As shown in FIG. 3, the size of the central visual field area 202 is changed depending on the operation mode, i.e., the operating status of the remote machine 2. Note that in FIG. 3, it is assumed that the display unit 10 of the visual display device 1 is a dome-shaped projection device, and the image display area 201 is shown as a circle (within a range of a specified angle from the direction of the operator's line of sight). However, the image display area 201 is not limited to this shape. For example, the image display area 201 may be rectangular, and even if the image display area 201 is rectangular, the central visual field area 202 will be circular.

[0023] 3 illustrates an example in which the operator's line of sight is at the center of the image display area 201, but the location of the central visual field area 202 moves depending on where in the image display area 201 the operator's line of sight corresponds. The operator's line of sight is set by the operator as, for example, a gaze point, as will be described later. The gaze point may be determined in advance depending on the conditions under which the visual presentation device 1 is used. For example, in an environment in which the position of the operator's face is approximately fixed and the position of the work object is also fixed, the gaze point may be determined in advance.

[0024] In this embodiment, the visual presentation device 1 determines video parameters according to the communication conditions while taking into account the above-described human visual characteristics. The video parameters include, for example, at least one of resolution, frame rate, color, and the position and size of the central visual field 202. For example, when an operator focuses on a specific object in the video captured by the camera unit 21, the operator can perceive the video as natural, even if the video transmission rate of the peripheral visual field video is reduced by lowering the resolution of the peripheral visual field video below that of the central visual field video or limiting color information. For this reason, the visual presentation device 1 monitors the communication conditions and, for example, when a sufficient transmission rate is available, displays high-resolution color video at a high frame rate across the entire video display area 201. When the transmission rate decreases, the video transmission rate is reduced by changing at least one of the resolution, frame rate, color, and size of the central visual field 202. This allows the operator to view natural video while reducing the bandwidth used for video transmission.

[0025] Returning to the explanation of Figure 1, the visual presentation device 1 includes a presentation unit 10, a communication status monitoring unit 11, a presentation image control unit 12, a remote machine status monitoring unit 13, a control information generation unit 14, a communication unit 15, a gaze point setting device 16, and an operation input device 17.

[0026] The communication unit 15 communicates with the remote machine 2. For example, the communication unit 15 receives video from the remote machine 2 and transmits control information for controlling the remote machine 2 to the remote machine 2. When the communication unit 15 receives a detection result from the status acquisition device 25 of the remote machine 2, it outputs the received detection result to the presentation image control unit 12. The communication status monitoring unit 11 monitors the communication status in the communication unit 15 and notifies the presentation image control unit 12 of information indicating the communication information. The communication status monitoring unit 11 may, for example, monitor the transmission speed (communication bandwidth) in the communication unit 15, or may monitor communication quality such as the bit error rate and received signal strength, or may monitor both. The communication status monitoring unit 11 also monitors the communication status of at least the direction in which information is received from the remote machine 2, that is, the downstream communication in the visual presentation device 1. That is, the communication status monitoring unit 11 detects at least the communication status in receiving video. The communication status monitoring unit 11 may monitor both the downstream and upstream communication statuses. The communication status detected by the communication status monitoring unit 11 is preferably an accurate measured value, but may be an estimated value if it is difficult to detect an accurate measured value.

[0027] The remote machine status monitoring unit 13 detects the operating status of the remote machine 2. For example, the remote machine status monitoring unit 13 detects at least one of the movement speed of the remote machine 2 and the distance between the remote machine 2 and an object operated by the remote machine 2. In detail, the remote machine status monitoring unit 13 receives video captured by the photographing unit 26 of the camera unit 21 via the communication unit 15, monitors the status of the remote machine 2 using the received video, and notifies the presentation image control unit 12 of information indicating the status of the remote machine 2. The information indicating the status of the remote machine 2 is, for example, information indicating the movement speed of the remote machine 2 estimated from the video, the position of a work object on the remote machine 2 estimated from the video, etc. The work object may be, for example, set in advance, or may be extracted by the remote machine status monitoring unit 13 based on specified conditions. For example, the remote machine status monitoring unit 13 detects the position of the work object by detecting the work object from the video based on an image showing the work object. Furthermore, if the remote machine status monitoring unit 13 can calculate the distance between the work object and the remote machine 2 based on the size of the detected work object or the like, it may estimate the distance and notify the presentation image control unit 12 of the distance as information indicating the status of the remote machine 2. Note that the status acquisition device 25 in the remote machine 2 may also be configured to acquire the status of the remote machine 2. situation This is another example of a monitoring unit. That is, the remote machine 2 is configured to detect the operation status of the remote machine 2. situation The monitoring unit may be located on the remote machine 2. 、 They may be provided in the visual presentation device 1. The remote machine status monitoring unit 13 and the status acquisition device 25 may detect, as the operation status, the relative distance and relative speed between the remote machine 2 being operated and moving objects or obstacles (including the remote machine 2 itself being operated) around the remote machine 2 being operated (which are not the target of operation).

[0028] The gaze point setting device 16 sets the gaze point of the operator. Specifically, the gaze point setting device 16 receives input of the gaze point from the operator and notifies the presented image control unit 12 of information indicating the input gaze point. The gaze point setting device 16 may be, for example, an input device such as a joystick, touchpad, button, keyboard, mouse, or game controller, or a device that receives input by detecting the operator's gestures based on a captured image of the operator or the operator's muscle movements. The gaze point setting device 16 may be any input device that can receive input from the operator and is not limited to the above-mentioned examples. The gaze point may be specified as a position in the image display area 201 displayed by the presentation unit 10, or as an object in the image displayed in the image display area 201. When the gaze point is specified as an object, the image processing unit 27 of the camera unit 21 receives information indicating the gaze point from the visual presentation device 1 via the communication unit 28, extracts the object from the image, and determines the position of the gaze point.

[0029] The presented image control unit 12 sets an operation mode and determines image parameters to instruct the camera unit 21 using the set operation mode, information indicating the communication status received from the communication status monitoring unit 11, and information indicating the gaze point received from the gaze point setting device 16. For example, the presented image control unit 12 sets the operation mode using at least one of the detection results transmitted from the status acquisition device 25 received from the communication unit 15 and information indicating the status of the remote machine 2 received from the remote machine status monitoring unit 13. Alternatively, the presented image control unit 12 may set the operation mode based on an input from an operator via the gaze point setting device 16 or the operation input device 17. The presented image control unit 12 determines the central visual field area 202 based on the gaze point set by the gaze point setting device. That is, the presented image control unit 12 determines the central visual field area 202 and the peripheral visual field area 203 of the display area where the image is displayed. The presented image control unit 12 also determines image parameters related to the transmission rate (image transmission rate) of the central visual field image and the peripheral visual field image based on the communication status detected by the communication status monitoring unit 11. The video parameters relating to the transmission rate are video parameters that affect the transmission rate.

[0030] The image parameters include, for example, as described above, at least one of resolution, frame rate, color, and the position and size of the central visual field area 202. The presentation image control unit 12 outputs the determined image parameters to the communication unit 15, and the communication unit 15 transmits the image parameters received from the presentation image control unit 12 to the camera unit 21. Note that, of these image parameters, only the parameters to be changed may be transmitted when the image parameters are changed, or all parameters may be transmitted when any parameter is changed, or all parameters may be transmitted periodically. The camera unit 21 transmits an image to the visual presentation device 1 in accordance with the image parameters instructed by the presentation image control unit 12. When the presentation image control unit 12 receives an image from the camera unit 21 via the communication unit 15, the presentation image control unit 12 displays the image on the presentation unit 10.

[0031] The presentation unit 10 is a device that displays the image captured by the camera unit 21, and may be, for example, a dome-type monitor, a display, a head-mounted display, a terminal such as a tablet or smartphone, a projector, a screen, or the like, as long as it is capable of displaying an image, but is not limited to these. If an image presentation device that can see up and down, such as a dome-type monitor, is used as the presentation unit 10, the operator can ensure a field of view in the up and down directions. For example, if the remote machine 2 is a moving machine, it becomes easier for the operator to check the surroundings below the remote machine 2, i.e., the feet of the remote machine 2, which is useful for supporting the remote operation of the movement of the remote machine 2.

[0032] The operation input device 17 is an input means for accepting input of an operation of the remote machine 2 from an operator, and may be an input means for accepting an operation by detecting the operator's gestures, or may be an input means such as a joystick, touchpad, button, keyboard, mouse, or game controller. The operation input device 17 outputs operation information indicating the operation accepted from the operator to the control information generation unit 14. In FIG. 1 , the gaze point setting device 16 is illustrated separately from the operation input device 17, but the gaze point setting device 16 may be the same as the operation input device 17 or may be a part of the operation input device 17.

[0033] The control information generation unit 14 generates control information for controlling the operation of the remote machine 2 using the operation information received from the operation input device 17, and outputs the generated control information to the communication unit 15. The control information is transmitted to the remote machine 2 by the communication unit 15.

[0034] Next, the video control of this embodiment will be described. Fig. 4 is a flowchart showing an example of a video control processing procedure in the visual presentation device 1 of this embodiment. As shown in Fig. 4, the visual presentation device 1 sets an operation mode (step S1). In detail, as described above, for example, the presentation video control unit 12 may set the operation mode using at least one of the detection result transmitted from the status acquisition device 25 and information indicating the status of the remote machine 2 received from the remote machine status monitoring unit 13, or the operation mode may be set by input by the operator.

[0035] The visual presentation device 1 sets the image parameters based on the operation mode (step S2). Specifically, the presentation image control unit 12 sets the image parameters based on the operation mode. For example, the presentation image control unit 12 holds initial values ​​of the image parameters for each operation mode as image setting information, and sets the image parameters according to the operation mode by referring to the image setting information.

[0036] FIG. 5 is a diagram showing an example of the image setting information of this embodiment. In the example shown in FIG. 5, each row of the image setting information indicates one operation mode, and the operation mode is indicated by a combination of a work mode and a movement speed (the movement speed of the remote machine 2). The image setting information includes initial values ​​of image parameters corresponding to each operation mode. In FIG. 5, the image setting information includes the resolution, the size of the central field of view image (the size of the central field of view region 202), and the frame rate as the initial values ​​of the image parameters. The image parameters included in the image setting information are not limited to these. Furthermore, the initial values ​​of the image parameters may include information indicating whether the image is color or monochrome as color information. Note that the image parameters in the image setting information are set to values ​​corresponding to, for example, the best communication conditions, i.e., when there are no transmission speed constraints. Here, the initial values, such as the resolution and frame rate, are assumed to be the same for the central field of view region 202 and the peripheral field of view region 203, but the initial values ​​may be different for the central field of view region 202 and the peripheral field of view region 203. In this case, the image setting information includes values ​​for the central visual field 202 and the peripheral visual field 203 as image parameters. The image setting information is determined, for example, through experiments or the like, so that the operator feels comfortable working in each operation mode, but the method for determining the image setting information is not limited to this example. For example, the image setting information can also be determined by learning from data during operation.

[0037] Returning to the explanation of FIG. 4, the visual presentation device 1 detects the communication status (step S3). More specifically, the communication status monitoring unit 11 detects the communication status and notifies the presentation image control unit 12 of the detection result. Next, the visual presentation device 1 determines whether the communication status has changed (step S4). More specifically, the presentation image control unit 12 determines whether the communication status has changed. Note that in the first step S4, the image parameters are set assuming that the communication status is the best, so the presentation image control unit 12 determines whether the communication status has changed from the communication status in the best case. In the second or subsequent step S4, the presentation image control unit 12 determines whether the communication status has changed from the communication status at the time of the previous determination in step S4. The determination of whether the communication status has changed is made, for example, by determining whether the transmission speed has changed by a certain amount or more. For example, the transmission speed may be defined in multiple ranges, such as level 1 when the transmission speed is greater than or equal to Z1 (Mbps) and less than Z2 (Mbps), and level 2 when the transmission speed is greater than or equal to Z2 (Mbps) and less than Z3 (Mbps), and if there is a change in the level, it may be determined that there has been a change in the communication conditions.

[0038] If the communication situation has changed (step S4 Yes), the visual presentation device 1 changes the image parameters (step S5). In detail, the presentation image control unit 12 changes the image parameters so that the image parameters correspond to the communication situation. The image parameters to be changed include, for example, the resolution of the peripheral vision image, the frame rate of the peripheral vision image, the color of the peripheral vision area, and the color of the central vision area 20. 2The video parameter includes at least one of the magnitudes of the video parameters. For example, a change priority is assigned to each of these video parameters for each operation mode, and the presentation video control unit 12 changes the video parameters in descending order of change priority until the video transmission rate becomes possible at the current transmission speed. For example, if the priority of changing the resolution is highest in a certain operation mode, the presentation video control unit 12 reduces the resolution of the peripheral visual field video so that the video transmission rate becomes possible at the current transmission speed. Note that a lower limit value for each video parameter when reducing the resolution is assigned. If the video transmission rate is not possible at the current transmission speed even after reducing the resolution of the peripheral visual field video to the lower limit value, the presentation video control unit 12 changes the video parameter with the next highest priority (change priority). In this way, by lowering the video transmission rate by changing the video parameters in descending order of change priority, the presentation video control unit 12 determines the video parameters so that the video transmission rate becomes appropriate for the communication conditions. Note that if the communication conditions improve, the presentation video control unit 12 may similarly change the video parameters so that the video transmission rate increases to become appropriate for the communication conditions.

[0039] Next, the visual presentation device 1 determines whether the operation mode has changed (step S6). Specifically, the presentation image control unit 12 determines the operation mode using at least one of the detection result transmitted from the status acquisition device 25 and information indicating the status of the remote machine 2 received from the remote machine status monitoring unit 13, and determines whether the determined operation mode differs from the currently set operation mode. Alternatively, the presentation image control unit 12 determines whether the operation mode input by the operator differs from the currently set operation mode.

[0040] If the operation mode has not changed (step S6 No), the presentation image control unit 12 performs the process again from step S3. If the operation mode has changed (step S6 Yes), the presentation image control unit 12 determines whether the work has ended (step S7). In detail, the presentation image control unit 12 may determine that the work has ended when there is an input from the operator indicating that the work has ended, or may determine that the work has ended when the operation mode includes a mode indicating that the work has ended and the changed operation mode is a mode indicating that the work has ended.

[0041] If the task is not completed (step S7 No), the presentation image control unit 12 sets the image parameters based on the operation mode (step S8) and repeats the process from step S3. In step S8, the presentation image control unit 12 may set the image parameters based on the image setting information, as in step S2, or may change some of the image parameters indicated by the image setting information. For example, in step S8, only the size of the central visual field 202, among the image parameters indicated by the image setting information, may be changed to a value corresponding to the changed operation mode.

[0042] If the work has been completed (Yes in step S7), the presentation image control unit 12 ends the process. If No in step S4, the presentation image control unit 12 advances the process to step S6.

[0043] Through the above processing, image parameters are set according to the operation mode and communication status. Note that the processing shown in FIG. 4 is an example, and the specific processing procedure is not limited to the example shown in FIG. 4. Furthermore, in the above example, the image setting information includes the initial values ​​of image parameters for the best communication status for each operation mode, but image setting information may be determined for each communication status. For example, image setting information may be determined for each level of the above-mentioned communication status, and in step S2, presentation image control unit 12 may set image parameters using image setting information according to the communication status.

[0044] In the above example, an image captured by the remote machine 2 has been described as an example. However, the image presented by the visual presentation device 1 is not limited to an image captured by the remote machine 2, and may be a landscape, person, animal, or other image captured by a camera. For example, a camera may be installed on a person's shoulder, in a forest, a river, or in outer space, and the communication unit 15 of the visual presentation device 1 may receive the image from the camera. In this case, too, the image parameters of the central field of view image and the peripheral field of view image may be set according to the communication conditions, for example, by lowering the image transmission rate of the peripheral field of view image when the communication conditions deteriorate. In the case where the camera is mounted on a person's shoulder, the image parameters of the central field of view image and the peripheral field of view image may be further determined according to the person's moving speed.

[0045] Next, the hardware configuration of the visual presentation device 1 of this embodiment will be described. In the visual presentation device 1 of this embodiment, a computer program that describes the processing in the visual presentation device 1 is executed on a computer system, causing the computer system to function as the visual presentation device 1. FIG. 6 is a diagram showing an example configuration of a computer system that realizes the visual presentation device 1 of this embodiment. As shown in FIG. 6, this computer system includes a control unit 101, an input unit 102, a memory unit 103, a display unit 104, a communication unit 105, and an output unit 106, which are connected via a system bus 107. The control unit 101 and the memory unit 103 form a processing circuit.

[0046] In FIG. 6, the control unit 101 is a processor such as a CPU (Central Processing Unit) that executes a program describing the processing of the visual presentation device 1 of this embodiment. Note that a portion of the control unit 101 may be implemented by dedicated hardware such as a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array). As described above, the input unit 102 may be an input device such as a joystick, touchpad, button, keyboard, mouse, or game controller, or may be an input device that detects an operator's gestures based on a captured image of the operator or the operator's muscle movements and accepts input. The storage unit 103 includes various types of memory such as RAM (Random Access Memory) and ROM (Read Only Memory) and a storage device such as a hard disk, and stores the program to be executed by the control unit 101, necessary data obtained during processing, and the like. The storage unit 103 is also used as a temporary storage area for programs. As described above, the display unit 104 is, for example, a dome-shaped monitor, a display, a head-mounted display, a terminal such as a tablet or a smartphone, a projector, a screen, etc. The display unit 104 and the input unit 102 may be integrated and realized by a touch panel, etc. The communication unit 105 is a receiver and a transmitter that perform communication processing. The output unit 106 is, for example, a speaker. Note that FIG. 6 is an example, and the configuration of the computer system is not limited to the example of FIG. 6.

[0047] Here, an example of the operation of the computer system until the program of this embodiment is ready to be executed will be described. In the computer system having the above-described configuration, for example, a computer program is installed in storage unit 103 from a CD-ROM or DVD-ROM inserted in a CD (Compact Disc)-ROM drive or DVD (Digital Versatile Disc)-ROM drive (not shown). Then, when the program is executed, the program read from storage unit 103 is stored in the main storage area of ​​storage unit 103. In this state, control unit 101 executes processing as visual presentation device 1 of this embodiment in accordance with the program stored in storage unit 103.

[0048] In the above description, a program describing the processing in the visual presentation device 1 is provided using a CD-ROM or DVD-ROM as a recording medium, but this is not limited to this. Depending on the configuration of the computer system, the capacity of the program to be provided, etc., it is also possible to use a program provided via a transmission medium such as the Internet.

[0049] For example, the program of this embodiment causes the visual presentation device 1 to execute the steps of detecting the communication status for receiving video, dividing the display area for displaying the video into a central visual field area 202 and a peripheral visual field area 203, and determining video parameters related to the transmission rates of the central visual field image to be displayed in the central visual field area 202 and the peripheral visual field image to be displayed in the peripheral visual field area 203 based on the detected communication status.

[0050] The communication status monitoring unit 11, the presentation image control unit 12, the remote machine status monitoring unit 13, and the control information generation unit 14 shown in FIG. 1 are realized by the control unit 101 shown in FIG. 6 executing a computer program stored in the storage unit 103 shown in FIG. 6. The storage unit 103 shown in FIG. 6 is also used to realize the communication status monitoring unit 11, the presentation image control unit 12, the remote machine status monitoring unit 13, and the control information generation unit 14 shown in FIG. 1. The communication unit 15 shown in FIG. 1 is realized by the communication unit 105 shown in FIG. 6. The gaze point setting device 16 and the operation input device 17 shown in FIG. 1 are realized by the input unit 102 shown in FIG. 6. The presentation unit 10 shown in FIG. 1 is realized by the display unit 104 shown in FIG. 6.

[0051] 1, the gaze point setting device 16, the operation input device 17, and the presentation unit 10 are included in the visual presentation device 1, but at least one of the gaze point setting device 16, the operation input device 17, and the presentation unit 10 may not be included in the visual presentation device 1. In this case, the visual presentation device 1 exchanges information by connecting to at least one of the gaze point setting device 16, the operation input device 17, and the presentation unit 10 that are not included in the visual presentation device 1. Also, in the example shown in FIG. 1, the communication status monitoring unit 11 and the presentation image control unit 12 are included in the visual presentation device 1, but either one or both may be included in the remote machine 2.

[0052] The video processing unit 27 in the remote machine 2 shown in Fig. 1 is realized by, for example, a processing circuit including the control unit 101 and memory unit 103 shown in Fig. 6. This processing circuit is also a computer system. The communication unit 22 and the communication unit 28 shown in Fig. 1 are a receiver and a transmitter.

[0053] As described above, the visual presentation device 1 of this embodiment divides the video display area 201 into the central visual area 202 and the peripheral visual area 203, and determines the video parameters to be displayed in each area depending on the operating status of the remote machine 2 and the communication status. This makes it possible to present a video that is suited to the communication status. In particular, when the actual transmission speed is higher than the video transmission rate corresponding to the currently set video parameters, low If the transmission rate is high, the image parameters are determined to reduce the image transmission rate of the peripheral vision image, thereby suppressing the bandwidth used for image transmission and presenting a natural image to the operator.

[0054] Embodiment 2 7 is a diagram showing an example of the configuration of a remote machine operation system according to the second embodiment. A remote machine operation system 100a according to this embodiment includes a visual presentation device 1a and a remote machine 2 similar to that of the first embodiment. Components having the same functions as those of the first embodiment are given the same reference numerals as those of the first embodiment, and redundant explanations will be omitted. Below, differences from the first embodiment will be mainly explained.

[0055] The visual presentation device 1a is similar to the visual presentation device 1 of the first embodiment, except that it includes a gaze measurement device 18, which is another example of a gaze point setting device, instead of the gaze point setting device 16. The gaze measurement device 18, which is an example of a gaze point setting device, measures the gaze (direction of gaze) of the operator and outputs the measurement result to the presentation image control unit 12. The gaze is information indicating the gaze point, and the presentation image control unit 12 determines the gaze point based on the measurement result and determines the central visual field area 202 using the gaze point. Note that the gaze measurement device 18 may not be included in the visual presentation device 1a, and the visual presentation device 1a may acquire the measurement result from the gaze measurement device 18. The visual presentation device 1a also includes the gaze point setting device 16, and the gaze point setting device 16 and It may be possible to switch between the line measuring device 18. The operation of this embodiment other than that described above is the same as that of the first embodiment.

[0056] The visual presentation device 1a of this embodiment determines the gaze point using the line-of-sight measurement device 18, so that the operator does not need to set the gaze point, and the burden on the operator can be reduced.

[0057] Embodiment 3 8 is a diagram showing an example of the configuration of a remote machine operation system according to a third embodiment. A remote machine operation system 100b according to this embodiment includes a visual presentation device 1 similar to that of the first embodiment, and a remote machine 2a. Components having the same functions as those of the first embodiment are given the same reference numerals as those of the first embodiment, and redundant explanations will be omitted. Below, differences from the first embodiment will be mainly explained.

[0058] The remote machine 2a is similar to the remote machine 2 of the first embodiment except that it has a camera unit 21a instead of the camera unit 21. The camera unit 21a has a compound eye photographing unit 29 and an image processing unit 27a instead of the photographing unit 26 and the image processing unit 27. 21 The compound eye photographing unit 29 is a compound eye camera. The compound eye photographing unit 29 is, for example, a trinocular camera, and can provide a stereoscopic image and a monoscopic image by photographing a left eye image, a right eye image, and a monoscopic image. Alternatively, the compound eye photographing unit 29 may photograph a left eye image and a right eye image, and the monoscopic image may be created using the left eye image and the right eye image. The communication unit 28 transmits the image photographed by the compound eye photographing unit 29 to the visual presentation device 1.

[0059] In this embodiment, the image parameters may include the presence or absence of stereoscopic vision, i.e., whether the image is stereoscopic or monoscopic. More specifically, for example, the image parameters may include information indicating whether the central field of view image and the peripheral field of view image are stereoscopic. FIG. 9 is a diagram showing an example of image setting information in this embodiment. For example, when the remote machine 2a is operated in close proximity to an object, using stereoscopic vision makes it easier for the operator to grasp the distance between the remote machine 2a and the object, facilitating the task of avoiding interference between the remote machine 2a and the object. On the other hand, when the remote machine 2a is a vehicle or the like moving at high speed, the need for stereoscopic vision is low. Furthermore, for example, depending on the operating mode, if the central field of view image is stereoscopic, the operator can perceive a natural image without feeling uncomfortable, even if the peripheral field of view image is monoscopic.

[0060] In this embodiment, the presentation image control unit 12 may change the presence or absence of stereoscopic viewing as an image parameter depending on the communication conditions. For example, if the transmission speed decreases when a stereoscopic image is displayed over the entire image display area 201, the peripheral vision image may be changed to a monoscopic image, thereby lowering the image transmission rate of the peripheral vision image. Furthermore, the presentation image control unit 12 may add the presence or absence of stereoscopic viewing to the image parameters described in the first embodiment, and by determining the priority of changing each image parameter as described in the first embodiment, change the image parameters in descending order of priority so that the image transmission rate is appropriate for the communication conditions.

[0061] When the image processing unit 27a of the camera unit 21a receives an instruction from the visual presentation device 1 to change whether or not to enable stereoscopic vision, the image processing unit 27a adjusts the image to be output to the communication unit 28 based on the instruction. For example, when the image processing unit 27a receives an instruction to set the central field of view image as a stereoscopic image and the peripheral field of view image as a monocular image, the image processing unit 27a sets the portion corresponding to the central field of view image as a stereoscopic image and the portion corresponding to the peripheral field of view image as a monocular image. As described above, the compound eye imaging unit 29 may capture left eye images and right eye images, and the image processing unit 27a may use the left eye images and right eye images to estimate and create a monocular image corresponding to an image intermediate between the two. Alternatively, the compound eye imaging unit 29 may include a compound eye camera for stereoscopic vision and a monocular camera for monocular vision, and the image processing unit 27a may use the images captured by these two types of cameras as the stereoscopic image and the monocular image.

[0062] The presentation unit 10 of the visual presentation device 1 may be anaglyph, polarized glasses, active shutter glasses, a head-mounted display, a lenticular display, or the like. The stereoscopic images from multiple cameras may include images synthesized using an image synthesis function. Other than the above, the operation of this embodiment is the same as that of the first embodiment.

[0063] The remote machine operation system 100b of this embodiment can present a stereoscopic image to the operator because it uses the compound eye imaging unit 29. Furthermore, by making the central visual field image a stereoscopic image and the peripheral visual field image a monocular image, the image transmission rate can be reduced compared to when the stereoscopic image is displayed over the entire image display area 201.

[0064] Embodiment 4 FIG. 10 is a diagram showing an example of the configuration of a remote machine operation system according to a fourth embodiment. A remote machine operation system 100c according to this embodiment includes a visual presentation device 1 similar to that of the first embodiment, and a remote machine 2b. Components having the same functions as those of the first and third embodiments are given the same reference numerals as those of the first and third embodiments, and redundant explanations will be omitted. Below, differences from the first and third embodiments will be mainly explained.

[0065] Remote machine 2b is similar to remote machine 2a of embodiment 3, except that it has camera unit 21b instead of camera unit 21a. Camera unit 21b is similar to camera unit 21a of embodiment 3, except that camera drive unit 30 is added.

[0066] The camera driver 30 controls at least one of the orientation of the compound eye photographing unit 29, i.e., the direction of the camera, the distance of the compound eye photographing unit 29 (the distance between the two cameras of the compound eye camera), and the convergence angle. If the camera driver 30 can change the orientation of the compound eye photographing unit 29, the camera driver 30 receives, for example, information indicating the orientation of the compound eye photographing unit 29 specified by input to the visual presentation device 1 through an operator's operation from the communication unit 22, and changes the orientation of the compound eye photographing unit 29 based on the received information. This allows the orientation of the compound eye photographing unit 29 to be set in response to the operator's request. If the camera driver 30 can change the distance of the compound eye photographing unit 29, the camera driver 30 receives, for example, information indicating the distance of the compound eye photographing unit 29 input to the visual presentation device 1 through an operator's operation from the communication unit 22, and changes the distance of the compound eye photographing unit 29 based on the received information. This allows the depth of the stereoscopic image to be set in response to the operator's request. The presented image controller 12 may also set the distance of the compound eye photographing unit 29 depending on the operating mode. For example, the image parameters may include the distance of the compound eye imaging unit 29. This allows an appropriate depth resolution to be set depending on the operating mode. Furthermore, if the camera driving unit 30 is capable of changing the convergence angle of the compound eye imaging unit 29, the camera driving unit 30 receives, for example, information indicating the convergence angle of the compound eye imaging unit 29 input to the visual presentation device 1 by an operator's operation from the communication unit 22, and changes the convergence angle of the compound eye imaging unit 29 based on the received information. Adjusting the convergence angle may make stereoscopic vision easier, but not adjusting the convergence angle may result in higher depth perception resolution. Therefore, by making the convergence angle settable by the operator's operation, an image that meets the operator's needs can be provided. Other than the above, the operation of this embodiment is the same as that of embodiment 1.

[0067] 10 shows an example in which the gaze point setting device 16 is used, but the gaze measurement device 18 described in the second embodiment may be used instead of the gaze point setting device 16. Furthermore, when a monocular camera is used as in the first embodiment, a camera driving unit 30 may be added so that the orientation of the image capturing unit 26 can be changed.

[0068] Embodiment 5 FIG. 11 is a diagram showing a configuration example of a remote machine operation system according to a fifth embodiment. A remote machine operation system 100d according to this embodiment includes a visual presentation device 1b and a remote machine 2b similar to that of the fourth embodiment. Components having the same functions as those of the first and fourth embodiments are given the same reference numerals as those of the first and fourth embodiments, and redundant explanations will be omitted. Below, differences from the first and fourth embodiments will be mainly explained.

[0069] The visual presentation device 1b is similar to the visual presentation device 1 of the first embodiment, except that it further includes a device surroundings monitoring device 19. The device surroundings monitoring device 19 detects whether an object exists in the operator's field of view corresponding to the central field of view image. For example, the device surroundings monitoring device 19 detects whether an object exists between the operator and the presentation unit 10 on which the image is displayed, and in an area corresponding to the central field of view area 202. The device surroundings monitoring device 19 may be, for example, a camera capturing an image between the operator and the presentation unit 10 and an image processing device, or may be a distance sensor. When the device surroundings monitoring device 19 is a camera and an image processing device, the image processing device determines whether an object exists in a position corresponding to the central field of view area 202 between the operator and the presentation unit 10 from the image captured by the camera, and outputs the determination result as a detection result to the presentation image control unit 12.

[0070] 12 is a diagram showing an example of an object detected by the device surroundings situation monitoring device 19 of this embodiment. In the example shown in FIG. 12, an object 211 is present in an area corresponding to the central visual field area 202 between the operator 210 and the presentation unit 10. When a stereoscopic image is displayed as the central visual field image, if the presentation image control unit 12 determines, based on the detection result of the device surroundings situation monitoring device 19, that the object 211 is present in the area corresponding to the central visual field area 202 between the operator 210 and the presentation unit 10, the presentation image control unit 12 reduces the resolution of the peripheral visual field image. This is based on experimental results showing that, although the operator 210 loses stereoscopic vision when the object 211 is present in the area corresponding to the central visual field area 202 between the operator 210 and the presentation unit 10 based on the detection result of the device surroundings situation monitoring device 19, the operator 210 can continue to have stereoscopic vision if the resolution of the peripheral visual field image is low. Therefore, when a stereoscopic image is displayed as the central field of view image, if the presented image control unit 12 determines based on the detection result of the device surroundings situation monitoring device 19 that an object 211 exists in the area corresponding to the central field of view area 202 between the operator 210 and the presentation unit 10, the presented image control unit reduces the resolution of the peripheral field of view image, thereby enabling the operator to continue stereoscopic viewing of the central field of view image. In other words, when the central field of view image is a stereoscopic image, if the device surroundings situation monitoring device 19 detects that an object exists, the presented image control unit reduces the resolution of the peripheral field of view image. Note that the area corresponding to the central field of view area 202 includes the area of ​​the peripheral field of view close to the central field of view, near the central field of view, where stereoscopic vision does not occur.

[0071] 11 shows an example in which the gaze point setting device 16 is used, but the gaze measurement device 18 described in the second embodiment may be used instead of the gaze point setting device 16. Similarly, the remote machine operation system 100b described in the third embodiment may be added with an apparatus surroundings situation monitoring device 19, and the presented image control unit 12 may perform the operations described in this embodiment. Furthermore, when a monocular camera is used as in the first embodiment, the apparatus surroundings situation monitoring device 19 may be added, and the presented image control unit 12 may perform the operations described in this embodiment.

[0072] Embodiment 6 FIG. 13 is a diagram showing an example of the configuration of a remote machine operation system according to a sixth embodiment. A remote machine operation system 100e according to this embodiment includes a visual presentation device 1b similar to that of the fifth embodiment, and a remote machine 2c. Components having the same functions as those of the first, fourth, and fifth embodiments are given the same reference numerals as those of the first, fourth, and fifth embodiments, and redundant explanations will be omitted. Below, differences from the first, fourth, and fifth embodiments will be mainly explained.

[0073] Remote machine 2c is similar to remote machine 2b of embodiment 4, except that it has camera unit 21c instead of camera unit 21b. Camera unit 21c is similar to camera unit 21b of embodiments 4 and 5, except that zoom control unit 31 is added.

[0074] The zoom control unit 31 controls the enlargement (zoom in) and reduction (zoom out) of the image captured by the compound eye imaging unit 29. The zoom control unit 31 may control the enlargement and reduction using optical zoom or digital zoom. When the zoom control unit 31 controls the enlargement and reduction using optical zoom, for example, the compound eye imaging unit 29 may be able to change the focus by driving an optical system, or the compound eye imaging unit 29 may use a variable-focus lens. When the zoom control unit 31 controls the enlargement and reduction using digital zoom, the zoom control unit 31 enlarges the image by cropping and stretching a portion of the captured image. For example, the presented image control unit 12 may set the zoom (zoom in, zoom out) according to the operating mode. For example, the image parameters may include zoom. This allows the zoom to be set according to the operating mode, improving the operator's depth perception and operability. The zoom control unit 31 may also receive information indicating zoom in or zoom out input to the visual presentation device 1b by the operator's operation from the communication unit 22 and control the zoom of the compound eye imaging unit 29 based on the received information. The zoom control unit 31 may also control the compound eye imaging unit 29 in conjunction with the zooming in and out of the image. For example, the camera driving unit 30 may control at least one of the distance between the cameras in the compound eye imaging unit 29 and the convergence angle in accordance with the magnification determined by the zoom control unit 31.

[0075] In addition, one method to reduce the video transmission rate when communication conditions deteriorate is to use a center A method may be used in which the range captured as the field of view image is reduced and the image of that area is stretched and enlarged, or a method may be used in which the entire range captured is reduced and the image of that area is stretched and enlarged. In other words, zoom may be included as one of the image parameters that is changed depending on the communication situation. Other than what has been described above, the operation of this embodiment is the same as that of embodiment 5.

[0076] 13 shows an example in which the gaze point setting device 16 is used, but the gaze measurement device 18 described in the second embodiment may be used instead of the gaze point setting device 16. Furthermore, when a monocular camera is used as in the first embodiment, the zoom control described in this embodiment may be applied. Furthermore, the zoom control described in this embodiment may be added to the remote machine operation systems 100b and 100c described in the third and fourth embodiments.

[0077] Embodiment 7 Fig. 14 is a diagram showing an example of the configuration of a remote machine operation system according to a seventh embodiment. A remote machine operation system 100f according to this embodiment includes a visual presentation device 1b similar to that of the fifth embodiment, and a remote machine 2d. Components having the same functions as those of the first, fifth, and sixth embodiments are given the same reference numerals as those of the first, fifth, and sixth embodiments, and redundant explanations will be omitted. Below, differences from the first, fifth, and sixth embodiments will be mainly explained.

[0078] The remote machine 2d is similar to the remote machine 2c of the sixth embodiment except for the addition of a distance sensor 32. The distance sensor 32 measures the distance between the remote machine 2d and an object, such as an object to be operated by the remote machine 2d (operation target). For example, if the remote machine 2d has an end effector, the distance sensor 32 measures the distance between the end effector and the operation target. The distance sensor 32 outputs the measurement result to the zoom control unit 31. The zoom control unit 31 determines a magnification based on the measurement result and controls the enlargement and reduction of the image captured by the compound eye photographing unit 29 based on the determined magnification. For example, based on the measurement result, the zoom control unit 31 enlarges the image when the distance between the remote machine 2d and the operation target decreases, and reduces the image when the distance between the remote machine 2d and the operation target increases. For example, the zoom control unit 31 controls the enlargement and reduction of the image captured by the compound eye photographing unit 29 by determining a magnification indicating the ratio of enlargement or reduction for each distance between the remote machine 2d and the operation target.

[0079] As a result, for example, when the end effector of the remote machine 2d approaches an operation target, the image is enlarged without the need for operator operation, thereby improving operability while reducing the burden on the operator. Also, if the remote machine 2d is a drone and the operation target is a landing location such as the ground, the image is enlarged as the remote machine approaches the landing location, thereby improving operability while reducing the burden on the operator. Also, if the remote machine 2d is a vehicle and traveling through an area with an obstacle, the image is enlarged as the remote machine approaches the obstacle, thereby improving operability while reducing the burden on the operator. The operation of this embodiment other than that described above is the same as that of embodiment 6. Note that the operator may be able to turn on / off (enable or disable) automatic zoom control using the measurement results of the distance measurement sensor 32.

[0080] Alternatively, the measurement results of the distance measurement sensor 32 may be transmitted to the visual presentation device 1b by the communication unit 22, and the presentation image control unit 12 may receive the measurement results via the communication unit 15 and determine the zoom magnification using the received measurement results. The zoom control unit 31 receives the determined magnification via the communication units 15 and 28 and controls the zoom of the compound eye photographing unit 29 based on the received magnification.

[0081] 14 shows an example in which the gaze point setting device 16 is used, but the gaze measurement device 18 described in the second embodiment may be used instead of the gaze point setting device 16. Furthermore, when a monocular camera is used as in the first embodiment, zoom control using the measurement results of the distance measurement sensor 32 described in this embodiment may be applied. Furthermore, zoom control using the measurement results of the distance measurement sensor 32 described in this embodiment may be added to the remote machine operation systems 100b and 100c described in the third and fourth embodiments.

[0082] Embodiment 8 FIG. 15 is a diagram showing an example of the configuration of a remote machine operation system according to the eighth embodiment. A remote machine operation system 100g of this embodiment includes a visual presentation device 1c and a remote machine 2d similar to that of the seventh embodiment. Components having the same functions as those of the first and seventh embodiments are given the same reference numerals as those of the first and seventh embodiments, and redundant explanations will be omitted. Below, differences from the first and seventh embodiments will be mainly explained.

[0083] The visual presentation device 1c is similar to the visual presentation device 1b of the seventh embodiment except that it further includes a gain adjustment unit 20. In this embodiment, the measurement results of the distance measurement sensor 32 are transmitted to the visual presentation device 1c by the communication unit 22, and the presentation image control unit 12 receives the measurement results via the communication unit 15. The gain adjustment unit 20 determines the zoom magnification based on the measurement results, as in the seventh embodiment, and determines the gain of the motion operation of the remote machine 2d (the gain of the operation for moving the remote machine 2d) based on the magnification, modifies the control information generated by the control information generation unit 14 based on the determined gain, and transmits the modified control information to the remote machine 2d via the communication unit 15. The gain adjustment unit 20 also transmits the determined zoom magnification to the camera unit 21c of the remote machine 2d via the communication unit 15.

[0084] Here, the gain adjustment unit 20 determines the zoom magnification, but the zoom control unit 31 may determine the zoom magnification based on the measurement results of the distance measurement sensor 32 and transmit the determined zoom magnification to the visual presentation device 1c via the communication unit 28. In this case, the gain adjustment unit 20 of the visual presentation device 1c determines a gain for the operation of the remote machine 2d based on the zoom magnification received via the communication unit 15, modifies the control information generated by the control information generation unit 14 based on the determined gain, and transmits the modified control information to the remote machine 2d via the communication unit 15. Alternatively, the gain adjustment unit 20 may be provided in the remote machine 2d, and the remote machine 2d may determine a gain based on the zoom magnification received from the zoom control unit 31, modify the control information, and output the modified control information to the drive control unit 24.

[0085] Fig. 16 is a diagram showing an example of an image displayed by the visual presentation device 1c of this embodiment when the image is not enlarged or reduced. In the example shown in Fig. 16, the remote machine 2d is a construction machine, and a shovel (bucket) is gradually approaching the soil, which is the object to be operated. The size of the display graphic 301 representing the remote machine 2d in the image does not change.

[0086] FIG. 17 is a diagram illustrating an example of an image displayed by the visual presentation device 1c of this embodiment when the image is enlarged or reduced. The example shown in FIG. 17 is similar to the example shown in FIG. 16, but the image is enlarged and the size of the display graphic 301 also changes as the shovel approaches the soil, which is the object of operation. Comparing FIG. 16 and FIG. 17, the example shown in FIG. 17 clearly displays the situation around the shovel. However, when the operator performs operations based on the image, the amplitude of the operation is determined based on the distance within the displayed image. However, when the image is enlarged as shown in FIG. 17, the ratio between the distance within the displayed image and the distance in the actual captured area changes, which may result in a discrepancy between the operator's intended operation and the control amount operated by the operator. For example, when the operator performs operations based on an enlarged screen, the operator's operation amount is larger than when the operator performs operations based on a non-enlarged screen. Therefore, the gain adjustment unit 20 reduces the control amount of the operation of the remote machine 2d corresponding to the operator's operation amount based on the zoom magnification, making it easier for the operator to achieve the intended operation and improving operability.

[0087] 15 shows an example in which the gaze point setting device 16 is used, but the gaze measurement device 18 described in the second embodiment may be used instead of the gaze point setting device 16. Furthermore, when a monocular camera is used as in the first embodiment, the operation described in this embodiment may be applied. Furthermore, the operation described in this embodiment may be applied to the remote machine operation systems 100b and 100c described in the third and fourth embodiments.

[0088] Embodiment 9 FIG. 18 is a diagram showing an example of the configuration of a remote machine operation system according to a ninth embodiment. A remote machine operation system 100h according to this embodiment includes a visual presentation device 1c similar to that of the eighth embodiment, and a remote machine 2e. Components having the same functions as those of the first and eighth embodiments are given the same reference numerals as those of the first and eighth embodiments, and redundant explanations will be omitted. Below, differences from the first and eighth embodiments will be mainly explained.

[0089] Remote machine 2e is similar to remote machine 2d of embodiment 8, except that it includes camera unit 21d instead of camera unit 21c. Camera unit 21d is similar to camera unit 21c of embodiment 8, except that automatic tracking unit 33 is added.

[0090] The automatic tracking unit 33 controls the shooting direction of the compound eye shooting unit 29 so that a predetermined object is included in the central field of view image. The automatic tracking unit 33 controls the shooting direction of the compound eye shooting unit 29 so that the predetermined object is located at the center of the field of view of the compound eye shooting unit 29. The predetermined object is, for example, the end effector of the remote machine 2e. The automatic tracking unit 33 detects the predetermined object from the image captured by the compound eye shooting unit 29 and controls the shooting direction of the compound eye shooting unit 29 so that a predetermined location, such as the center of the predetermined object, is located at the center of the field of view of the compound eye shooting unit 29. This automatically tracks the end effector of the remote machine 2e, thereby reducing the operational burden on the operator. Note that the predetermined object is not limited to the end effector and may be any object that the operator wishes to focus on. The operator may also be able to turn automatic tracking on or off, i.e., whether to enable or disable automatic tracking. The predetermined object may also be set by the operator. For example, the operator may specify an area to be the defined object while the video is displayed, thereby specifying the defined object.

[0091] 18 shows an example in which the gaze point setting device 16 is used, but the gaze measurement device 18 described in the second embodiment may be used instead of the gaze point setting device 16. Furthermore, when a monocular camera is used as in the first embodiment, the operation described in this embodiment may be applied. Furthermore, the operation described in this embodiment may be applied to the remote machine operation systems 100b and 100c described in the third and fourth embodiments.

[0092] The configurations shown in the above embodiments are merely examples, and may be combined with other known technologies, or different embodiments may be combined with each other. It is also possible to omit or modify parts of the configurations as long as they do not deviate from the gist of the invention. [Explanation of symbols]

[0093] 1, 1a, 1b, 1c visual presentation device, 2, 2a, 2b, 2c, 2d, 2e remote machine, 10 presentation unit, 11 communication status monitoring unit, 12 presentation image control unit, 13 remote machine status monitoring unit, 14 control information generation unit, 15, 22, 28 communication unit, 16 gaze point setting device, 17 operation input device, 18 gaze measurement device, 19 device surrounding situation monitoring device, 20 gain adjustment unit, 21, 21a, 21b, 21c, 21d camera unit, 23 drive unit, 24 drive control unit, 25 situation acquisition device, 26 shooting unit, 27, 27a image processing unit, 29 compound eye shooting unit, 30 camera drive unit, 31 zoom control unit, 32 distance measurement sensor, 33 Automatic tracking unit, 100, 100a, 100b, 100c, 100d, 100e, 100f, 100g, 100h remote machine operation system, 201 image display area, 202 central visual field area, 203 peripheral visual field area.

Claims

1. a camera for photographing a remotely controlled remote machine; a visual presentation device that receives an image captured by the camera and displays the received image to present the image to an operator who operates the remote machine; Equipped with The visual presentation device is a communication status monitoring unit that detects a communication status in receiving the video; a presentation image control unit that determines a central visual field area and a peripheral visual field area in a display area in which the image is displayed, and determines image parameters related to the transmission rates of the central visual field image to be displayed in the central visual field area and the peripheral visual field image to be displayed in the peripheral visual field area based on the communication status detected by the communication status monitoring unit and the operation status of the remote machine; Equipped with A remote machine operation system, characterized in that the central field of view image and the peripheral field of view image are each part of the same image, and the peripheral field of view image is a part of the image that is outside the central field of view image.

2. a gaze point setting device for setting a gaze point of the operator; Equipped with 2. The remote machine operation system according to claim 1, wherein the presented image control unit determines the central visual field area based on the gaze point set by the gaze point setting device.

3. 3. The remote machine operation system according to claim 2, wherein the gaze point setting device is a line-of-sight measurement device that measures the line of sight of the operator.

4. 4. The remote machine operation system according to claim 1, wherein the image parameters include at least one of a resolution of the peripheral vision image, a frame rate of the peripheral vision image, a color of the peripheral vision area, and a size of the central vision area.

5. an automatic tracking unit that controls the shooting direction of the camera so that a predetermined object is included in the central field of view image; 4. The remote machine operation system according to claim 1, further comprising:

6. a zoom control unit that controls the enlargement and reduction of the image captured by the camera; 4. The remote machine operation system according to claim 1, further comprising:

7. a gain adjustment unit that adjusts a gain of the motion operation of the remote machine in accordance with the magnification set by the zoom control unit; 7. The remote machine operation system according to claim 6, further comprising:

8. the camera is a compound eye camera, 4. The remote machine operation system according to claim 1, wherein the image parameters include information indicating whether each of the central field of view image and the peripheral field of view image is a stereoscopic image.

9. the camera is a compound eye camera, 7. The remote machine operation system according to claim 6, wherein the image parameters include information indicating whether each of the central field of view image and the peripheral field of view image is a stereoscopic image.

10. a camera driver that controls at least one of the shooting direction of the compound eye camera, the distance between the cameras of the compound eye camera, and the convergence angle of the compound eye camera; 10. The remote machine operation system according to claim 9, further comprising:

11. The remote machine operation system according to claim 9 , wherein the zoom control unit controls the camera in association with the zooming in and out of the image.

12. a device for monitoring the surroundings of the device, which detects whether or not an object is present in the operator's field of view corresponding to the central field of view image; Equipped with 10. The remote machine operation system of claim 9, wherein the presentation image control unit reduces the resolution of the peripheral field of view image when the central field of view image is a stereoscopic image and the presence of the object is detected by the device surroundings situation monitoring device.

13. a camera for photographing a remotely controlled remote machine; a visual presentation device that receives an image captured by the camera and displays the received image to present the image to an operator who operates the remote machine; Equipped with The visual presentation device is a communication status monitoring unit that detects a communication status in receiving the video; a presentation image control unit that determines a central visual field area and a peripheral visual field area in a display area where the image is displayed, and determines image parameters related to transmission rates of the central visual field image to be displayed in the central visual field area and the peripheral visual field image to be displayed in the peripheral visual field area based on the communication status detected by the communication status monitoring unit; Equipped with the camera is a compound eye camera, the image parameters include information indicating whether each of the central field of view image and the peripheral field of view image is to be a stereoscopic image; a device for monitoring the surroundings of the device, which detects whether or not an object is present in the operator's field of view corresponding to the central field of view image; Equipped with The presentation image control unit is characterized in that, when the central field of view image is a stereoscopic image, if the presence of the object is detected by the device surroundings situation monitoring device, the resolution of the peripheral field of view image is reduced.

14. A visual presentation device that receives an image from a camera that captures a remotely operated remote machine, and presents the image to an operator who operates the remote machine by displaying the received image, a communication status monitoring unit that detects a communication status in receiving the video; a presentation image control unit that determines a central visual field area and a peripheral visual field area in a display area in which the image is displayed, and determines image parameters related to the transmission rates of the central visual field image to be displayed in the central visual field area and the peripheral visual field image to be displayed in the peripheral visual field area based on the communication status detected by the communication status monitoring unit and the operation status of the remote machine; Equipped with A visual presentation device characterized in that the central field of view image and the peripheral field of view image are each part of the same image, and the peripheral field of view image is a part of the image that is outside the central field of view image.

15. 1. A visual presentation method for a visual presentation device that receives an image from a camera that captures a remotely operated remote machine, and presents the image to an operator who operates the remote machine by displaying the received image, comprising: detecting a communication status in receiving the video; determining a central visual field area and a peripheral visual field area in a display area for displaying the image, and determining image parameters related to the transmission rates of the central visual field image to be displayed in the central visual field area and the peripheral visual field image to be displayed in the peripheral visual field area based on the detected communication status and the operation status of the remote machine; Including, A visual presentation method characterized in that the central field of view image and the peripheral field of view image are each part of the same image, and the peripheral field of view image is a part of the image that is outside the central field of view image.

16. A program executed by a visual presentation device that receives an image from a camera that captures a remotely operated remote machine and displays the received image to present the image to an operator operating the remote machine, detecting a communication status in receiving the video; determining a central visual field area and a peripheral visual field area in a display area for displaying the image, and determining image parameters related to the transmission rates of the central visual field image to be displayed in the central visual field area and the peripheral visual field image to be displayed in the peripheral visual field area based on the detected communication status and the operation status of the remote machine; and The central vision image and the peripheral vision image are each a part of the same image, and the peripheral vision image is a part of the image that is outside the central vision image.

17. A visual presentation device that receives an image from a camera that photographs a remotely operated remote machine, and presents the image to an operator operating the remote machine by displaying the received image, a communication status monitoring unit that detects a communication status in receiving the video; a presentation image control unit that determines a central visual field area and a peripheral visual field area in a display area where the image is displayed, and determines image parameters related to transmission rates of the central visual field image to be displayed in the central visual field area and the peripheral visual field image to be displayed in the peripheral visual field area based on the communication status detected by the communication status monitoring unit; Equipped with the camera is a compound eye camera, the image parameters include information indicating whether each of the central field of view image and the peripheral field of view image is to be a stereoscopic image; The visual presentation device is characterized in that, when the central field of view image is a stereoscopic image, the presentation image control unit reduces the resolution of the peripheral field of view image when an apparatus surrounding situation monitoring device detects whether an object is present in the operator's field of view corresponding to the central field of view image and detects that the object is present.

18. A visual presentation method in a visual presentation device that receives an image from a camera that photographs a remotely operated remote machine, and presents the image to an operator operating the remote machine by displaying the received image, comprising: a communication status monitoring step for detecting a communication status in receiving the video; a presentation image control step of determining a central visual field area and a peripheral visual field area in a display area where the image is displayed, and determining image parameters related to transmission rates of the central visual field image to be displayed in the central visual field area and the peripheral visual field image to be displayed in the peripheral visual field area based on the communication status detected in the communication status monitoring step; Including, the camera is a compound eye camera, the image parameters include information indicating whether each of the central field of view image and the peripheral field of view image is to be a stereoscopic image; In the presented image control step, when the central field of view image is a stereoscopic image, if a device surroundings monitoring device that detects whether an object is present in the operator's field of view corresponding to the central field of view image detects that the object is present, the visual presentation method is characterized in that the resolution of the peripheral field of view image is reduced.

19. A program executed by a visual presentation device that receives an image from a camera that photographs a remotely operated remote machine, and displays the received image to present the image to an operator operating the remote machine, a communication status monitoring step for detecting a communication status in receiving the video; a presentation image control step of determining a central visual field area and a peripheral visual field area in a display area where the image is displayed, and determining image parameters related to transmission rates of the central visual field image to be displayed in the central visual field area and the peripheral visual field image to be displayed in the peripheral visual field area based on the communication status detected in the communication status monitoring step; and the camera is a compound eye camera, the image parameters include information indicating whether each of the central field of view image and the peripheral field of view image is to be a stereoscopic image; In the presented image control step, when the central field of view image is a stereoscopic image, if a device surroundings monitoring device that detects whether an object is present in the operator's field of view corresponding to the central field of view image detects that the object is present, the program reduces the resolution of the peripheral field of view image.

Images